diff --git a/.github/workflows/registry-monitor.yml b/.github/workflows/registry-monitor.yml index e06c027..553634a 100644 --- a/.github/workflows/registry-monitor.yml +++ b/.github/workflows/registry-monitor.yml @@ -4,11 +4,16 @@ # the `registry-snapshots` branch so `git log -p registry-snapshots` is a # permanent, auditable record of what changed day over day. # +# Scans with the rule engine + threat intel only (no per-skill LLM calls) so a +# whole-registry sweep is fast and cheap; the rule engine alone catches 100% of +# malicious samples in the benchmark. Use `malwar crawl scan ` for an LLM +# deep-dive on an individual flagged skill. +# # Two cadences: # * Daily (06:00 UTC) — INCREMENTAL. Only skills whose version/updated_at -# changed since the last snapshot are re-fetched, re-scanned, and (if -# flagged) escalated to the LLM. The first run scans everything; every run -# after is cheap and fast. +# changed since the last snapshot are re-fetched and re-scanned. The first +# runs build the baseline (see --max-scans below); after that each run is +# cheap and fast. # * Weekly (Sunday 07:00 UTC) — FULL (`--full`). Re-scans every skill to catch # silent same-version content swaps (trojanized updates that keep the same # version number), which incremental detection would otherwise miss. @@ -17,11 +22,6 @@ # checks), so a scheduled bot cannot push to it. The snapshot history therefore # lives on its own long-running `registry-snapshots` branch, created # automatically on the first run. -# -# The LLM escalation needs an Anthropic API key. Add it as a *repository* secret -# named MALWAR_ANTHROPIC_API_KEY (Settings -> Secrets and variables -> Actions -> -# New repository secret). Without it the sweep still runs on the rule engine + -# threat intel; only the LLM second opinion on flagged skills is skipped. name: Registry Monitor @@ -97,15 +97,23 @@ jobs: else echo "Mode: incremental (only changed skills)." fi - # ClawHub is rate-limited (~2 req/s), so a full ~6k-skill registry - # can't be swept in one CI run. Cap scans per run; the overflow is - # deferred and picked up on subsequent runs, so the baseline builds - # up over a few days and then daily runs only touch what changed. + # Rules + threat-intel only (--no-escalate): the rule engine alone + # detects 100% of malicious samples in the benchmark, and skipping + # per-skill LLM calls keeps a whole-registry sweep fast and cheap. + # (Run `malwar crawl scan ` for an LLM deep-dive on any one + # flagged skill.) + # + # ClawHub is rate-limited (~2 req/s), so even so a full ~6k-skill + # registry can't be swept in one CI run. Cap scans per run; the + # overflow is deferred and picked up on subsequent runs, so the + # baseline builds up over a couple of runs and then daily runs only + # touch what changed. malwar crawl monitor \ --snapshot-dir data/registry-snapshots \ --format json \ --output data/registry-snapshots/latest-diff.json \ - --max-scans 1500 \ + --no-escalate \ + --max-scans 3000 \ $FULL - name: Publish snapshot to data branch diff --git a/docs/crawl.md b/docs/crawl.md index 70c6ce2..baf0825 100644 --- a/docs/crawl.md +++ b/docs/crawl.md @@ -116,7 +116,7 @@ malwar crawl monitor --max 100 --no-escalate # quick partial run, rules only **Options:** `--snapshot-dir`, `--full`, `--max-scans`, `--max`, `--no-escalate`, `--concurrency`, `--format`/`-f` (`console`|`json`), `--output`/`-o`, `--no-save`, `--digest`, `--publish`, `--fail-on-malicious`. Publishing to X requires the `MALWAR_X_*` credentials (see [Configuration](deployment/configuration.md#x-twitter-publishing)). -The bundled GitHub Actions workflow (`.github/workflows/registry-monitor.yml`) runs this on two cadences: **daily incremental** and a **weekly `--full`** re-scan, committing each snapshot to the `registry-snapshots` branch. +The bundled GitHub Actions workflow (`.github/workflows/registry-monitor.yml`) runs this on two cadences: **daily incremental** and a **weekly `--full`** re-scan, committing each snapshot to the `registry-snapshots` branch. It runs **rules-only (`--no-escalate`)** so a whole-registry sweep stays fast and cheap — the rule engine alone catches every malicious sample in the benchmark; use `malwar crawl scan ` for an LLM deep-dive on an individual flagged skill. ---