Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 21 additions & 13 deletions .github/workflows/registry-monitor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,16 @@
# the `registry-snapshots` branch so `git log -p registry-snapshots` is a
# permanent, auditable record of what changed day over day.
#
# Scans with the rule engine + threat intel only (no per-skill LLM calls) so a
# whole-registry sweep is fast and cheap; the rule engine alone catches 100% of
# malicious samples in the benchmark. Use `malwar crawl scan <slug>` for an LLM
# deep-dive on an individual flagged skill.
#
# Two cadences:
# * Daily (06:00 UTC) — INCREMENTAL. Only skills whose version/updated_at
# changed since the last snapshot are re-fetched, re-scanned, and (if
# flagged) escalated to the LLM. The first run scans everything; every run
# after is cheap and fast.
# changed since the last snapshot are re-fetched and re-scanned. The first
# runs build the baseline (see --max-scans below); after that each run is
# cheap and fast.
# * Weekly (Sunday 07:00 UTC) — FULL (`--full`). Re-scans every skill to catch
# silent same-version content swaps (trojanized updates that keep the same
# version number), which incremental detection would otherwise miss.
Expand All @@ -17,11 +22,6 @@
# checks), so a scheduled bot cannot push to it. The snapshot history therefore
# lives on its own long-running `registry-snapshots` branch, created
# automatically on the first run.
#
# The LLM escalation needs an Anthropic API key. Add it as a *repository* secret
# named MALWAR_ANTHROPIC_API_KEY (Settings -> Secrets and variables -> Actions ->
# New repository secret). Without it the sweep still runs on the rule engine +
# threat intel; only the LLM second opinion on flagged skills is skipped.

name: Registry Monitor

Expand Down Expand Up @@ -97,15 +97,23 @@ jobs:
else
echo "Mode: incremental (only changed skills)."
fi
# ClawHub is rate-limited (~2 req/s), so a full ~6k-skill registry
# can't be swept in one CI run. Cap scans per run; the overflow is
# deferred and picked up on subsequent runs, so the baseline builds
# up over a few days and then daily runs only touch what changed.
# Rules + threat-intel only (--no-escalate): the rule engine alone
# detects 100% of malicious samples in the benchmark, and skipping
# per-skill LLM calls keeps a whole-registry sweep fast and cheap.
# (Run `malwar crawl scan <slug>` for an LLM deep-dive on any one
# flagged skill.)
#
# ClawHub is rate-limited (~2 req/s), so even so a full ~6k-skill
# registry can't be swept in one CI run. Cap scans per run; the
# overflow is deferred and picked up on subsequent runs, so the
# baseline builds up over a couple of runs and then daily runs only
# touch what changed.
malwar crawl monitor \
--snapshot-dir data/registry-snapshots \
--format json \
--output data/registry-snapshots/latest-diff.json \
--max-scans 1500 \
--no-escalate \
--max-scans 3000 \
$FULL

- name: Publish snapshot to data branch
Expand Down
2 changes: 1 addition & 1 deletion docs/crawl.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ malwar crawl monitor --max 100 --no-escalate # quick partial run, rules only

**Options:** `--snapshot-dir`, `--full`, `--max-scans`, `--max`, `--no-escalate`, `--concurrency`, `--format`/`-f` (`console`|`json`), `--output`/`-o`, `--no-save`, `--digest`, `--publish`, `--fail-on-malicious`. Publishing to X requires the `MALWAR_X_*` credentials (see [Configuration](deployment/configuration.md#x-twitter-publishing)).

The bundled GitHub Actions workflow (`.github/workflows/registry-monitor.yml`) runs this on two cadences: **daily incremental** and a **weekly `--full`** re-scan, committing each snapshot to the `registry-snapshots` branch.
The bundled GitHub Actions workflow (`.github/workflows/registry-monitor.yml`) runs this on two cadences: **daily incremental** and a **weekly `--full`** re-scan, committing each snapshot to the `registry-snapshots` branch. It runs **rules-only (`--no-escalate`)** so a whole-registry sweep stays fast and cheap — the rule engine alone catches every malicious sample in the benchmark; use `malwar crawl scan <slug>` for an LLM deep-dive on an individual flagged skill.

---

Expand Down
Loading