feat(monitor): incremental registry sweeps + per-skill resilience

.github/workflows/registry-monitor.yml

@@ -1,10 +1,17 @@
-# Daily ClawHub registry sweep.
+# ClawHub registry monitor.
 #
-# Runs `malwar crawl monitor` once a day: crawls the whole registry, fast-scans
-# every skill (escalating flagged ones to the LLM), diffs against the previous
-# snapshot, and commits the new snapshot + diff to the `registry-snapshots`
-# branch so `git log -p registry-snapshots` is a permanent, auditable record of
-# what changed day over day.
+# Runs `malwar crawl monitor` on a schedule and commits the snapshot + diff to
+# the `registry-snapshots` branch so `git log -p registry-snapshots` is a
+# permanent, auditable record of what changed day over day.
+#
+# Two cadences:
+#   * Daily (06:00 UTC) — INCREMENTAL. Only skills whose version/updated_at
+#     changed since the last snapshot are re-fetched, re-scanned, and (if
+#     flagged) escalated to the LLM. The first run scans everything; every run
+#     after is cheap and fast.
+#   * Weekly (Sunday 07:00 UTC) — FULL (`--full`). Re-scans every skill to catch
+#     silent same-version content swaps (trojanized updates that keep the same
+#     version number), which incremental detection would otherwise miss.
 #
 # Why a dedicated branch and not `main`? `main` is protected (PR + required
 # checks), so a scheduled bot cannot push to it. The snapshot history therefore
@@ -20,8 +27,14 @@ name: Registry Monitor
 
 on:
   schedule:
-    - cron: "0 6 * * *" # 06:00 UTC daily
+    - cron: "0 6 * * *" # daily 06:00 UTC — incremental
+    - cron: "0 7 * * 0" # Sunday 07:00 UTC — full re-scan
   workflow_dispatch: # manual "run now" from the Actions tab
+    inputs:
+      full:
+        description: "Force a full re-scan of every skill"
+        type: boolean
+        default: false
 
 # Never let two sweeps race on the snapshot history / the push.
 concurrency:
@@ -35,7 +48,9 @@ jobs:
   monitor:
     name: Sweep registry and commit report
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    # The weekly full re-scan is the long pole; incremental daily runs finish
+    # far sooner once the baseline exists.
+    timeout-minutes: 60
 
     steps:
       - name: Checkout repository
@@ -70,11 +85,23 @@ jobs:
       - name: Sweep registry and diff against baseline
         env:
           MALWAR_ANTHROPIC_API_KEY: ${{ secrets.MALWAR_ANTHROPIC_API_KEY }}
+          # Full re-scan on the weekly cron or when manually requested;
+          # incremental otherwise.
+          IS_WEEKLY: ${{ github.event.schedule == '0 7 * * 0' }}
+          DISPATCH_FULL: ${{ github.event.inputs.full }}
         run: |
+          FULL=""
+          if [ "$IS_WEEKLY" = "true" ] || [ "$DISPATCH_FULL" = "true" ]; then
+            FULL="--full"
+            echo "Mode: FULL re-scan (every skill)."
+          else
+            echo "Mode: incremental (only changed skills)."
+          fi
           malwar crawl monitor \
             --snapshot-dir data/registry-snapshots \
             --format json \
-            --output data/registry-snapshots/latest-diff.json
+            --output data/registry-snapshots/latest-diff.json \
+            $FULL
 
       - name: Publish snapshot to data branch
         run: |

data/registry-snapshots/README.md

@@ -8,14 +8,21 @@ This directory is the on-disk history for `malwar crawl monitor`.
 
 Each snapshot records, per skill: the published version, a SHA-256 of the
 `SKILL.md`, the scan verdict and risk score, matched rule IDs, install count,
-and ClawHub moderation flags. Because the files are committed to the repo,
-`git diff` (or `git log -p`) is a permanent, auditable record of exactly what
-changed in the registry day over day — new skills, removed skills, trojanized
-updates, and verdict regressions.
+and ClawHub moderation flags. It also records `scanned_count` vs `reused_count`
+for the run. Because the files are committed to the repo, `git diff` (or
+`git log -p`) is a permanent, auditable record of exactly what changed in the
+registry day over day — new skills, removed skills, trojanized updates, and
+verdict regressions.
+
+The sweep is **incremental**: only skills whose version/updated_at changed since
+`latest.json` are re-scanned; the rest are carried forward. The first run scans
+everything. A periodic `--full` run re-scans every skill to catch silent
+same-version content swaps.
 
 Run the monitor:
 
 ```bash
-malwar crawl monitor                 # full sweep, diff against latest.json, save
+malwar crawl monitor                 # incremental sweep, diff against latest.json, save
+malwar crawl monitor --full          # full re-scan (weekly cadence recommended)
 malwar crawl monitor --fail-on-malicious   # non-zero exit on newly-flagged skills
 ```

docs/crawl.md

@@ -97,17 +97,22 @@ malwar crawl url https://raw.githubusercontent.com/user/repo/main/SKILL.md --for
 
 ### crawl monitor
 
-Sweep the **entire** registry, fast-scan every skill (rule engine + threat intel, escalating flagged ones to the LLM), and diff the result against the previous snapshot to surface what changed -- newly published skills, removed skills, trojanized updates (content changed under the same version), and verdict regressions, with a headline list of skills that newly turned malicious. Snapshots persist under `data/registry-snapshots/` (`latest.json` is the diff baseline, plus a dated archive per run), so committing that directory makes `git diff` a permanent day-over-day record. Designed to run on a schedule.
+Sweep the registry, fast-scan skills (rule engine + threat intel, escalating flagged ones to the LLM), and diff the result against the previous snapshot to surface what changed -- newly published skills, removed skills, trojanized updates (content changed under the same version), and verdict regressions, with a headline list of skills that newly turned malicious. Snapshots persist under `data/registry-snapshots/` (`latest.json` is the diff baseline, plus a dated archive per run), so committing that directory makes `git diff` a permanent day-over-day record. Designed to run on a schedule.
+
+**Incremental by default.** The monitor only re-fetches and re-scans skills whose `version`/`updated_at` changed since the last snapshot; unchanged skills are carried forward untouched. The **first run scans everything**; every run after only pays for what actually changed — so a daily sweep is fast and cheap. Because incremental detection trusts the registry's version metadata, run a periodic **`--full`** sweep (e.g. weekly) to also catch *silent same-version content swaps* — trojanized updates that keep the same version number. Each snapshot records `scanned_count` vs `reused_count` so you can see how much a run actually did.
 
 ```bash
-malwar crawl monitor                     # full sweep -> snapshot -> diff
+malwar crawl monitor                     # incremental sweep -> snapshot -> diff
+malwar crawl monitor --full              # re-scan every skill (catches same-version tampering)
 malwar crawl monitor --digest            # also print a shareable digest + draft post
 malwar crawl monitor --publish           # post the digest to X when skills newly turn malicious
 malwar crawl monitor --fail-on-malicious # exit non-zero when newly-flagged skills are found (CI/cron)
 malwar crawl monitor --max 100 --no-escalate   # quick partial run, rules only
 ```
 
-**Options:** `--snapshot-dir`, `--max`, `--no-escalate`, `--concurrency`, `--format`/`-f` (`console`|`json`), `--output`/`-o`, `--no-save`, `--digest`, `--publish`, `--fail-on-malicious`. Publishing to X requires the `MALWAR_X_*` credentials (see [Configuration](deployment/configuration.md#x-twitter-publishing)).
+**Options:** `--snapshot-dir`, `--full`, `--max`, `--no-escalate`, `--concurrency`, `--format`/`-f` (`console`|`json`), `--output`/`-o`, `--no-save`, `--digest`, `--publish`, `--fail-on-malicious`. Publishing to X requires the `MALWAR_X_*` credentials (see [Configuration](deployment/configuration.md#x-twitter-publishing)).
+
+The bundled GitHub Actions workflow (`.github/workflows/registry-monitor.yml`) runs this on two cadences: **daily incremental** and a **weekly `--full`** re-scan, committing each snapshot to the `registry-snapshots` branch.
 
 ---
 

src/malwar/cli/commands/crawl.py

@@ -515,13 +515,24 @@ def crawl_monitor(
             help="Exit non-zero when newly-flagged skills are found",
         ),
     ] = False,
+    full: Annotated[
+        bool,
+        typer.Option(
+            "--full",
+            help="Re-scan every skill instead of only changed ones "
+            "(defends against silent same-version content swaps)",
+        ),
+    ] = False,
 ) -> None:
-    """Scan every skill in the registry and diff against the last snapshot.
+    """Scan the registry incrementally and diff against the last snapshot.
 
-    Designed to run on a schedule (e.g. daily): it crawls the whole registry,
-    fast-scans each skill (escalating flagged ones to the LLM), diffs the
-    result against the previous snapshot, and reports what changed — with a
-    headline list of skills that newly turned malicious.
+    Designed to run on a schedule (e.g. daily): it crawls the whole registry
+    but only re-fetches and re-scans skills whose version changed since the last
+    snapshot (escalating flagged ones to the LLM), diffs the result, and reports
+    what changed — with a headline list of skills that newly turned malicious.
+
+    The first run scans everything. Pass ``--full`` on a periodic cadence (e.g.
+    weekly) to re-scan every skill and catch silent same-version tampering.
     """
     exit_code = asyncio.run(
         _async_monitor(
@@ -535,6 +546,7 @@ def crawl_monitor(
             digest=digest,
             publish=publish,
             fail_on_malicious=fail_on_malicious,
+            full=full,
         )
     )
     raise typer.Exit(exit_code)
@@ -552,6 +564,7 @@ async def _async_monitor(
     digest: bool,
     publish: bool,
     fail_on_malicious: bool,
+    full: bool = False,
 ) -> int:
     from rich.progress import BarColumn, Progress, TextColumn
 
@@ -560,8 +573,9 @@ async def _async_monitor(
     store = SnapshotStore(snapshot_dir)
     previous = store.load_latest()
 
+    mode = "full re-scan" if full else ("incremental" if previous else "first run")
     scope = f" (max {max_skills})" if max_skills else ""
-    console.print(f"Crawling ClawHub registry{scope}...", style="dim")
+    console.print(f"Crawling ClawHub registry{scope} [{mode}]...", style="dim")
 
     with Progress(
         TextColumn("[progress.description]{task.description}"),
@@ -578,6 +592,8 @@ def _on_progress(done: int, total: int, slug: str) -> None:
             )
 
         snapshot = await build_snapshot(
+            previous=previous,
+            force_rescan=full,
             max_skills=max_skills,
             escalate=escalate,
             concurrency=concurrency,
@@ -588,6 +604,11 @@ def _on_progress(done: int, total: int, slug: str) -> None:
         console.print("[yellow]No skills found in the registry.[/yellow]")
         return 0
 
+    console.print(
+        f"[dim]Scanned {snapshot.scanned_count} changed/new, "
+        f"reused {snapshot.reused_count} unchanged.[/dim]"
+    )
+
     diff = diff_snapshots(previous, snapshot)
 
     if save:

src/malwar/monitor/models.py

@@ -64,6 +64,10 @@ class RegistrySnapshot(BaseModel):
     skills: dict[str, SkillRecord] = Field(default_factory=dict)
     # Slugs the crawl knew about but failed to fetch/scan, for transparency.
     errors: dict[str, str] = Field(default_factory=dict)
+    # How this snapshot was built: skills actually fetched + scanned this run
+    # vs. carried forward unchanged from the previous snapshot (incremental).
+    scanned_count: int = 0
+    reused_count: int = 0
 
     @property
     def skill_count(self) -> int: