feat(monitor): incremental registry sweeps + per-skill resilience#16
Merged
Conversation
The registry monitor re-scanned every skill on every run, so a daily sweep cost the same as the first (~25 min) and one flaky network fetch (an unwrapped httpx ConnectTimeout) could abort the whole run. Make the sweep incremental and resilient: - build_snapshot() now takes the previous snapshot and only re-fetches + re-scans skills whose version/updated_at changed; unchanged skills are carried forward. The version/updated_at metadata comes free from the list endpoint, so change detection needs no per-skill fetch. First run (or --full) still scans everything. - Add a --full flag / force_rescan to re-scan every skill on a periodic cadence, catching silent same-version content swaps that metadata-based detection would miss (the ClawHavoc trojanization pattern). - Harden _scan_slug and the worker to catch any per-skill exception (timeouts included) so one bad skill can't kill the sweep. - Record scanned_count/reused_count on each snapshot for visibility. - Workflow: daily incremental + weekly (--full) cron, dispatch input to force full, timeout raised to 60m for the weekly full run. - Tests for reuse, version/updated_at change, new skill, --full, errored- record retry, and non-ClawHubError containment. Docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
✅ Malwar Scan Results
Scanned by Malwar — malware detection for agentic AI skills. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The registry monitor re-scanned every skill on every run, so a daily sweep cost the same as the baseline (~25 min + LLM re-escalation), and one flaky network fetch (an unwrapped
httpx.ConnectTimeout) aborted the whole run — which is exactly what killed the last two live runs.This makes the sweep incremental and resilient, so it behaves the way you'd expect: first run is the expensive baseline, every run after only pays for what actually changed.
What changed
build_snapshot()now takes the previous snapshot and only re-fetches + re-scans skills whoseversion/updated_atchanged since the last snapshot; unchanged skills are carried forward untouched. The version/updated_at metadata comes free from the list endpoint, so change detection costs zero extra per-skill fetches. A daily run drops from ~25 min to seconds–minutes, and LLM cost drops to just the skills that changed.--full/force_rescan. Re-scans every skill on a periodic cadence to catch silent same-version content swaps — trojanized updates that keep the same version number (the ClawHavoc pattern), which metadata-based detection can't see on its own._scan_slugand the worker now catch any per-skill exception (timeouts included) and record it as an error instead of letting it abort the whole sweep.scanned_countvsreused_count; the CLI printsScanned N changed/new, reused M unchanged.--fullcron (Sun 07:00 UTC), aworkflow_dispatchinput to force a full run, and the timeout raised to 60 min for the weekly full sweep.The one honest caveat, handled
Incremental detection trusts the registry's version metadata. A silent same-version swap would slip past a pure daily incremental — so the weekly
--fullsweep exists specifically to close that gap. Best of both: cheap dailies, full coverage weekly.Type of Change
Checklist
--full, errored-record retry, non-ClawHubError containment) — 1585 pass, 16 skippedruff check src/ tests/passesmypycleandocs/crawl.md, snapshots README)🤖 Generated with Claude Code
https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
Generated by Claude Code