HTML capture: extraction gate, archetype routing, newsletters (Substack + email digests) (#2)

[janikmu]

Closes the implementation half of #2 ("Decide on and test HTML capture expected outcomes"). The analysis + agreed roadmap live in the issue comments; this PR builds Phases 0–3 plus newsletter subscriptions and HTML-email digest ingest.

Why

Capturing the open web faithfully is the hard part of #2. Running the example URLs through raidar's real extractor showed two structural risks: capture would silently fabricate junk artifacts from login walls / JS-only pages / dead newsletter links, and blanket "bulk-capture every GitHub link" would ingest newsletter self-promo footers as signal. Priority 1 was newsletters — which turned out to be the least faithfully-capturable category until handled deliberately. Guiding doctrine throughout: a skipped capture beats a polluted knowledge base.

What changed

Phase 0 — do-no-harm extraction gate (lib/webextract.py)
Centralises fetch + a confidence gate; capture now refuses an unusable extraction (login wall, JS SPA, dead link, too-thin) with an actionable hint instead of classifying nav cruft or the bare URL.

Phase 1 — archetype routing + bulk-capture gate
New LLM page_kind (biased to article) refuses digest/index pages as single artifacts; pre-fetch URL hints for known newsletter landing/email links. bulk-capture gains an awesome-list shape gate (repo count + owner spread + diversity) so it stops eating newsletter footers and incidental partner links.

Phase 2 — Substack newsletters (lib/substack.py, lib/subscriptions.py, jobs/newsletter.py)

• Manual: a pasted Substack landing/archive URL auto-expands to per-post permalinks via the unauthenticated /api/v1/posts API; list-and-pick or --expand-all.
• Recurring: raidar newsletter {subscribe,unsubscribe,list,poll} + a weekly com.airadar.newsletters launchd agent. Cadence-agnostic — "new" is decided by canonical post URL, never URL arithmetic or an assumed schedule.
• Captured items carry provenance (source_url + via).

Phase 3 — in-body implementation repos
For a single article, the classifier returns linked_repos, grounded against the article text; --follow-repos captures them as siblings under the same concept (e.g. ghuntley.com/ralph → repomirror). Default is to suggest, not auto-ingest.

Mode C — HTML-email digest ingest (lib/email_digest.py)
Newsletters like AlphaSignal arrive as HTML email (web archive is auth/bot-walled). raidar newsletter ingest <file.eml> parses the MIME, LLM-triages links to drop ads/sponsors + chrome before resolving anything (sponsor trackers are never clicked), follows the click-tracking redirects to real destinations, auto-captures the GitHub repos (with provenance), and prints a review list of everything else (HF models, articles) with ready-to-run capture commands.

Verification

• Offline unit smoke tests for every new lib (webextract, substack, subscriptions, email_digest) wired into infra/smoke.sh.
• Live-verified against the Decide on and test HTML capture expected outcomes #2 example URLs: pollution cases (LinkedIn, AlphaSignal, Substack index) now refused; page_kind correctly labels ethical.institute→digest, ghuntley→article, refactoring.fm/s/essays→index; the bulk gate refuses the 6-link newsletter and passes a 40-repo list.
• Sandbox-verified end to end (isolated vault): Substack expand + provenance, subscription baseline + poll new-edition detection, ghuntley --follow-repos (repo lands in the same concept with signal tracking), and email ingest (triage 4 content / 1 ad / 1 chrome; sponsor never resolved; repos route to capture, HF model + news article to the review list).

Reviewer notes

• Bundles one pre-existing commit (896a67d "Make launchd agent install reachable after initial setup") that was the branch base and isn't on main yet — happy to split it out if you'd prefer it in its own PR.
• Deferred to their own issues (as agreed on Decide on and test HTML capture expected outcomes #2): Phase 4 (general non-Substack digest decomposition, LinkedIn), Phase 5 (non-GitHub artifact types & signals — HF/Ollama/Kaggle/arXiv). Phase 5 is what would let email ingest auto-capture the models it currently review-lists.
• Data model: adds source_url/via provenance and a subscriptions.json registry beside the config (not a vault entity); no changes to the concept/artifact schema.
• After merge, run raidar install-launchd to pick up the new weekly newsletters agent (raidar health warns until then).
• Unrelated pre-existing finding: raidar health flags a dangling ref (agent-development-frameworks → multica-ai/multica) in the vault — worth a separate cleanup.

🤖 Generated with Claude Code