HTML capture: extraction gate, archetype routing, newsletters (Substack + email digests) (#2)

README.md

@@ -16,9 +16,9 @@ A concept is a *capability several artifacts could implement as alternatives* 
 
 ## Architecture in one breath
 
-- **Three jobs**: `capture` (on-demand), `enrich` (Sunday 20:00 via launchd),
-  `digest` (Sunday 21:00 via launchd). All jobs and query utilities are accessed via a unified global CLI executable.
-- **Unified CLI**: `raidar` with subcommands (`capture`, `enrich`, `digest`, `seed`, `backfill`, `reevaluate`, `search`, plus vault-hygiene commands `health`, `merge-concept`, `reindex`).
+- **Four jobs**: `capture` (on-demand), `newsletter poll` (Sunday 19:00 via launchd),
+  `enrich` (Sunday 20:00 via launchd), `digest` (Sunday 21:00 via launchd). All jobs and query utilities are accessed via a unified global CLI executable.
+- **Unified CLI**: `raidar` with subcommands (`capture`, `newsletter`, `enrich`, `digest`, `seed`, `backfill`, `reevaluate`, `search`, plus vault-hygiene commands `health`, `merge-concept`, `reindex`).
 - **One LLM router** (`lib/llm.py`) routes per-task to a configured chain of
   OpenAI-wire-compatible providers (academic proxy → local LMStudio fallback).
 - **Local embeddings** via LMStudio (any OpenAI-compatible embedding model), flat JSON indexes, numpy cosine.
@@ -102,7 +102,7 @@ Ensure automated background processing runs weekly:
 
 ```bash
 raidar install-launchd
-launchctl list | grep airadar     # both com.airadar.enrich and com.airadar.digest should appear
+launchctl list | grep airadar     # com.airadar.newsletters, com.airadar.enrich and com.airadar.digest should appear
 ```
 
 (equivalently `./infra/install_launchd.sh`, which the command wraps — or pass `--launchd` to `raidar init` to do this in the same step). This setup:
@@ -121,8 +121,9 @@ Open this directory as a Claude Cowork project with filesystem access and shell
 ```
 ai-radar-tool/                    (this repo - stateless utility)
   jobs/
-    capture.py                    on-demand capture (URL or text -> artifact + concept)
-    bulk_capture.py               bulk capture from awesome-lists and newsletter pages
+    capture.py                    on-demand capture (URL or text -> artifact + concept); Substack expand + in-body repo follow
+    bulk_capture.py               bulk capture from awesome-lists (awesome-list shape gate)
+    newsletter.py                 newsletter subscriptions + recurring poll (Substack); HTML-email digest ingest
     enrich.py                     weekly signal refresh + LLM re-evaluation (two passes)
     digest.py                     weekly markdown digest
     backfill.py                   bulk star-history backfill for artifacts
@@ -140,11 +141,15 @@ ai-radar-tool/                    (this repo - stateless utility)
     embeddings.py                 Ollama embeddings + split numpy indexes
     github.py                     GitHub API client (httpx + tenacity)
     body.py                       canonical body renderer/parser for concepts and artifacts
+    webextract.py                 trafilatura fetch + extraction-quality gate (refuses junk pages)
+    substack.py                   Substack discovery (landing/archive -> per-post permalinks via API)
+    subscriptions.py              newsletter subscription registry (JSON, beside config)
+    email_digest.py               HTML-email newsletter parsing (MIME -> text + links, tracker resolution)
     config.py                     config.yaml loader / active config resolver
     secrets.py                    .env / env-var access
     logging_setup.py              logging configured once per process
   infra/
-    launchd/com.airadar.{enrich,digest}.plist   templates (placeholders substituted on install)
+    launchd/com.airadar.{newsletters,enrich,digest}.plist   templates (placeholders substituted on install)
     install_launchd.sh            install / uninstall with OS safeguards & PATH detection
     smoke.sh                      offline acceptance test
     test_sandbox.sh               sandboxed isolated integration test

SKILL.md

@@ -33,11 +33,39 @@ raidar init --vault "~/raidar-vault"
 ### Capture — add or update an artifact/concept
 ```bash
 raidar capture "<url-or-text>"
-raidar capture --force "<input>"        # bypass dedup warning
-raidar capture --dry-run "<input>"      # preview, no writes
+raidar capture --force "<input>"           # bypass dedup warning
+raidar capture --dry-run "<input>"         # preview, no writes
+raidar capture --follow-repos "<article>"  # also capture the article's in-body implementation repo(s)
+raidar capture --expand-all "<substack>"   # capture ALL posts of a pasted Substack newsletter
 ```
 Accepts a GitHub URL, any other web URL, or free-form text. The LLM automatically classifies the artifact, maps it to a concept (or creates a new one), and outputs the resulting IDs.
 
+**Capture refuses what it cannot understand** (do-no-harm: a skipped capture beats a polluted vault):
+- **Unfetchable / thin pages** (login walls, JS-only SPAs, dead newsletter email links) are refused with a hint, never classified from nav cruft or the bare URL.
+- **Newsletters / digests / archive indexes** are refused as single artifacts — they are containers of many items, not one piece of evidence. The message tells you to capture the individual items (or, for Substack, see below).
+- **A pasted Substack landing/archive URL auto-expands**: capture lists the per-article permalinks so you can capture the ones you want, or `--follow-repos`/`--expand-all` act on them. Each captured item records provenance (`source_url`, and `via` = the newsletter it came from).
+- **Single articles that introduce a concept** may link a reference implementation repo (e.g. ghuntley.com/ralph → repomirror). Capture *suggests* it; `--follow-repos` captures it as a sibling under the same concept with signal tracking.
+
+### Newsletters — recurring subscriptions (Substack)
+```bash
+raidar newsletter subscribe "<substack-url>"              # watch forward (baseline current posts)
+raidar newsletter subscribe --catch-up 3 "<substack-url>" # also capture the latest 3 now
+raidar newsletter list                                    # show subscriptions
+raidar newsletter poll                                    # capture every not-yet-seen post (runs weekly via launchd)
+raidar newsletter poll --dry-run                          # show what's new without capturing
+raidar newsletter unsubscribe "<url>"
+```
+`poll` is the recurring entry point (scheduled Sundays 19:00 by `install-launchd`). It is cadence-agnostic — "new" is decided by canonical post URL, so weekly/monthly/irregular newsletters are all handled. Only Substack publications can be subscribed today; other newsletter forms are refused at subscribe time. For a one-off non-Substack article, capture its individual URL directly.
+
+### Newsletters — HTML email digests (AlphaSignal, Beehiiv, Mailchimp, …)
+Many newsletters arrive as HTML email, not a fetchable web page. Save the message as a `.eml` file and ingest it:
+```bash
+raidar newsletter ingest "<file.eml>"            # harvest the GitHub repos it surfaces; list the rest
+raidar newsletter ingest --dry-run "<file.eml>"  # show the decomposition, capture nothing
+cat message.eml | raidar newsletter ingest -     # or pipe raw email on stdin
+```
+Ingest parses the email, uses the LLM to drop ads/sponsors and navigation **before resolving any link** (so sponsor trackers are never clicked), follows the remaining click-tracking redirects to their real destinations, then: **auto-captures the GitHub repos** (with `via` = the newsletter as provenance) and prints a **review list** of everything else (HuggingFace models, articles) with a ready-to-run `raidar capture "<url>"` line for each — because raidar has no artifact type for models/papers yet, those are never auto-written (do-no-harm).
+
 ### Search — query the vault
 ```bash
 raidar search keyword "<query>"             # frontmatter substring match (both layers)

infra/install_launchd.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Install AI Radar launchd agents (enrich + digest).
+# Install AI Radar launchd agents (newsletters + enrich + digest).
 #
 # Safeguard: checks if the OS is macOS (Darwin).
 # Detects if a global `raidar` binary is available in PATH. If so, templates
@@ -27,7 +27,7 @@ TOOL_DIR="$(cd "$(dirname "$0")/.." && pwd)"
 UV_PATH="$(command -v uv || true)"
 RAIDAR_PATH="$(command -v raidar || true)"
 LA_DIR="$HOME/Library/LaunchAgents"
-PLISTS=("com.airadar.enrich.plist" "com.airadar.digest.plist")
+PLISTS=("com.airadar.enrich.plist" "com.airadar.digest.plist" "com.airadar.newsletters.plist")
 
 if [[ -z "${UV_PATH}" ]]; then
     echo "error: uv not found in PATH" >&2
@@ -78,19 +78,25 @@ install() {
         label="${plist%.plist}"
         echo "installing ${label}"
 
-        # Build execution arguments
-        if [[ "${label}" == "com.airadar.enrich" ]]; then
-            if [[ -n "${RAIDAR_PATH}" ]]; then
-                exec_args="        <string>${RAIDAR_PATH}</string>\n        <string>enrich</string>"
-            else
-                exec_args="        <string>${UV_PATH}</string>\n        <string>run</string>\n        <string>raidar</string>\n        <string>enrich</string>"
-            fi
+        # Map the plist label to its raidar subcommand(s).
+        case "${label}" in
+            com.airadar.enrich)      sub="enrich" ;;
+            com.airadar.digest)      sub="digest" ;;
+            com.airadar.newsletters) sub="newsletter poll" ;;
+            *) echo "error: unknown plist label ${label}" >&2; exit 1 ;;
+        esac
+
+        # One <string> per subcommand token (handles multi-token like "newsletter poll").
+        sub_strings=""
+        for tok in ${sub}; do
+            sub_strings="${sub_strings}\n        <string>${tok}</string>"
+        done
+
+        # Build execution arguments: prefer the global raidar, else uv-run fallback.
+        if [[ -n "${RAIDAR_PATH}" ]]; then
+            exec_args="        <string>${RAIDAR_PATH}</string>${sub_strings}"
         else
-            if [[ -n "${RAIDAR_PATH}" ]]; then
-                exec_args="        <string>${RAIDAR_PATH}</string>\n        <string>digest</string>"
-            else
-                exec_args="        <string>${UV_PATH}</string>\n        <string>run</string>\n        <string>raidar</string>\n        <string>digest</string>"
-            fi
+            exec_args="        <string>${UV_PATH}</string>\n        <string>run</string>\n        <string>raidar</string>${sub_strings}"
         fi
 
         # Bootout any existing instance first so updates take effect.

infra/launchd/com.airadar.newsletters.plist

@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<!--
+  AI Radar — weekly newsletter poll.
+  Fires Sundays at 19:00 local time, before enrich (20:00) and digest (21:00),
+  so any new editions are captured into the vault ahead of the weekly passes.
+  The poll is cadence-agnostic: it captures whatever is unseen, so a weekly,
+  monthly, or irregular newsletter is all handled by this single fixed schedule.
+  install_launchd.sh substitutes the placeholders before copying to
+  ~/Library/LaunchAgents.
+-->
+<plist version="1.0">
+<dict>
+    <key>Label</key>
+    <string>com.airadar.newsletters</string>
+
+    <key>ProgramArguments</key>
+    <array>
+        <string>__PROGRAM_ARGUMENTS__</string>
+    </array>
+
+    <key>WorkingDirectory</key>
+    <string>__WORKING_DIRECTORY__</string>
+
+    <key>EnvironmentVariables</key>
+    <dict>
+        <key>PATH</key>
+        <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
+    </dict>
+
+    <key>StartCalendarInterval</key>
+    <dict>
+        <key>Weekday</key>
+        <integer>0</integer>
+        <key>Hour</key>
+        <integer>19</integer>
+        <key>Minute</key>
+        <integer>0</integer>
+    </dict>
+
+    <key>RunAtLoad</key>
+    <false/>
+
+    <key>StandardOutPath</key>
+    <string>__LOG_DIR__/launchd.newsletters.out.log</string>
+    <key>StandardErrorPath</key>
+    <string>__LOG_DIR__/launchd.newsletters.err.log</string>
+</dict>
+</plist>

infra/smoke.sh

@@ -61,13 +61,21 @@ check "lib.llm imports"   uv run python -c "from lib import llm"
 check "lib.embeddings imports" uv run python -c "from lib import embeddings"
 check "lib.github imports" uv run python -c "from lib import github"
 check "lib.body imports" uv run python -c "from lib import body"
-check "all jobs import"   uv run python -c "from jobs import capture, bulk_capture, enrich, digest, search, backfill, reevaluate, seed, cli"
+check "lib.webextract imports" uv run python -c "from lib import webextract"
+check "lib.substack imports" uv run python -c "from lib import substack"
+check "lib.subscriptions imports" uv run python -c "from lib import subscriptions"
+check "lib.email_digest imports" uv run python -c "from lib import email_digest"
+check "all jobs import"   uv run python -c "from jobs import capture, bulk_capture, enrich, digest, search, backfill, reevaluate, seed, newsletter, cli"
 
 echo "[lib smoke tests]"
 check "lib.vault smoke" uv run python -m lib.vault
 check_skip_ok "lib.embeddings smoke" uv run python -m lib.embeddings
 check_skip_ok "lib.llm smoke" uv run python -m lib.llm
 check "lib.github smoke (uses network)" uv run python -m lib.github
+check_skip_ok "lib.webextract smoke (uses network)" uv run python -m lib.webextract
+check_skip_ok "lib.substack smoke (uses network)" uv run python -m lib.substack
+check "lib.subscriptions smoke" uv run python -m lib.subscriptions
+check "lib.email_digest smoke" uv run python -m lib.email_digest
 
 echo "[CLI wiring]"
 check "raidar --help"  uv run raidar --help
@@ -84,6 +92,9 @@ check "merge-concept --help" uv run python -m jobs.merge --help
 check "rename-concept --help" uv run python -m jobs.rename --help
 check "reindex --help" uv run python -m jobs.reindex --help
 check "install-launchd --help" uv run python -m jobs.launchd --help
+check "newsletter --help" uv run python -m jobs.newsletter --help
+check "newsletter poll --help" uv run python -m jobs.newsletter poll --help
+check "newsletter ingest --help" uv run python -m jobs.newsletter ingest --help
 check "search list-concepts against empty vault" uv run python -m jobs.search list-concepts
 check "health against empty vault" uv run python -m jobs.health
 
@@ -93,6 +104,7 @@ check "capture --dry-run on free text" \
 
 echo "[infra]"
 check "install_launchd.sh syntax" bash -n infra/install_launchd.sh
+check "newsletters.plist validates" plutil -lint infra/launchd/com.airadar.newsletters.plist
 check "enrich.plist validates"   plutil -lint infra/launchd/com.airadar.enrich.plist
 check "digest.plist validates"   plutil -lint infra/launchd/com.airadar.digest.plist
 

jobs/bulk_capture.py

@@ -59,6 +59,37 @@ def _extract_github_slugs(text: str) -> list[str]:
     return result
 
 
+def _link_list_shape(slugs: list[str]) -> tuple[int, int, float]:
+    """Return (n_repos, n_owners, owner_diversity) for a set of owner/name slugs."""
+    n_repos = len(slugs)
+    owners = {s.split("/", 1)[0].lower() for s in slugs}
+    n_owners = len(owners)
+    diversity = (n_owners / n_repos) if n_repos else 0.0
+    return n_repos, n_owners, diversity
+
+
+def _looks_like_awesome_list(
+    slugs: list[str], *, min_repos: int, min_owners: int, min_diversity: float
+) -> tuple[bool, str]:
+    """Decide whether a page's GitHub links look like a curated awesome-list.
+
+    bulk-capture is correct for awesome-lists (links ARE the content) and harmful
+    for newsletters/announcements (the links are a publisher self-promo footer or
+    an incidental partner mention — verified on the issue #2 examples). The robust
+    separators are repo COUNT (newsletter footers carry a handful; lists carry
+    dozens) and owner spread (a single org's repo dump is not a curated list).
+    """
+    n_repos, n_owners, diversity = _link_list_shape(slugs)
+    stats = f"{n_repos} repo link(s) from {n_owners} owner(s), diversity {diversity:.2f}"
+    if n_repos < min_repos:
+        return False, f"{stats} — too few links to be an awesome-list (need ≥{min_repos})"
+    if n_owners < min_owners:
+        return False, f"{stats} — too few distinct owners (need ≥{min_owners})"
+    if diversity < min_diversity:
+        return False, f"{stats} — one owner dominates (need diversity ≥{min_diversity})"
+    return True, stats
+
+
 def _fetch_source_text(url: str) -> str | None:
     """Return raw text for slug extraction.
 
@@ -86,7 +117,7 @@ def bulk(
     url: Annotated[str, typer.Argument(help="GitHub awesome-list URL or any web page.")],
     dry_run: bool = typer.Option(False, "--dry-run", help="Show what would be captured without writing."),
     limit: int | None = typer.Option(None, "--limit", help="Max repos to process (default: all)."),
-    force: bool = typer.Option(False, "--force", help="Bypass soft dedup on each item."),
+    force: bool = typer.Option(False, "--force", help="Bypass the awesome-list shape gate and soft dedup on each item."),
 ) -> None:
     """Bulk-capture repos extracted from an awesome-list or web page."""
     cfg = config.load()
@@ -103,6 +134,24 @@ def bulk(
         print("No GitHub repo links found in the page.")
         raise typer.Exit(code=0)
 
+    # Shape gate: only blanket-capture pages that look like a curated awesome-list.
+    # Newsletters/announcements carry a few publisher-owned or incidental links that
+    # would pollute the vault if ingested wholesale. --force overrides.
+    ok, stats = _looks_like_awesome_list(
+        slugs,
+        min_repos=int(cfg.thresholds.get("bulk_min_repos", 10)),
+        min_owners=int(cfg.thresholds.get("bulk_min_owners", 3)),
+        min_diversity=float(cfg.thresholds.get("bulk_min_owner_diversity", 0.3)),
+    )
+    if not ok and not force:
+        print(f"Refusing to bulk-capture: {stats}.")
+        print(
+            "bulk-capture is for awesome-list-shaped pages. For a newsletter or "
+            "single article, capture items individually (`raidar capture <url>`), "
+            "or pass --force to override."
+        )
+        raise typer.Exit(code=0)
+
     if limit is not None:
         slugs = slugs[:limit]