Decide on and test HTML capture expected outcomes

[janikmu]

Expected outcomes:

• [easy] If there are github links in the HTML, bulk capture them
• [hard] Decide whether the HTML itself is an artifact (not true for newsletters, true for blog posts, true for white papers, true for tech reports)
• [hard] Decide what to do with non-Github links
  • ignore (probably default to that at this point as it opens up unknown complexity)
  • follow (only for links that lead to pdf -> tech report / white paper? what about blog posts introduced by newsletters, this is the really hard part)

[janikmu]

Examples for blog posts that are artifacts (model announcements):

• https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/
• https://www.anthropic.com/news/claude-opus-4-7

Examples for blog posts that are artifacts that introduce(?) concepts:

• https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/ (QAT itself or a generalization of it should be a concept)
• the ralph loop pattern (or any pattern) is a good example for blog posts that introduces concepts: https://ghuntley.com/ralph/ probably stuff like https://github.com/kunchenguid/gnhf then implements ralph

[janikmu]

Examples for newsletters that house repo links, whitepapers, and everything else:
https://ethical.institute/mle/389.html
https://www.linkedin.com/newsletters/7324756381780140032/?displayConfirmation=true

Examples for newsletters that focus more on concepts, announcements:
https://alphasignal.ai/email/305ad305be75cdf5

Examples for newsletters that are a little bit of everything but might have interesting concepts / references:
https://refactoring.fm/s/essays

[janikmu]

HTML capture — scrape + analysis of the example URLs (part 1/2: findings)

Before writing any code I ran all 8 example URLs from this issue through raidar's actual extractor (trafilatura.fetch_url → trafilatura.extract, exactly what _fetch_web_text does in jobs/capture.py), plus an independent full fetch of each, plus targeted verification of the link-handling and fetch-reliability questions. This part reports what raidar genuinely sees today; part 2/2 proposes the roadmap.

Guiding doctrine for everything below: a skipped capture beats a polluted knowledge base. The analysis is biased toward "refuse what we can't model faithfully."

TL;DR

• Of the 8 examples, only 3 are faithfully capturable today (A2 Opus 4.7, B1 QAT post, B2 ghuntley/ralph), 1 with guardrails (A1 Gemma 4), and 4 must be skipped (C1, C2, D1, E1).
• The painful irony: Priority 1 (newsletters) is the least faithfully-capturable category today. Every newsletter/index example is currently a hard no — wrong shape (N items, not one artifact), wrong content (auth / JS / dead-token walls), and in 3 of 4 cases active KB pollution.
• The single most dangerous mechanism is the 100-char fallback floor (jobs/capture.py:189). C2 (281 chars of LinkedIn nav) and E1 (1035 chars of teaser titles + "requires JavaScript") both clear it, so raidar confidently fabricates a junk artifact+concept instead of cleanly skipping. Silent false-positive — the worst failure mode.
• The "[easy] if there are github links, bulk-capture them" rule is wrong for newsletters. Verified on C1: raidar's regex finds 6 repos, all clustered in one late-document block, 5 of 6 owned by the author himself. Blanket capture ingests the author's footer as if it were this week's signal.
• But there is a faithful path for newsletters — it's just not the landing/email URL, it's the per-article permalink. Verified: refactoring.fm/p/monday → 5034 clean chars (vs 1035 of soup on /s/essays). AlphaSignal is itself a Substack (alphasignalai.substack.com), reachable via an unauthenticated JSON API. This makes a real Prio-1 newsletter win achievable on the existing pipeline (see part 2).
• A headless renderer (Playwright) is not warranted now — where content is reachable it's already server-rendered; the rest is hard auth that rendering can't bypass.

What raidar actually sees today

#	URL	Archetype	trafilatura	GitHub links it finds	What capture writes today	Pollutes KB?
A2	anthropic.com/news/claude-opus-4-7	model announcement	excellent (18 205 ch)	none	1 post/release → frontier-LLM concept	No
B1	blog.google …/quantization-aware-training-gemma-4	single-article post	excellent (4 680 ch)	none	1 post → quantization/QAT concept	No
B2	ghuntley.com/ralph	single-article post	excellent (27 797 ch incl. code)	1 — content (repomirrorhq/repomirror, in body)	1 post → agent-loop concept (misses the 1 repo as sibling)	No (lossy)
A1	blog.google …/gemma-4	model announcement	excellent (8 266 ch)	1 — boilerplate (NVIDIA-NeMo/Automodel, incidental partner doc)	1 post/release; bulk-capture would mint a misleading NeMo artifact	Lossy via capture; harmful via bulk-capture
C1	ethical.institute/mle/389.html	newsletter digest	degraded (6 974 ch, mashed blob)	6 — all boilerplate (doc-pos 0.70–0.74; 5/6 author-owned)	Forces ~5 items → 1 artifact: umbrella concept or items dropped	Yes
C2	linkedin.com/newsletters/…	archive index (auth-walled)	failed (281 ch nav cruft)	none	Fabricates artifact+concept from nav garbage (clears 100-ch floor)	Yes
D1	alphasignal.ai/email/<token>	newsletter (dead token)	failed (HTTP 403)	none	Falls back to text mode → classifies the bare URL string	Yes
E1	refactoring.fm/s/essays	archive index (JS)	degraded (1 035 ch title soup)	none	Classifies essay titles + "requires JavaScript" → bogus post+concept	Yes

Read of the table: every clean win is a single, server-rendered, self-contained article. Every pollution case is either a digest (N→1 impedance mismatch) or a fetch wall whose thin output slips past the length floor. The GitHub-link signal is unreliable — it is content in exactly one case (B2), incidental in another (A1), and publisher self-promo in the newsletter (C1).

The C1 GitHub-links finding, verified deterministically

raidar's regex finds these 6 slugs in the trafilatura output of issue #389:

EthicalML/awesome-production-machine-learning, besanson/sarc-governance, axsaucedo/kaos, KomputeProject/kompute, EthicalML/awesome-artificial-intelligence-regulation, EthicalML/awesome-production-genai

Their byte positions in the raw HTML all fall at document-fraction 0.70–0.74 — a single late block, not distributed through the prose. Five of the six are the author's own (the newsletter is by Alejandro Saucedo; EthicalML/*, axsaucedo/*, KomputeProject/* are all his). Only besanson/sarc-governance is third-party — and it sits in the same block.

Consequence for the link-handling design: a "most-frequent-owner" self-promo filter catches only the 3 EthicalML repos (it misses axsaucedo and KomputeProject, both 1× but the same human). So document position is the robust signal here, owner-frequency is leaky.

Answers to the issue's three questions, grounded in the data

(a) Is the HTML page itself an artifact? → it depends on archetype, and the split is clean

• YES — the page is the artifact: single-article posts (B1, B2) and self-contained announcements (A2). Evidence lives on the page; post/release fit exactly; one page → one concept, no fan-out. This is what default capture was built for.
• YES, but the thing it describes lives elsewhere (pointer/landing): A1. The page is a real post, but the model it announces canonically lives on HuggingFace/Kaggle/Ollama/ai.google.dev — artifact kinds raidar can't represent. Capturable as a post, lossy as the release.
• NO — the page is a container/feed: newsletter digests (C1, D1) and archive indexes (C2, E1). A digest is N independent items across N would-be concepts; an index is navigation. Mapping either to one concept forces a doctrine-violating umbrella ("ml-newsletters") or drops most of the content. The durable artifacts are the items it points to, not the digest URL — which is also ephemeral (next week it's issue #390).

(b) GitHub links → blanket bulk-capture is wrong for newsletters; here is the minimal safe rule

Blanket bulk-capture (jobs/bulk_capture.py:_extract_github_slugs) regex-matches every github.com/owner/repo with no DOM/position/owner filtering. Correct for an awesome-list, harmful for a newsletter:

• Awesome-list (e2b-dev/awesome-ai-agents): ~120 slugs, high owner diversity, the links are the content. Blanket capture is right.
• Newsletter (C1): 6 slugs, one block, 5/6 author-owned. Blanket capture mints off-target repo artifacts unrelated to the issue's content.
• Announcement (A1): 1 slug, an incidental partner doc. Blanket capture creates a misleading artifact.

Minimal safe rule (cheap, no extra fetches, in this order):

1. Owner-diversity gate — bulk-capture only when distinct_owners ≥ 3 and distinct_owners / distinct_repos ≥ 0.6. The awesome-list passes trivially; the self-owned newsletter fails. (~10 lines; owners already parse via lib.github.parse_repo_url.)
2. Position / block filter — drop links that cluster in a single late block or in nav/footer regions (the robust signal for C1, where owner-frequency leaks). Needs raw HTML rather than trafilatura's link-flattened extract.
3. Drop-self-promo — exclude slugs whose owner matches the page's home owner (domain or most-frequent owner) as a cheap complement to (2).

Framing: keep bulk-capture an explicit, awesome-list-only command. During normal single-URL capture, surface in-body GitHub links as suggestions (review_needed) — never auto-bulk-capture from arbitrary HTML.

(c) Non-GitHub links → default ignore; follow only a narrow, typed subset later

Across every example the non-GitHub links are either boilerplate to ignore (nav/footer/social/subscribe chrome) or genuine signal raidar cannot model (HuggingFace model/dataset pages, Kaggle/Ollama/Docker, arXiv/PDF). The brutal finding from A1 and C1: the most load-bearing links on a page are precisely the ones raidar is blind to (where the model lives; the per-item primaries of a digest). Following them today manufactures unclassifiable junk.

Default: ignore. Future relaxation (gated behind --follow-links): PDFs → paper, known blog hosts → post only — everything else waits for first-class non-GitHub artifact types. Record unfollowed primaries as plain references inside the artifact body so provenance survives without minting junk.

The page-archetype taxonomy (final)

The core split (page-is-artifact vs page-is-container) is sound, but it really rides on three axes: (A) keep the page as evidence? (B) does it point to a canonical artifact elsewhere? (C) does it enumerate N children to harvest? With that lens:

Archetype	Page is…	Example(s)	Faithful today?
single-article post	artifact	B1, B2	YES
model / release announcement	artifact (+ pointer)	A2	YES
pointer / landing page	pointer to off-page artifact	A1; arXiv /abs; docs landing	GUARDRAILS (capture text, record off-page target, block product-named concepts)
newsletter digest	container of N items	C1, D1	NO → faithful only via per-item / Substack expansion
archive / index / feed	navigation surface	C2, E1	NO (skip; not one artifact)
dual: artifact AND container	both	awesome-list that is a repo; concept-introducing newsletter	DEFER (needs both verbs in one pass)
GitHub release page	event vs index	/releases/tag/X vs /releases	GUARDRAILS / DEFER (no "update existing artifact" path)
docs / API reference	living multi-page tree	framework docs	DEFER (attach-to-concept, not its own artifact)
discussion / thread	essay or link-container	X / HN thread	UNSUPPORTED (extraction fails)
media	audio/video + transcript	YouTube, podcast	UNSUPPORTED (needs transcript fetch)

The newsletter problem specifically (Priority 1)

This is where goal and capability are furthest apart, so it deserves explicit sequencing.

Why the examples fail today: (1) wrong shape — a digest is N independent artifacts but raidar's atom is 1 URL → 1 artifact → 1 concept; (2) fetch walls — D1's per-recipient token is dead (403), C2's LinkedIn landing is auth-walled, E1's Substack index is JS-gated; (3) active pollution — C2/E1 clear the floor and get classified, D1 classifies the bare URL string; (4) inverted link signal — the GitHub links raidar sees are publisher self-promo, while the real per-item signal lives in non-GitHub links that extract() discards.

The minimum reliable ingest unit is the per-article permalink, not the landing/email URL — verified against raidar's own trafilatura:

Platform	Faithful unit	Evidence	Not this
Substack (covers refactoring.fm and AlphaSignal)	https://{pub}/p/{slug}	refactoring.fm/p/monday → 5034 ch	/s/essays index → 1035 ch soup
AlphaSignal	its Substack mirror alphasignalai.substack.com/p/{slug}	API returns body_html 26 KB / 22 KB	alphasignal.ai/email/<token> → 403, no faithful path
LinkedIn	https://www.linkedin.com/pulse/{slug} (guest view)	~6979 ch body	/newsletters/{id}/ landing → auth wall

Substack also exposes an unauthenticated discovery API: https://{pub}/api/v1/posts returns each post's canonical_url (verified — returns refactoring.fm/p/monday etc.). So a pasted Substack landing/archive/email-mirror link can be expanded into per-post URLs that the existing pipeline already captures faithfully. That is the tractable Prio-1 win (part 2, Phase 2). LinkedIn has no such API and applies anti-bot walls after a few views — fine for occasional single-shot capture, not for polling.

---

Part 2/2 (proposed phased roadmap + open questions to agree on before implementation) follows in the next comment.

[janikmu]

HTML capture — proposed roadmap (part 2/2)

Built on the findings in part 1. Ordered to honor Priority 1 = newsletters while keeping the doctrine first: nothing that can pollute the vault ships before the guardrails that stop it. Each phase says what it does, why it's safe, rough effort, and — importantly — what it explicitly does not do. These are proposals for us to agree on; I haven't written any code.

Phase 0 — Do-no-harm extraction-quality gate (do first; best value/effort ratio)

• What: Replace the crude 100-char floor in _fetch_web_text (jobs/capture.py:184-191) with a confidence gate: raise the floor and detect wall sentinels ("This site requires JavaScript", nav-domination like repeated "Skip to main content" / repeated author name, subscribe-wall markers) → treat as fetch failure. Make the web→text fallback (capture.py:486-494) a hard skip for URL inputs: never feed a bare URL string to the LLM as content (raise CaptureSkipped("fetch produced no usable text")).
• Why safe: Purely subtractive — it can only prevent writes. Directly kills the C2 / D1 / E1 pollution cases.
• Effort: Small. Does NOT: add archetype detection, touch bulk-capture, or add capability.

Phase 1 — Archetype routing + bulk-capture safety gates (stop ingesting junk)

• What (capture): Cheap heuristics to detect newsletter/digest and archive/index pages and refuse single-artifact capture with a hint ("looks like a digest/index — capture the individual items instead"). Signals: host patterns (linkedin.com/newsletters, alphasignal.ai/email, Substack archive/index, /\d+\.html issue paths, Brevo/Mailchimp/beehiiv footers), many disjoint outbound links across mixed kinds, subscribe/subscriber-count text, JS-wall sentinel.
• What (bulk-capture): Add the owner-diversity gate (min_owners=3, diversity≥0.6) and position/self-promo filter from part 1(b), config-driven via cfg.thresholds. Add doc-path slugs (…/docs/….md) to the non-repo skip set.
• Why safe: Converts the skip-cases from "silently pollute" to "explicitly refuse"; bulk-capture stops eating publisher footers (C1) and incidental partner docs (A1).
• Effort: Small–medium. Does NOT: decompose newsletters, follow links, or add artifact types.

Phase 2 — Substack-aware newsletter ingestion (the tractable Priority-1 win; verified faithful)

• What: When a pasted URL is a Substack landing / archive / /email mirror, expand it into per-post permalinks via the unauthenticated /{pub}/api/v1/posts API (canonical_url per post), then capture each /{pub}/p/{slug} through the existing, unchanged pipeline (each per-post page extracts cleanly — verified 5 KB on refactoring.fm). Map alphasignal.ai → alphasignalai.substack.com. Likely surfaced as a confirm-list ("found N posts — capture which?") rather than auto-ingesting all.
• Why safe & high-value: Reuses the already-faithful single-article path; no new extraction risk. Covers two of the four Prio-1 newsletter examples (refactoring.fm and AlphaSignal) with real, clean captures — turning "skip" into "ingest correctly."
• Effort: Medium. Does NOT: handle non-Substack digests, LinkedIn, or heterogeneous (non-blog) items — those are Phase 4.

Phase 3 — Single-article in-body repo follow (the ghuntley/ralph case)

• What: For confirmed single-article posts (Phase 1), when the body links 1–2 in-body GitHub repos, also capture them as sibling artifacts under the same concept (so the repo gets signal tracking and the concept shows post + repo as alternative evidence — exactly B2's repomirrorhq/repomirror). Gated by the owner/position filters so A1's incidental NeMo link and C1's footer are excluded; borderline links → review_needed.
• Why safe: Fires only on the clean shape, behind Phase-1 archetype + Phase-1 link gates.
• Effort: Medium. Does NOT: run blanket bulk-capture, follow non-GitHub links, or touch newsletters.

Phase 4 — DEFER: general digest decomposition + LinkedIn + heterogeneous items

• What: LLM-segment an arbitrary (non-Substack) newsletter body into per-item candidates (title + primary link + kind), let the user pick; LinkedIn /pulse/{slug} best-effort single-shot. Depends on item-type support (Phase 5) to be useful, since most digest primaries are non-GitHub.
• Why deferred: Highest complexity, lowest faithfulness today; Phases 0–2 already keep the KB clean and give a working newsletter path. Does NOT (when built): auto-capture all items, pass /feed to extract(), or bulk-poll LinkedIn.

Phase 5 — DEFER (separate track): non-GitHub artifact types & signals

• What: First-class artifact types for HuggingFace / Ollama / Kaggle model releases and arXiv papers, signals beyond GitHub stars/forks, and arXiv /abs == /pdf == html canonicalization. This is what A1 (Gemma 4 lives on HF/Kaggle/Ollama) and C1's dataset link actually need.
• Why deferred: A schema-level change to the data model, out of scope for "decide HTML expected outcomes." A1 is the canonical motivating example to cite when this is prioritized. Does NOT: belong in the HTML-capture milestone.

Suggested acceptance test (taxonomy completeness)

Run the issue's example URLs plus one arXiv /abs, a GitHub /releases/tag, an HN thread, and a YouTube link through capture --dry-run and bulk-capture --dry-run; assert each lands in its intended bucket with a faithful — or explicitly-skipped-with-reason — outcome. Specifically assert: A2 → exactly one post/release and no spurious repo artifact; C1/C2/D1/E1 → skipped (or, for Substack, expanded to per-post); A1 bulk-capture → captures nothing.

---

Open questions to agree on before implementation

1. Newsletter UX (Prio 1): Is the Phase-2 model — paste a Substack landing/email link, we expand to per-post and you pick which to capture — the right shape? Or do you want full auto-capture of all new posts (drifts toward a subscription/feed-monitor, closer to the enrich job than capture)?
2. Headless renderer: Confirm we are not adding Playwright for this milestone (it buys near-zero faithful coverage). Agree to defer indefinitely?
3. Recording the newsletter as provenance: When we capture a per-item artifact discovered via a newsletter, how should the newsletter itself be recorded — a non-artifact "source/feed" entity to monitor, a provenance/via field on the item, or not at all? (No "source" notion exists in the data model today.)
4. Bulk-capture scope: Confirm bulk-capture stays an explicit, awesome-list-only command and the "[easy] if-github-then-bulk" auto-rule is downgraded to review_needed suggestions during normal capture.
5. Gate thresholds: OK to start with min_owners=3, diversity≥0.6 (config-driven)? These pass e2b-dev/awesome-ai-agents and fail ethical.institute in the verified data.
6. Scope of this milestone: Do you want this issue to deliver Phases 0–3 (safety + Substack + ghuntley-style follow) and spin Phases 4–5 into their own issues? That matches "do no harm" and gives a faithful Prio-1 newsletter win without the schema change.

Once we settle these — especially Q1, Q3, and Q6 — I'll start implementing, Phase 0 first.

[janikmu]

Agreed roadmap — decisions locked

Following the analysis above, the decisions for this milestone:

• Scope: implement Phases 0–3 here (do-no-harm extraction gate → archetype routing + bulk-capture gates → Substack newsletter ingestion → single-article in-body repo follow). Phase 4 (general non-Substack digest decomposition + LinkedIn) and Phase 5 (non-GitHub artifact types & signals: HF/Ollama/Kaggle/arXiv) spin out into their own issues — no data-model change in this milestone.
• Newsletter UX — two modes (Phase 2 covers both):
  • Manual capture of a Substack-shaped URL (landing / archive / /email mirror) → expand to per-post permalinks and pick which to capture.
  • Registered subscription → a newsletter deliberately added to the recurring job, which auto-captures new posts it hasn't seen.
• Provenance: lightweight via / source_url field on each captured artifact recording which newsletter surfaced it. Mode-b subscriptions live in a small list (config-level), not a first-class vault "source" entity.
• Headless renderer (Playwright): deferred indefinitely — verified to buy near-zero faithful coverage.

Starting implementation with Phase 0 (extraction-quality gate; purely subtractive safety) on a feature branch.