Make the quickstart finish, and stop eval scoring its own template

[sahilkalgutkar]

Part of the audit that began with sahilkalgutkar/modelforge#18 — looking for things CI never runs. I ran this repo's quickstart top to bottom on a clean clone. Four problems, none of which CI can see, because CI runs pytest, not the README.

1. One of the two default feeds is dead

https://www.anthropic.com/news/rss.xml returns 404, and Anthropic no longer publishes an RSS feed at any path I could find. feedparser reports that as a clean parse with zero entries, so nothing failed — half the corpus was just silently missing. Swapped for https://github.blog/changelog/feed/, which is a genuine product changelog and fits what config/feeds.yaml already recommends. poll now prints when a feed returns nothing, with its HTTP status.

2. The first documented command ran for seventeen minutes with no output

python -m ingest.poll took every entry a feed offered, and the OpenAI blog publishes its entire history in one document — 1161 entries, each costing a full trafilatura page fetch. The very first command in the quickstart therefore ran silently for about seventeen minutes before printing anything. It is not hung, but there is no way to tell that while it is happening.

Added max_articles_per_feed (default 25, newest first, 0 for no limit) and a per-feed line as it goes:

$ python -m ingest.poll
github-changelog: 10 new from 10 entries
openai-blog: 25 new from 25 entries
Ingested 35 new articles -> data/articles.jsonl

26.4s total

The cap is applied before fetching, so an entry past the limit is never downloaded at all — that is what the test asserts, rather than just checking the row count.

3. eval.run scored its own placeholder

eval/questions.yaml ships as a template with expected_article_id: "REPLACE_WITH_REAL_ARTICLE_ID". Running the documented command against it produced:

[MISS] What did <source> announce in <article title>?

recall@5: 0/1 (0%)

That is a real-looking number that measures nothing and reads as a broken retriever. eval.run now refuses to score an unfilled set, says how to fill it, and exits 1 without running a search.

The README also said "I hand-built a question/answer set to measure retrieval recall and citation correctness" — there is no such set in the repo, and there is a good reason there isn't: an expected_article_id is only meaningful against the feeds it came from, so shipping mine would leave every id dead for anyone pointing the bot at a different beat. The README now says that instead of claiming otherwise.

The guard does not get in the way of real use — with three questions filled in from an actual index, eval.run reports recall@5: 3/3 (100%) and exits 0.

4. A wrong comment in the quickstart

cp .env.example .env # add your feeds + API key — .env holds only ANTHROPIC_API_KEY; feeds live in config/feeds.yaml.

Verification

ruff check clean; 41 tests pass (8 new) at 96% coverage. Ran the full quickstart afterwards on a clean clone: poll, index build, and eval both refusing the template and scoring a filled-in set.

[codecov]

Codecov Report

❌ Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
eval/run.py	93.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!