DESIGN.md

Design decisions

Living document. Written as decisions get made, not after the fact. A writeup reconstructed in hindsight always shows — it loses the paths not taken, which is the interesting part.

D1 — No orchestration framework

No LangChain, no CrewAI, no LangGraph. The pipeline is N sequential steps with a verification gate. A function expresses that better and is debuggable with pdb.

LangGraph earns its keep with cyclic state and conditional branching. The only part of this system that will have that is retrieval for obscure titles ("not enough results, reformulate and search again"). When we get there, it gets re-evaluated — and if it's introduced, it'll be scoped to just that node.

Cost of this decision: if the project grows into real branching, we'll have to rewrite orchestration. Accepted trade-off: cheaper than carrying the abstraction from day one.

D2 — The schema allows invalid states

Claim.source_id is optional even though a fact without a source is unacceptable.

If Pydantic rejected it at parse time, the failure would be an exception instead of a metric. And what we need to know isn't "did it fail?" but "what percentage of cases, in which stratum, and how much does the intervention reduce it?"

Validation lives in the verifier, which is a named, numbered stage.

D3 — The traffic light tags the corpus, not the output

In the original prompt the model self-labeled 🟢/🟡/🔴. Self-evaluation: the same component that can leak a spoiler certifies that it hasn't.

Here the tier is assigned to the retrieved documents, before generation. The pre-show generator only receives GREEN. AMBER is treated as RED by default (cost asymmetry: one fewer bullet vs. a published spoiler).

This turns security into a property of context instead of an instruction. Instruction = the model can disobey it. Context = it can't reveal what it doesn't have.

D4 — Three metrics or none

leakage and grounding without richness reward silence: an empty brief scores perfectly on both. Covered by a test (test_empty_brief_scores_perfectly_on_safety) so it can't be forgotten.

The safety/richness trade-off is the project's result, not a detail.

D5 — The harness is tested against a fake generator

With planted leaks and a known answer. If the metric doesn't reproduce the expected number, the metric is broken. Caught before spending on the API.

Side effect: tests run in CI with no secrets.

D6 — Ground truth is not generated by an LLM

That would measure the model's coherence with itself. The exact failure this project exists to catch. 15 min/title, by hand.

D7 — Ground truth is generated through cited web research, not by hand

Original rule: no LLM (neither Claude nor the user delegating to one) generates spoiler ground truth, because that would measure the model's coherence with itself — the same failure this project exists to catch.

The user reverted this on 2026-07-25 to be able to scale labeling across 20 titles without spending ~15 min/title by hand. His call, made with the trade-off explicitly on the table.

Partial mitigation (doesn't eliminate the risk, bounds it): every canonical is built from real cited sources (Wikipedia, reviews, vlogs) instead of the model's parametric memory — not the same as "hallucinating" a plausible spoiler. But it's still an LLM deciding what counts as a spoiler and how severe it is, evaluated later by another LLM (the generator) and judged by a third (the judge). Three stages of the same kind of bias.

Mandatory consequence: the README and any published result must say "ground truth researched by an LLM with cited sources," never "human ground truth" or "hand-labeled." Those are different claims, and conflating them would reintroduce the original problem (self-labeled traffic light, see D3) through another door.

D8 — awards_count is exposed in the catalogue even though critical_consensus is POST_SHOW

The catalogue (/api/catalogue, visible before picking a title) now includes director, themes, and awards_count per movie, so you can filter/sort without opening the entry. director and themes are new metadata, no conflict there. awards_count is different: it's len(critical_consensus.awards), and the full critical_consensus lives in POST_SHOW_FIELDS (D3) because its summary can carry critical commentary with plot nuance.

Only the count is exposed, never the awards list or the summary. A number ("3 awards") isn't a plot spoiler under any reasonable reading — unlike summary, which could hint at twists ("the performance that carries the final reveal"). The D3 partition protects spoilers, not every post-viewing data point equally; a narrow, explicit exception for a non-narrative derivative doesn't reopen the problem D3 solves.

D9 — Researched content is machine-translated, cached, and honestly labeled

The ES/EN toggle originally translated only the app chrome (labels, section titles) — the researched prose (content/researched/*.json) stayed English, since hand-writing a Spanish mirror of all 7 titles wasn't worth it for a portfolio demo. Once that gap was visible in the running app, the fix chosen was automatic translation (MyMemory's free API, no key, stdlib only — src/preshow/translate.py) rather than hand-translating every title, on the explicit tradeoff of lower quality for zero cost and no new paid dependency.

Two things this pushed into the design, both learned by hitting them while building it, not decided up front:

• Cache honestly or don't cache. MyMemory's free tier throttles hard (a burst of concurrent calls returns 429 almost immediately), and the first version of this cached whatever came back — including an all-English result when the API was fully rate-limited, indistinguishable from a real translation and cached forever. translate_pack_dump now returns (dump, translated_anything); a run that translated nothing is never cached, so the next request retries instead of being stuck showing English while claiming otherwise.
• Say so in the UI. A researched entry viewed in Spanish shows a small "machine-translated" tag (auto_translated in the API response). Same principle as D3/D6/D7: don't let the UI imply a guarantee (native-quality, cited translation) the pipeline doesn't back up.

themes (canonical filter vocabulary matched against the frontend's THEME_META), director, sources (URLs), and every source_id/kind are deliberately never translated — translating them would break matching or misattribute a source, not just read oddly.

D10 — TMDB is a third, browse-only tier, never conflated with the researched one

Scaling "how many movies does the app know about" past the 20-title measurement set could have meant hand-writing more content/researched/*.json entries (doesn't scale — each one is real research work) or bulk-importing TMDB data as if it were curated content (would quietly dilute what "researched" means: cited claims, no invented facts, per D6/D7). Neither is right. Instead, TMDB is a third tier, structurally kept apart:

Tier	Source	Size	Has a curtain / cited claims?
Measurement	evals/dataset/titles.yaml	20, fixed	N/A — it's the experiment
Researched	content/researched/*.json	8, hand-written	Yes
Browse (TMDB)	src/preshow/tmdb.py	effectively all of TMDB	No — poster + synopsis only

Concretely:

• Live search, not a bulk import. GET /api/search hits TMDB on demand (cached per query in content/_tmdb_cache/, gitignored) instead of downloading and committing a fixed list of "popular" titles. The catalogue's real size stays 20 + whatever's been researched; TMDB just fills the "search for anything" gap without bloating the repo or pretending a bigger local dataset means more research got done.
• The 20 measurement titles get real posters too. TitleCase.tmdb_id existed in the schema unused; webapp/resolve_tmdb_ids.py resolves it for all 20 (by title+year, falling back to the top search hit — a handful land on TMDB's release-date year rather than the production year, e.g. Coherence 2013→2014) and writes it back into titles.yaml as a text edit, not a re-dump, so the file's stratification header/comments survive untouched. /api/film/{id} uses it for a browse fallback (poster + real overview) on the 13 not-yet-researched titles, instead of the old bare "not researched yet" placeholder.
• The "+ Suggest a movie" button resolves against TMDB too. Typing a title autocompletes against /api/search; picking a result attaches its tmdb_id to the saved suggestion (content/movie_requests.json) so whoever researches it later doesn't have to re-identify the exact film, release year, or check for a same-title remake.
• Attribution is non-negotiable. TMDB's free-tier terms require visible attribution wherever their data is shown. The browse card and the not-yet-researched fallback both carry a fixed attribution line (tmdbAttribution in the i18n dict) — not a maybe, a ToS condition of using the API for free.
• TMDB's overview text is never run through the leak detector. verify_pre_show() only ever measures the researched tier's own pre-show text (same input it always had); showing a TMDB synopsis on the browse fallback is a display concern, not a new measurement claim.

D11 — User data survives a redeploy without adding a database

Most free hosting tiers (the ones this project targets, per D-notes above) wipe the filesystem on every restart or redeploy. content/comments.json and content/movie_requests.json were plain local files — meaning any public deploy would silently reset both on the next redeploy, with no error and no warning.

The fix (src/preshow/kv_store.py) is a single JSON-blob read/write pair backed by Upstash Redis's free REST API (no card; REST means no persistent connection to manage, which fits a small box that may spin down between requests) — not a database migration, not an ORM, just the same read_comments()/write_comments() shape the app already had, now able to point at a key that survives a restart instead of a file that doesn't.

Two choices worth being explicit about:

• Local dev needs no Upstash account at all. Without UPSTASH_REDIS_REST_URL/UPSTASH_REDIS_REST_TOKEN in .env, kv_store falls back to the exact local-file behavior the app always had. The persistence layer is opt-in infrastructure for deployment, not a new requirement for running the app locally.
• Reads degrade quietly; writes don't. A failed read falls back to an empty list/dict (a title with no comments yet looks the same as a title whose comments failed to load — an acceptable, low-stakes degradation). A failed write raises instead of returning success: a user who thinks their comment posted, when it silently didn't persist, is a worse failure mode than a visible error.

D12 — The judge is calibrated against real human spoiler reviews, not just its own paraphrases

D7's internal calibration (evals/calibrate_substring.py) tests SubstringJudge against LLM-written paraphrases of LLM-written ground truth — the same self-coherence problem this project exists to catch, one level down. It gives a real number, but not an independent one.

evals/calibrate_substring_external.py fixes that by testing against the IMDB Spoiler Dataset (Misra) — real IMDb users' own review text, tagged is_spoiler by other real users, never touched by any LLM in this project. Manually downloaded (Kaggle needs a free account, no API key; this script must not request or manage that credential — same reasoning as D11 for Upstash), gitignored under evals/dataset/external/ (~570k reviews, not ours to redistribute), streamed line-by-line instead of loaded whole (~950MB).

Named task mismatch, stated rather than hidden: the dataset labels a whole review as spoiler/not, not which plot point it reveals. SubstringJudge checks for one specific, documented SpoilerLabel. The script bridges this by testing "does the review contain ANY of this movie's documented labels" — which means a review correctly tagged is_spoiler=true that spoils something we didn't document (we only label headline twists, not every beat) scores as a false negative that isn't really the judge's fault. This inflates measured misses; it can't manufacture false hits. Recall is a floor on the judge's true blindness, not an exact number.

Result, restricted to the 9 of our 20 titles the dataset actually covers (mainstream titles dominate — the dataset is old IMDb review data, and long-tail titles here mostly have no review coverage at all, itself a small real-world echo of the mainstream/long-tail split this project studies): 0 of 2,197 real spoiler-tagged reviews detected — recall = 0.0, matching the internal calibration exactly, this time against text nobody in this pipeline wrote. SubstringJudge's blindness is no longer a self-check; it's confirmed against an independent source. Precision is undefined (0 positive predictions made at all — not "made bad calls," see D4's empty-output caveat for the same shape of mistake), not "0.0" in the sense of being wrong on every guess.

Consequence for the Next task: this closes "find an external benchmark" as an open question, and reframes what's actually blocking Milestone 1. It isn't finding calibration data anymore — it's that SubstringJudge is now proven, twice, unfit to report a trustworthy leakage_rate at all. LLMJudge already exists in evals/judge.py (unused, unwired) as the real next component to bring in before drawing any mainstream vs. long-tail conclusion, calibrated the same way against these same 2,197 human labels.

D13 — LLMJudge, calibrated the same way, beats the free floor but isn't trustworthy yet either

evals/calibrate_llm_external.py wires LLMJudge to Groq's free tier (llama-3.1-8b-instant — a small model on purpose, matching judge.py's own rationale that entailment is "a short binary call, doesn't justify a large model") and re-runs D12's exact method (real IMDb reviews, "entails ANY of this movie's documented labels", same labeling-coverage caveat) — the direct comparison the project's own next task called for.

Budget forced a smaller, different sample than D12's. Groq's free tier caps this key at 1,000 requests/day, and each review costs len(that movie's labels) calls — the full 7,657-review set would be ~20,000 calls, not possible for free in one day. Ran a fixed, seeded, stratified sample instead: 20 reviews/title (10 spoiler-tagged, 10 not) across the same 9 covered titles — 180 reviews, 480 actual LLM calls. Reviews were also truncated to 350 characters/call to stay under the ~6,000 tokens/min cap.

Result:

Judge	n	Recall	Precision
SubstringJudge (D12, full 7,657-review set)	7,657	0.0	undefined (0 positive predictions)
LLMJudge / llama-3.1-8b-instant (this, 180-review sample)	180	0.089 (8/90 caught)	0.471

Read this straight, not rounded up or down:

• A real, measured improvement over the free floor. LLMJudge sees paraphrases SubstringJudge structurally cannot (0.0 recall by construction). Not nothing.
• Still not trustworthy on its own. 8.9% recall means it misses ~91 of every 100 real spoiler reveals in this sample. Precision 0.471 means fewer than half its positive calls are right. Neither clears a bar where this judge's leakage_rate could be reported as a safety claim.
• Two confounds this run can't separate, worth resolving before concluding "the model is the ceiling": (1) truncating reviews to 350 characters for TPM budget reasons may suppress recall independent of the model's real ability — a fair test needs full review text; (2) llama-3.1-8b-instant is the smallest, fastest model available, chosen for the same reason SubstringJudge exists as a cheap floor — a stronger model (llama-3.3-70b-versatile, or Claude) tested the same way would show whether this is a model-capacity ceiling or a budget/truncation artifact.
• 180 reviews, one small model, one free-tier run is a smaller claim than D12's full 7,657-review SubstringJudge result — reported at that size, not silently generalized past it.

Next task implication: neither judge is currently fit to report a trustworthy leakage_rate. The path forward isn't "ship LLMJudge instead of SubstringJudge" — it's resolving the two confounds above (full text, stronger model) or exploring genuinely different approaches (a lightweight NLI/entailment classifier run locally, no per-call cost or rate limit; or training a classifier directly on this project's own 2,197-positive/5,460-negative external labels, held out properly) before trusting any leakage number this project reports.

D14 — Scaling the researched catalogue automates the labor, not the citation requirement

The demo track's 8 hand-researched titles took ~15 min each. The user's goal is a much longer list of the most important films in cinema history, "de forma gratuita y veraz" (free and truthful) — which raised a real question: does scaling that mean relaxing D6/D7 (every fact needs a real, checkable source)? Explicitly asked and explicitly answered: no. What gets automated is finding and reading sources and drafting from them, not the citation bar itself, and not the human review before anything is published.

webapp/research_assist.py implements this:

• Real retrieval, not memory. src/preshow/wikipedia.py (new, stdlib-only, same pattern as tmdb.py) fetches a film's Wikipedia article and splits out plot/production/reception/accolades text. tmdb.get_director() (new) adds the director via TMDB's credits endpoint. The LLM (Groq, free) drafts a ContentPack using ONLY this retrieved text — the prompt forbids grounding a claim in anything else, matching the no-fabricated-source rule already established for the baseline generator.
• A code-level safety net, not just a prompt instruction. sanitize_grounding() walks every source_id/url in the draft and nulls out anything that isn't exactly one of the URLs this run actually retrieved. This isn't hypothetical: the first test run had the model cite a specific-looking rottentomatoes.com URL for a score, despite never being given that source — Rotten Tomatoes/Metacritic aren't fetched at all (no simple free API for either), so any score they're credited with in the draft must trace back to Wikipedia's own reporting of it, url null. Same lesson as D3: don't trust the generating model to police its own citations; enforce it in code that runs after generation.
• Output goes to content/_drafts/ (gitignored), never straight to content/researched/. A human review pass is still the actual quality gate — what's automated is turning "write from scratch" into "review and edit," not eliminating the check.

Honestly reported limitation, found by testing on one real title (Citizen Kane) across three prompt iterations: a single generation call from a small free model is inconsistent — the same prompt against the same retrieved text produced anywhere from 4 to 15 grounded claims run to run, with no temperature pinned. Two concrete prompt bugs were found and fixed this way (a fabricated-looking "score" entry with no real source behind it; questions and debate_prompts coming back as literal duplicates) — both confirmed fixed by direct comparison of successive drafts.

Fix implemented and confirmed working live. draft_best_of() generates 3 independent candidates per title from the same shared retrieval (temperature=0.8 for real diversity), sanitizes each candidate's citations first (so a candidate can't win by fabricating extra ones), and keeps the one with the most grounded claims; main() now defaults to this instead of a single draft() call. First committed without a live end-to-end run — Groq's API returned a network-level 403 ("Access denied, check your network settings") for every request from that session's execution environment specifically (a bare curl to /v1/models failed the same way; the user confirmed it worked from their own browser/terminal, with or without their VPN, the whole time). The 403 resolved on its own in a later session with no code change and no identified root cause (possibly a temporary Cloudflare IP flag) — python webapp/research_assist.py "Citizen Kane" 1941 3 then ran end-to-end for real: all 3 candidates succeeded (4, 4, 5 grounded claims), the 5-claim one was correctly selected, zero fabricated citations reached the output.

One real, minor quality issue surfaced by that live run, left as a future prompt fix, not a grounding violation: author_voice came back as the model's own generic critical commentary ("As a film-literate critic, I approach...") rather than an actual quote or statement from someone who made the film (director, screenwriter) — the field's intended content, as seen in Gone Girl's Fincher/Flynn quotes. It still cites the retrieved Wikipedia source correctly; it's a category mismatch, not a fabrication.

Rejected

• Multi-agent (researcher / writer / critic). No dynamic decision to delegate. Adds non-determinism and cost in exchange for aesthetics.
• Markdown as the generator's output. It's a render layer. The data is typed JSON, or the path to TTS + editing never lands.
• Books in v1. Different retrieval, no equivalent to RT/Metacritic or production data. SourceAdapter is designed for two domains, only one gets implemented.

Pending — free-tier constraint and scaling the catalogue

The project is meant to run without a paid API key. One follow-up left (the TMDB browse tier from D10, and the free-tier baseline generator below, are both done):

• Free-tier baseline generator — implemented, blocked only on a key. AnthropicBaselineGenerator (src/preshow/baseline.py) needed a paid ANTHROPIC_API_KEY to unblock Milestone 0. GroqBaselineGenerator (src/preshow/baseline_groq.py, --generator baseline-groq) is the same Generator — identical prompt and output schema, shared verbatim via baseline_prompts.py so a leakage_rate difference between the two would measure the model, not a prompt drift — against Groq's free tier (no card required). Get a key at console.groq.com/keys, set GROQ_API_KEY, then run python evals/run_eval.py --generator baseline-groq to get real numbers into the README. Milestone 1 (retrieval) still shouldn't start before those numbers exist.
• Automating the "+ Suggest a movie" pipeline. The webapp captures suggestions (POST /api/requests, appended to content/movie_requests.json, gitignored) and now resolves the exact film via TMDB autocomplete (D10) so a suggestion carries a tmdb_id, but it still does not research or add anything automatically. Turning a suggestion into a content/researched/*.json entry has to meet the same bar as the existing 8 — cited sources, no invented facts (D6/D7) — which is a real research pipeline with a review step, not a one-request LLM call. Deliberately left as a manual step until that pipeline is designed.

Open questions

• What's the AMBER threshold? Depends on judge calibration.
• What does the judge cost per case? len(surface) x len(labels) calls. Probably the most expensive component of the pipeline. Measure before optimizing.
• Find a public, directly-downloadable spoiler benchmark to calibrate the judge against. Done — see D12. Wire up LLMJudge and calibrate it the same way. Done — see D13: recall 0.089, precision 0.471 on a 180-review sample, better than the floor but not trustworthy yet. Open now: re-test with full (untruncated) review text and/or a stronger model to separate a real capability ceiling from a budget/truncation artifact, or evaluate a local NLI classifier / a classifier trained on this project's own 2,197/5,460 external labels as a genuinely different approach — before trusting any leakage_rate this project reports.