From 31d5539f7ffc3becd5bcea5ff5babdee1280e1dd Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 20:08:10 +0000 Subject: [PATCH] docs: sync D14 with best-of-3 confirmation and fix stale counts - docs/DESIGN.md D10: correct researched-title count in the tier table (7 -> 8; the Gone Girl addition was never reflected there). - docs/DESIGN.md D14: the 'best-of-3 draft generation not implemented yet' note was stale -- it shipped and was confirmed working live (Citizen Kane, 3/3 candidates, 4/4/5 grounded claims) two commits after it was written, but only CLAUDE.md was updated at the time. Replaces the stale paragraph with what actually happened: the Groq 403 that blocked the first live run, its unexplained resolution, and the live confirmation numbers. Also records the one real but minor quality gap found in that run (author_voice returning generic critic commentary instead of a real quote) so it isn't only living in CLAUDE.md. - README.md Stack section: corrected a wrong claim that Anthropic Claude powers research-assist drafting. webapp/research_assist.py is Groq-only (hardcoded llama-3.3-70b-versatile, no Anthropic client anywhere in the file) -- there is no Claude path there at all, so 'research-assist's default model' incorrectly implied an alternative that doesn't exist. Everything else checked against the running repo (test count, CI workflow name, all 20 titles/tmdb_ids/spoiler labels, 8 researched entries + their translations, gitignore vs project-layout table, README anchor links, v1.0.0 release tag) and found accurate -- no further changes needed there. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01YSnwX5AyHkAXxGgFxE7uu6 --- README.md | 8 ++++---- docs/DESIGN.md | 28 +++++++++++++++++++++++----- 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index cf93e17..0367290 100644 --- a/README.md +++ b/README.md @@ -81,10 +81,10 @@ small enough that a framework would be cost without benefit, not "doesn't know how to use one." **AI / evaluation:** Anthropic Claude (tool use / structured output for the -paid baseline generator + research-assist drafting) · Groq/Llama (free-tier -alternative for the same baseline generator, the `LLMJudge`, and -research-assist's default model, so nothing in this project *requires* a -paid key) · a custom evals harness design (leakage / grounding / richness) +paid baseline generator) · Groq/Llama (free-tier alternative for the same +baseline generator, the `LLMJudge`, and the only model `research_assist.py` +uses — no Anthropic path exists there, so nothing in this project *requires* +a paid key) · a custom evals harness design (leakage / grounding / richness) with a calibrated judge, verified against planted leaks and real human spoiler reviews. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 082f261..aa01cf8 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -150,7 +150,7 @@ is right. Instead, TMDB is a **third tier**, structurally kept apart: | Tier | Source | Size | Has a curtain / cited claims? | |---|---|---|---| | Measurement | `evals/dataset/titles.yaml` | 20, fixed | N/A — it's the experiment | -| Researched | `content/researched/*.json` | 7, hand-written | Yes | +| Researched | `content/researched/*.json` | 8, hand-written | Yes | | **Browse (TMDB)** | `src/preshow/tmdb.py` | effectively all of TMDB | No — poster + synopsis only | Concretely: @@ -353,10 +353,28 @@ three prompt iterations**: a single generation call from a small free model is to 15 grounded claims run to run, with no temperature pinned. Two concrete prompt bugs were found and fixed this way (a fabricated-looking "score" entry with no real source behind it; `questions` and `debate_prompts` coming back as literal duplicates) — both confirmed fixed -by direct comparison of successive drafts. What's still open, not yet built: run-to-run -depth/quality variance is real and unresolved. The likely fix is generating 2-3 drafts per -title and picking the most complete one programmatically (e.g. by grounded-claim count), -rather than trusting a single call — not implemented yet, left for a follow-up session. +by direct comparison of successive drafts. + +**Fix implemented and confirmed working live.** `draft_best_of()` generates 3 independent +candidates per title from the same shared retrieval (`temperature=0.8` for real diversity), +sanitizes each candidate's citations first (so a candidate can't win by fabricating extra +ones), and keeps the one with the most grounded claims; `main()` now defaults to this +instead of a single `draft()` call. First committed without a live end-to-end run — Groq's +API returned a network-level 403 ("Access denied, check your network settings") for every +request from that session's execution environment specifically (a bare `curl` to +`/v1/models` failed the same way; the user confirmed it worked from their own +browser/terminal, with or without their VPN, the whole time). The 403 resolved on its own +in a later session with no code change and no identified root cause (possibly a temporary +Cloudflare IP flag) — `python webapp/research_assist.py "Citizen Kane" 1941 3` then ran +end-to-end for real: all 3 candidates succeeded (4, 4, 5 grounded claims), the 5-claim one +was correctly selected, zero fabricated citations reached the output. + +One real, minor quality issue surfaced by that live run, left as a future prompt fix, not a +grounding violation: `author_voice` came back as the model's own generic critical +commentary ("As a film-literate critic, I approach...") rather than an actual quote or +statement from someone who made the film (director, screenwriter) — the field's intended +content, as seen in Gone Girl's Fincher/Flynn quotes. It still cites the retrieved +Wikipedia source correctly; it's a category mismatch, not a fabrication. ## Rejected