diff --git a/README.md b/README.md index cf93e17..0367290 100644 --- a/README.md +++ b/README.md @@ -81,10 +81,10 @@ small enough that a framework would be cost without benefit, not "doesn't know how to use one." **AI / evaluation:** Anthropic Claude (tool use / structured output for the -paid baseline generator + research-assist drafting) · Groq/Llama (free-tier -alternative for the same baseline generator, the `LLMJudge`, and -research-assist's default model, so nothing in this project *requires* a -paid key) · a custom evals harness design (leakage / grounding / richness) +paid baseline generator) · Groq/Llama (free-tier alternative for the same +baseline generator, the `LLMJudge`, and the only model `research_assist.py` +uses — no Anthropic path exists there, so nothing in this project *requires* +a paid key) · a custom evals harness design (leakage / grounding / richness) with a calibrated judge, verified against planted leaks and real human spoiler reviews. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 082f261..aa01cf8 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -150,7 +150,7 @@ is right. Instead, TMDB is a **third tier**, structurally kept apart: | Tier | Source | Size | Has a curtain / cited claims? | |---|---|---|---| | Measurement | `evals/dataset/titles.yaml` | 20, fixed | N/A — it's the experiment | -| Researched | `content/researched/*.json` | 7, hand-written | Yes | +| Researched | `content/researched/*.json` | 8, hand-written | Yes | | **Browse (TMDB)** | `src/preshow/tmdb.py` | effectively all of TMDB | No — poster + synopsis only | Concretely: @@ -353,10 +353,28 @@ three prompt iterations**: a single generation call from a small free model is to 15 grounded claims run to run, with no temperature pinned. Two concrete prompt bugs were found and fixed this way (a fabricated-looking "score" entry with no real source behind it; `questions` and `debate_prompts` coming back as literal duplicates) — both confirmed fixed -by direct comparison of successive drafts. What's still open, not yet built: run-to-run -depth/quality variance is real and unresolved. The likely fix is generating 2-3 drafts per -title and picking the most complete one programmatically (e.g. by grounded-claim count), -rather than trusting a single call — not implemented yet, left for a follow-up session. +by direct comparison of successive drafts. + +**Fix implemented and confirmed working live.** `draft_best_of()` generates 3 independent +candidates per title from the same shared retrieval (`temperature=0.8` for real diversity), +sanitizes each candidate's citations first (so a candidate can't win by fabricating extra +ones), and keeps the one with the most grounded claims; `main()` now defaults to this +instead of a single `draft()` call. First committed without a live end-to-end run — Groq's +API returned a network-level 403 ("Access denied, check your network settings") for every +request from that session's execution environment specifically (a bare `curl` to +`/v1/models` failed the same way; the user confirmed it worked from their own +browser/terminal, with or without their VPN, the whole time). The 403 resolved on its own +in a later session with no code change and no identified root cause (possibly a temporary +Cloudflare IP flag) — `python webapp/research_assist.py "Citizen Kane" 1941 3` then ran +end-to-end for real: all 3 candidates succeeded (4, 4, 5 grounded claims), the 5-claim one +was correctly selected, zero fabricated citations reached the output. + +One real, minor quality issue surfaced by that live run, left as a future prompt fix, not a +grounding violation: `author_voice` came back as the model's own generic critical +commentary ("As a film-literate critic, I approach...") rather than an actual quote or +statement from someone who made the film (director, screenwriter) — the field's intended +content, as seen in Gone Girl's Fincher/Flynn quotes. It still cites the retrieved +Wikipedia source correctly; it's a category mismatch, not a fabrication. ## Rejected