Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,10 @@ small enough that a framework would be cost without benefit, not "doesn't
know how to use one."

**AI / evaluation:** Anthropic Claude (tool use / structured output for the
paid baseline generator + research-assist drafting) · Groq/Llama (free-tier
alternative for the same baseline generator, the `LLMJudge`, and
research-assist's default model, so nothing in this project *requires* a
paid key) · a custom evals harness design (leakage / grounding / richness)
paid baseline generator) · Groq/Llama (free-tier alternative for the same
baseline generator, the `LLMJudge`, and the only model `research_assist.py`
uses — no Anthropic path exists there, so nothing in this project *requires*
a paid key) · a custom evals harness design (leakage / grounding / richness)
with a calibrated judge, verified against planted leaks and real human
spoiler reviews.

Expand Down
28 changes: 23 additions & 5 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ is right. Instead, TMDB is a **third tier**, structurally kept apart:
| Tier | Source | Size | Has a curtain / cited claims? |
|---|---|---|---|
| Measurement | `evals/dataset/titles.yaml` | 20, fixed | N/A — it's the experiment |
| Researched | `content/researched/*.json` | 7, hand-written | Yes |
| Researched | `content/researched/*.json` | 8, hand-written | Yes |
| **Browse (TMDB)** | `src/preshow/tmdb.py` | effectively all of TMDB | No — poster + synopsis only |

Concretely:
Expand Down Expand Up @@ -353,10 +353,28 @@ three prompt iterations**: a single generation call from a small free model is
to 15 grounded claims run to run, with no temperature pinned. Two concrete prompt bugs were
found and fixed this way (a fabricated-looking "score" entry with no real source behind it;
`questions` and `debate_prompts` coming back as literal duplicates) — both confirmed fixed
by direct comparison of successive drafts. What's still open, not yet built: run-to-run
depth/quality variance is real and unresolved. The likely fix is generating 2-3 drafts per
title and picking the most complete one programmatically (e.g. by grounded-claim count),
rather than trusting a single call — not implemented yet, left for a follow-up session.
by direct comparison of successive drafts.

**Fix implemented and confirmed working live.** `draft_best_of()` generates 3 independent
candidates per title from the same shared retrieval (`temperature=0.8` for real diversity),
sanitizes each candidate's citations first (so a candidate can't win by fabricating extra
ones), and keeps the one with the most grounded claims; `main()` now defaults to this
instead of a single `draft()` call. First committed without a live end-to-end run — Groq's
API returned a network-level 403 ("Access denied, check your network settings") for every
request from that session's execution environment specifically (a bare `curl` to
`/v1/models` failed the same way; the user confirmed it worked from their own
browser/terminal, with or without their VPN, the whole time). The 403 resolved on its own
in a later session with no code change and no identified root cause (possibly a temporary
Cloudflare IP flag) — `python webapp/research_assist.py "Citizen Kane" 1941 3` then ran
end-to-end for real: all 3 candidates succeeded (4, 4, 5 grounded claims), the 5-claim one
was correctly selected, zero fabricated citations reached the output.

One real, minor quality issue surfaced by that live run, left as a future prompt fix, not a
grounding violation: `author_voice` came back as the model's own generic critical
commentary ("As a film-literate critic, I approach...") rather than an actual quote or
statement from someone who made the film (director, screenwriter) — the field's intended
content, as seen in Gone Girl's Fincher/Flynn quotes. It still cites the retrieved
Wikipedia source correctly; it's a category mismatch, not a fabrication.

## Rejected

Expand Down
Loading