docs(bob): re-run WO-14 voice baseline with both halves (#95)

[sergi-torres]

Re-runs the WO-14 voice-matching evaluation so that both halves of the A/B exist and were produced under the same, correct generation parameters. Appends to bob/sessions/Sprint_1/baseline_eval.md; no code is touched.

Why re-run

Two things invalidated the run recorded on this branch (2026-07-28 10:52 UTC):

1. The max_tokens defect ([backend] El tope de tokens de salida se ignora en silencio: max_new_tokens vs max_tokens en ModelInference.chat() #106, PR fix(ml,back): enforce the output token cap (#106) #123). _GENERATION_PARAMS sent max_new_tokens, which does not exist in the TextChatParameters schema ModelInference.chat() accepts. It was discarded silently and the service default of 1024 applied. Appendix B item 1 of the original document predicted exactly this. Run 1's generations were produced under a cap nobody chose.
2. The corpus is seeded ([ml] WO-05 — Hacer que make seed deje la base lista para la demo (embeddings + StyleProfiles) #86). The original run could not do the conditioned half at all — style_profiles was empty and there were no chunks, so a "conditioned" run would have been a second vanilla run wearing a hat. The DB now holds 3 authors, 10 documents, 4 529 chunks with 0 NULL embeddings, and 3 style profiles.

What was run

Same five prompts, byte for byte — the harness parses them out of the ```text blocks of §4 of the document itself rather than retyping them.

• vanilla: generate(prompt, None, model_id, params) — no system prompt, no profile, no RAG.
• conditioned: orchestrate steps 1–4 reproduced call for call — embedder.encode → db.retrieve_top_k(k=5, author_id=…, database_url=…) → conditioner.build_system_prompt → generate(...). The real conditioner; no hand-written prompt.
• Both halves: params={"max_tokens": 512, "temperature": 0.7, "top_p": 0.9}, passed as a literal dict, not imported from _GENERATION_PARAMS, so the measurement cannot drift with whatever branch the primary checkout is on.
• Both issued as one asyncio.gather, the same shape as orchestrate step 4, so pair latency is a real side-by-side figure.

Results

10/10 generations succeeded on the first attempt — zero retries, zero timeouts, zero errors. Run 1 had burned an 8s timeout on P3 under the 1024-token cap; that is the #106 fix showing up as reliability.

First real measurement against the docs/MVP.md "side-by-side < 8s P95" SLA (n=5, so the document reports max and spread and explicitly refuses to call it a P95):

Stage	min	median	mean	max
Vanilla call alone	3.12	3.91	3.81	4.30
Conditioned call alone	5.38	6.14	6.15	6.78
Pair (side-by-side)	5.38	6.14	6.16	6.80
Pair + embed + RAG	6.36	6.83	6.90	7.38

Worst observed: 7.38s against an 8s budget — ~0.6s of headroom, and that headroom still has to absorb fit_score ×2, passport signing, Supabase round-trips and HTTP, none of which this run timed. Stated as a lower bound, not as a pass.

The conditioned branch set the pair latency in 5/5 pairs, but normalised for output length the two branches are the same speed (22.2 vs 22.1 ms/word) — the gap is entirely that the conditioned model writes ~1.6× more, not conditioning overhead.

Scoring

Nothing is scored. All ten cells are blank, per §2 of the document — the 1–10 voice-similarity judgement is Sergi's. Run 1's score table is relabelled superseded and left blank so no mean can be quoted out of a configuration that never shipped; run 1 itself is not deleted.

New open items (reported, not fixed — Appendix D)

1. The conditioner blows its own token budget by ~2×. Its docstring claims the system prompt is "kept under ~1200 tokens"; measured at 2 589–2 591 tokens on every call. _MAX_CHUNKS = 5 cannot enforce a token budget it does not measure — each chunk is ~500 tokens on its own.
2. distinctive_vocab is not distinctive. 7 of the 15 terms (come, know, little, make, say, think, time) are identical across Dickens, Austen and Poe. Looks like raw frequency ranking rather than frequency relative to a reference corpus. One of three conditioning signals is close to inert. (fix/wo-12-distinctive-vocab exists, so possibly known.)
3. The 8s SLA equals the 8s per-attempt client timeout. A generation that is merely slow is killed and retried rather than degrading. No graceful band.

Refs #95 — does not close it. The document is AWAITING HUMAN SCORING; the R1 gate is discharged only once Sergi fills in §9 and records a verdict in §6.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
author-ai	Ready	Preview, Comment	Jul 28, 2026 9:34pm