sweep_quote_fabricated swings 3x between runs and inverts between arms, so it cannot gate a PR

[amiddavid]

What

sweep_quote_fabricated — the check that catches extract_llm_sweep inventing the obligation quote
it justifies a keep with — moves by a factor of 3 between benchmark runs of the same two builds,
and in opposite directions. It is currently the only guard against the model making its
justification up, so nobody can say from the published numbers whether a change makes fabrication
better or worse.

How it was found

PR #137's body reports, as a regression that change causes:

sweep_quote_fabricated as a share of verdicts returned: 15.8% on main (29/184) against 24.9% on
this branch (112/450)

Re-measuring the same two builds for the #137 review, three passes per arm
(aws/claude-sonnet-5, LOCA S2L cg_128k_3, pre_expiry_seconds: 7200, block_fallback: true,
min_tokens: 400 — the same knobs the earlier run used):

	origin/main	#137 branch
verdicts returned	168	837
sweep_quote_fabricated	41	70
as a share of verdicts	24.4%	8.4%
earlier run, same builds	15.8%	24.9%

Both arms moved by roughly 3x and they swapped order. main went 15.8% -> 24.4%; the branch went
24.9% -> 8.4%. On the earlier run the branch looked 1.6x worse than main; on this one it looks 2.9x
better.

The two runs differ in the box and in how far each LOCA trajectory happened to get, which is the
point: a metric that inverts under that much variation is not measuring the property it names at a
usable precision.

Why it matters

extract_llm_sweep DROPS tool outputs from a transcript on the strength of a model verdict. The
contract requires a keep to quote the verbatim transcript text creating the obligation, and
sweep_quote_fabricated fires when that quote is not found. It is the whole of the
model-is-inventing check. If its rate is dominated by workload variance then:

• no PR can be held to "does not make fabrication worse", because the noise exceeds any plausible
  effect;
• a real regression is invisible inside the band;
• and the 8-25% band itself is worth understanding on its own terms. Even the low end means roughly
  one verdict in twelve cites text that is not there.

Severity

Moderate. Not a correctness bug in the drop path — a fabricated quote is detected, and the
component does not act on it — but it is a measurement instrument that cannot currently support the
decisions it is being used for, on a component whose failure mode is silent content loss.

Fix options

1. Report it per ask, not per verdict, and with a denominator that is stated. A share of
   "verdicts returned" moves when batch size moves, and batch size tracks how much tool output a
   trajectory produced. Per-ask, or per-keep-verdict (only keeps carry an obligation quote at
   all), would remove one confound for free.
2. Split near-miss from invention. Check whether the quote is a normalised substring
   (whitespace, unicode, ellipsis) before calling it fabricated. If a large share is whitespace
   drift, the current number is mostly a string-matching artefact and the real rate is far lower —
   that is the first thing to find out, and it is cheap.
3. Fix the sample size. Run enough passes to put a confidence interval on it, and publish the
   interval rather than a point estimate. Costs benchmark budget and fixes nothing by itself, but no
   other option can be evaluated without it.
4. Decline on a fabricated quote instead of only counting it. The safe behaviour, and arguably
   what the check is for — but it should not be turned on while the rate is this poorly characterised,
   or the component may be silently disabled for a string-matching reason (see option 2).

Recommendation: 2 first (it is cheap and may explain most of the band), then 1, then decide about 4
on the corrected numbers.

Found while measuring #137. Explicitly not folded into that PR: the earlier run recorded it as a
regression that PR caused, and this run shows the opposite, so attributing it to that change either
way is unsupported.