metrics: extract_llm and extract_llm_sweep measure saved tokens against different baselines, so their per-arm savings are not comparable

[amiddavid]

The inconsistency

After #188, the two extraction components measure "saved tokens" against different baselines:

Component	Measures	Site
extract_llm_sweep	content − the message as spliced	extract_sweep.go, phase 3
extract_llm	content − the projection	extract_llm.go, out[k].saved

In marker_mode: full — the default — the text actually written is replacement + "\n" + marker + recovery hint. So the projection-based figure omits the marker and the hint, and overstates by that much per candidate. Measured on the sweep's stocked fixture before it was corrected: 86,458 claimed against 86,201 actually sent, i.e. ~23 tokens per candidate.

Both feed metrics.RecordExtractionSaving / RecordExtractionValue, and both surface in /stats under by_component, so the two arms of a comparison are no longer measuring the same thing.

Why the gap matters more than its size

It is systematic, not noise: a fixed per-candidate overhead in one component and not the other, so it scales with the number of removals rather than averaging out. A configuration that removes more looks disproportionately better under the projection-based figure.

That matters specifically because extract_llm's savings figure is what iteration 024's re-run will be judged on. Anyone reading the two arms side by side needs to know one is wire-measured and the other is not.

The sweep is the correct one

content − MessageText(req.Input[i]) after the splice is the number an operator can check against their bill: it is what the request actually shrank by. extract_llm should move to the same basis.

Why it was not fixed in #188

Deliberate, and the reasoning is worth keeping: #188's subject is reserve durability, and moving a published savings figure inside it would make the iteration-024 comparison unreadable for a reason unrelated to what that PR changed. A savings number should move in a change whose subject is the savings number — otherwise the re-run cannot separate "the mechanism got better" from "we started counting differently". The sweep's figure moved there only because a comment added in the same commit asserted it was the wire figure, which made it wrong rather than merely inconsistent.

Suggested fix

Mirror the sweep: compute saved in phase 3 from the spliced message rather than carrying the projection-only delta out of runCall. out[k].saved already exists for that purpose, so it is a one-line change plus its test.

Note the same question applies to the replay paths, which credit content − cached.Projected at the repeat rate — also projection-only.

Sequencing

Worth landing after the iteration-024 re-run, or as an explicit note beside its results, so the re-run's numbers stay comparable to the ones already published. Otherwise a savings change lands in the middle of the comparison it would invalidate.

[amiddavid]

Correction: the gap is variable and summary-dominated, not a fixed per-candidate overhead

The description above says the projection-based figure "omits the marker and the hint" and puts the overhead at ~23 tokens per candidate. That is true of the sweep's old figure and wrong for extract_llm, which is the component this issue is actually about. Correcting it here because the magnitude changes the sequencing argument.

extract_llm.apply splices:

return projected + "\n[" + summary + "] " + tok   // tok = marker + hint

while out[k].saved counts before − TextTokens(res), the projection alone. So the overstatement is summary + marker + hint, and the summary is the dominant term by an order of magnitude — a 900-token output compacted to 100 with a 60-token summary books 800 saved where the message shrank by ~715.

So the cross-component gap is not a fixed ~23 tokens. It is variable, per-candidate, and driven by summary length, which is a model output and therefore not even stable across runs of the same configuration.

Why that matters beyond reporting

out[k].saved feeds e.ratios.observe, and the ratio tracker is what the economic gate consults to decide whether a call is worth making at all. So an optimistic saving is not only a reporting artefact — it biases the decision to spend, and it does so in the direction of spending more.

That makes this more than a comparability problem, and it strengthens the case for fixing it rather than merely documenting it. It does not change the sequencing conclusion: still after the iteration-024 re-run or with an explicit note beside the results, because a savings-basis change landing mid-comparison recreates the original iteration-024 confound one level up.

Affected sites, corrected list

Site	Books	Omits
extract_llm.go fresh (out[k].saved)	projection	summary + marker + hint
extract_llm.go replay	content − cached.Projected	summary + marker + hint
extract_sweep.go fresh	the spliced message	nothing — correct
extract_sweep.go replay	the replayed message	nothing — correct as of c442670

The sweep's replay path was descriptor-only until c442670, which also made that component internally inconsistent for one commit — fresh measuring the wire, replay measuring the descriptor. Both now measure the message, so the sweep is coherent and extract_llm is the only component left on a projection basis.

Credit: raised by the reviewer on #188, who also worked out that the summary rather than the marker is the dominant term.