Raised in review of #118, and now bounded rather than open-ended — but the coverage question is real and needs measurement.
What #118 does now. One prefix ask carries at most maxAskItems = 12 candidates, largest first, and what the cap leaves unasked is counted as sweep_over_ask_cap. Before the cap the ask was uncapped, which risked losing everything: the reply carries a verbatim quote per verdict against a 16,000-token budget, live measurement showed ~600 tokens per verdict (a 12-candidate ask produced a 7,191-token reply), so around 26 candidates exhausts it — and truncation is all-or-nothing, since the array never closes and no verdict parses. A 50-candidate transcript would have swept nothing while paying for the call.
Twelve because two independent arguments agree: reply-budget arithmetic puts the ceiling near 26 and something well inside it is prudent, and cc1aa9f measured quote fidelity degrading with size (4 of 37 non-verbatim at 16, against 0 of 16 at 10). That fidelity figure was taken when content was copied into the prompt, which it no longer is, so it corroborates rather than proves.
The open question
Should the sweep make more than one ask to cover a large transcript? On a transcript with 50 eligible candidates it currently sweeps 12 and leaves 38 — bounded, counted, fail-safe, and incomplete.
What a measurement has to settle:
- Does N asks over one transcript beat one ask, on tokens actually removed, net of N cache reads? Each ask re-reads the same prefix, so the marginal cost is one cache read plus one reply — cheap, but not free.
- Does fidelity hold across asks? The fidelity measurement was about items within one reply; nothing measures whether a second ask over the same transcript degrades.
- Does a second ask see the first's removals? It would read the same cached prefix, which still contains the outputs the first ask decided to drop — so the two asks judge the same transcript and could contradict each other on a shared candidate. The decision-freezing machinery would make the first write win, but the second's tokens are then spent on nothing.
That last point is the interesting one and it is not obviously solvable by looping: the prefix the ask reads is the previous turn's sent body, so it does not update between asks within a turn.
Not urgent
The cap makes the current behaviour honest rather than lossy, and sweep_over_ask_cap says how often it bites. On the one real session measured during verification there were 4 eligible candidates, so the cap was nowhere near binding — the per-output floor rejected 11 of 15 first. Worth revisiting once there is a distribution of sweep_offered from real traffic; if it rarely exceeds 12, this stays theoretical.
Raised in review of #118, and now bounded rather than open-ended — but the coverage question is real and needs measurement.
What #118 does now. One prefix ask carries at most
maxAskItems = 12candidates, largest first, and what the cap leaves unasked is counted assweep_over_ask_cap. Before the cap the ask was uncapped, which risked losing everything: the reply carries a verbatim quote per verdict against a 16,000-token budget, live measurement showed ~600 tokens per verdict (a 12-candidate ask produced a 7,191-token reply), so around 26 candidates exhausts it — and truncation is all-or-nothing, since the array never closes and no verdict parses. A 50-candidate transcript would have swept nothing while paying for the call.Twelve because two independent arguments agree: reply-budget arithmetic puts the ceiling near 26 and something well inside it is prudent, and
cc1aa9fmeasured quote fidelity degrading with size (4 of 37 non-verbatim at 16, against 0 of 16 at 10). That fidelity figure was taken when content was copied into the prompt, which it no longer is, so it corroborates rather than proves.The open question
Should the sweep make more than one ask to cover a large transcript? On a transcript with 50 eligible candidates it currently sweeps 12 and leaves 38 — bounded, counted, fail-safe, and incomplete.
What a measurement has to settle:
That last point is the interesting one and it is not obviously solvable by looping: the prefix the ask reads is the previous turn's sent body, so it does not update between asks within a turn.
Not urgent
The cap makes the current behaviour honest rather than lossy, and
sweep_over_ask_capsays how often it bites. On the one real session measured during verification there were 4 eligible candidates, so the cap was nowhere near binding — the per-output floor rejected 11 of 15 first. Worth revisiting once there is a distribution ofsweep_offeredfrom real traffic; if it rarely exceeds 12, this stays theoretical.