extract_llm_sweep: the prefix ask reads a transcript that excludes every candidate it judges

[amiddavid]

Found by live verification of #118 against the real gateway, with a verbatim wire recorder between
context-guru and the provider.

The defect

The prefix ask reads the previous turn's sent body — everything up to MaxCachedIdx. The
candidate set is the uncached tail — only messages past MaxCachedIdx.

Those two sets are disjoint by construction. So the model is asked to judge whether outputs are
spent, over a transcript that does not contain them.

Why: sweeping() implies !ColdCache, so TailOnlyCold(i, true) degrades to the ordinary tail
gate, and only post-boundary messages become candidates. The ask, meanwhile, is the pre-boundary
body. Nothing in between connects them.

What the model actually did

Live, with the provider's own usage off the wire:

• A/turn3 — two older outputs refused as cached_prefix=2, the two new ones offered. The model
  kept both and cited "Reply with the word ACK only." as the obligation for each. That is a real
  transcript string, so sweep_quote_fabricated correctly stayed silent — but it is not a
  judgement about the outputs. It is the only obligation the model could find, because the outputs
  were not there to reason about.
• D/turn2 — dropped its output on {"i":0,"needed_by":"none","quote":"","verdict":"drop"},
  having seen only begins: # ledger_b and a token count.

So a drop is being decided from a 90-character head and a size. The comparative judgement this
design rests on did not happen on either prefix-ask turn.

The second half: depth and the cache read are mutually exclusive

Both derive from the same tracker record. A tracker hit gives you a stash to ask over and a
tail-only boundary. A tracker miss gives you depth and sweep_no_prefix. Every
deep-candidate firing observed (runs C and E, MaxCachedIdx=-1) was on the fallback path — the
content-carrying prompt — not on the prefix ask.

So the mechanism that reaches deep history is the one the prefix ask was built to replace, and the
prefix ask only ever sees the tail it cannot judge.

Consequence: the shipped default is the refuted shape

sweep_inventory_of_one fires on the ordinary agent turn, not an edge case. The candidate set
is the tail; the tail of a normal agent turn is one tool result. Run D — the plain case — produced
sweep_inventory_of_one=1 sweep_dropped=1. Run A needed two tool calls crammed into a single turn
to reach an inventory of two.

That is the per-output shape 4ca1f13 records as "refuted at 6% live-kept", reached by default,
and the component drops on it.

Related bias in the fallback: conversationGoal opens with the first user message, so on B/turn3
the goal read [r1] Read data/orders_2021.csv … tell me the order count in each and the model kept
all three outputs as needed for "(a) the step you are on right now".

Why no test caught it

All 20 sweep unit tests run with MaxCachedIdx: -1, which disables the tail gate entirely — so
the disjointness cannot appear. And extract_sweep_test.go:104 claims the depth restriction is
lifted "so the candidates reach the inventory: the tail gate is exercised separately below". There
is no such test, in that file or the repo. That comment asserts coverage that does not exist.

This is the fifth vacuous-coverage instance in this component family (after the three trim tests,
the tool_result-counting precondition in #110, and the economic_gate: false blind spot in this
same component).

What needs deciding

This is design-level, not a patch. The options are not equivalent and each gives something up:

1. Ask over the whole transcript, not the previous sent body — put the candidates inside the
   region the model reads. Loses the cache read for anything past the boundary, which is the
   mechanism's whole economy.
2. Make the candidates the cached region — judge deep history, which is what the proposal says
   the component is for, and accept that acting on it invalidates the prefix. That is what the
   original cold-turn gating was for, and the pre-expiry window was meant to make it cheap.
3. Keep the tail candidates and stop pretending the model reads them — ship the outputs in the
   ask, i.e. the fallback path, and drop the prefix-ask premise for this component.

(2) looks closest to the stated intent: the proposal's problem statement is deep history, and the
pre-expiry window exists precisely so that invalidating the prefix costs little. Under (2) the
prefix ask and the candidate set coincide instead of excluding each other.

Also stale, and load-bearing for this decision

docs/proposals/sweep-adjudicator.md frames the problem entirely as deep history and never states
that the tail gate confines the component to the uncached tail — it says only that "the newest tool
output is invisible" to the ask, which understates this by the whole transcript.

docs/hosted.md:645-653 is wrong on five counts for this branch: "Both call a cheap model
(claude-haiku-4-5…)", "when the economic gate says", "runs only on a turn whose prompt cache has
expired", "adjudicates at depth", and "a batch of outputs at a time".

[amiddavid]

Decision from David: do not fire the sweep at all unless the inventory has at least X candidates, default 10.

This is well-grounded, and 10 is a measured inflection rather than a round number. cc1aa9f: at batch 3-6 the model dropped a genuinely-spent output only 2 times in 4; at batch 10, 4 in 4, clearing 100% of genuinely-spent candidates. And the selection experiment over 8,105 decisions: 1 candidate → 6% live-kept (haiku) / 14% (sonnet), both inside the drop-everything null model's error bar; ~15 → 58% at the lowest cost per output. So a floor of 10 sits where willingness-to-act is measured to arrive.

It also converts the current failure into a fail-safe one: below the floor the component does nothing, instead of asking a question the measurements say it answers badly and then dropping on the answer — which is what run D did from a 90-character head and a token count.

Two things to get right when implementing it.

It must be counted, not silent. A component that declines is indistinguishable from one that is broken unless the decline is recorded — that is how the economic_gate: false blind spot and the three trim tests all hid in this same family. Something like sweep_inventory_below_min, in the reviewed surface. Keep sweep_inventory_of_one as well: with a floor of 10 an inventory of one can never reach the model, so that counter becomes the tripwire for the degenerate extreme rather than the thing that fires in normal traffic.

It is a safety floor, not a fix for this issue. The defect here is that the candidate set and the region the model reads are disjoint. A minimum-inventory gate makes the component fire less often; when it does fire it is still judging outputs the model cannot see. And with the candidate set as it stands — the uncached tail — reaching 10 tool results in one tail is rare, so the practical effect of X=10 alone is to make the component nearly inert. That is the honest fail-safe, but it is not a working feature.

The two changes are complementary, and this is the argument for option 2 above rather than a reason to defer it: if the candidates become the cached region — deep history, which is what the proposal always said this component is for — then a long transcript supplies large inventories routinely, so the floor of 10 is satisfied in ordinary traffic and the model is reading the outputs it judges. Option 2 is what makes X=10 achievable rather than merely safe.

Suggested sequencing: option 2 first, then the floor, then re-run the live verification — the floor's effect can only be read off real traffic (how often it declines, and whether inventories clear it), which is exactly what the last run measured wrongly by assuming an inventory of one was an edge case.

[amiddavid]

Correction: this is a regression, not a design choice, and the three options above mis-framed it.

The tail gate exists for the WARM-turn compactor: rewriting inside a live cached prefix invalidates everything after it, so extract_llm must confine itself to the uncached tail. The sweep's whole premise is the opposite — it ACCEPTS that invalidation, originally because the cache had already expired and now because it is about to. Candidates were always meant to be the entire transcript, and the code on main did that: the tail gate was explicitly skipped when sweeping.

Root cause. The permission to touch depth was keyed on the cache having already expired rather than on this component is sweeping and accepts the invalidation cost. Moving the trigger from cold-turn to the pre-expiry window makes "already expired" false by construction, so the trigger change silently withdrew the depth permission and the candidate set collapsed to the tail. The condition needs re-keying to the sweep itself; it is not a choice between three architectures.

So: candidates come from the entire context window. Option 2 above is not one option of three, it is the restoration of intended behaviour. Options 1 and 3 should be disregarded — option 1 gives up the cache read, which is the mechanism's entire economy, and option 3 abandons the prefix ask.

What this makes coherent. With candidates spanning the transcript, the candidate set and the region the ask reads overlap almost completely: the ask carries the previous turn's sent body, which is the transcript up to the boundary. The only candidates outside the read region are the newest tool outputs in the tail — and a9d666f already recorded that as acceptable, since tail content "has had no turns in which to be superseded". That is a footnote, not the whole transcript, which is what the current disjointness amounts to.

It also dissolves the inventory problem rather than merely gating it: a long transcript carries many deep tool outputs, so inventories are naturally large and the minimum-inventory floor of 10 is satisfied in ordinary traffic instead of making the component inert.

Sequencing, revised: re-key the depth permission to sweeping (the actual fix), then the minimum-inventory floor of 10 with its decline counted, then #123's real tool_use_id, then re-run the live verification. And add the test that extract_sweep_test.go:104 claims exists and does not — one that runs with a real MaxCachedIdx rather than -1, since every one of the 20 sweep tests disables the tail gate and that is why this was invisible.

[amiddavid]

Fixed in de6f3fa on feat/sweep-adjudicator (#118).

Root cause, as diagnosed in the thread above: the depth permission was keyed on the cache having ALREADY expired — TailOnlyCold's optIn && c.ColdCache — which the pre-expiry window makes false by construction. Candidates therefore collapsed to the uncached tail while the ask reads the previous turn's sent body, i.e. everything up to the boundary. The trigger change withdrew the permission exactly where the window had just paid for it.

The objection the old comment raised against itself does not hold: inside the window the prefix has at most the window's WIDTH left to live, so the cost of invalidating early is bounded by that width rather than by the TTL. No measurement of early invalidation is needed to justify what the window exists to buy.

Candidates are now the entire transcript, and the effect is counted positively as sweep_candidate_at_depth — a refusal counter cannot distinguish "the gate refused depth" from "nothing was deep this turn", so its going to zero is what says this regressed again.

The inventory floor from the discussion above also landed, default 10 (min_inventory), declining below it rather than asking, with sweep_inventory_below_min recording the decline. Ten is the measured inflection, not a round number.

The test whose absence hid this now exists. extract_sweep_test.go claimed the tail gate "is exercised separately below" and no such test existed; all 20 sweep tests ran MaxCachedIdx: -1, which disables the gate. The new test runs a real boundary, and reverting the fix reproduces the defect and both fixes interacting in one line:

sweep_offered = 4, want 12: candidates inside the cached prefix were refused ...
gates=map[cached_prefix:8 sweep_inventory_below_min:4 sweep_offered:4]

Tail-only starves the inventory to 4, and the floor then declines it — which is exactly why a floor alone would have made the component inert.

Fixture consequences are recorded in the commit: every existing sweep test asserted counts against an inventory of one, so they opt in explicitly via newSweepSmall with a comment saying why a test may run in a refuted configuration while a deployment should not; sweep_inventory_of_one is now unreachable by default and its counter's job changes accordingly; and the housellm guard gets the smallest transcript the shipped preset will actually act on.

Also corrected docs/hosted.md, which this issue noted was wrong on five counts.

Still outstanding, and not closed by this: the live verification needs re-running. The previous run's evidence was collected against tail-only candidates, so its central observations — the cache read, the drop end to end, the inventory sizes — were measured on the broken shape. Tracked on #118.