extract_llm_sweep: the fallback path's goal makes it systematically refuse to drop

[amiddavid]

Found by live verification of #118, and it is the sharpest remaining asymmetry in that component: the cache read changes the answer, not just the bill.

Two runs, near-identical transcripts, twelve candidates each, opposite outcomes:

run I  fallback (no stash)       gates="sweep_kept=12 sweep_no_prefix=1"    declined, 0 dropped
run G  prefix ask (cache read)   sweep_offered=12 sweep_dropped=12          acted, saved 33,340

Same content. The prefix ask dropped all twelve; the fallback kept all twelve.

Cause

The fallback has no transcript — that is its definition — so relevance comes from conversationGoal, which opens with the FIRST user message. On run I that was the original read instruction, so every output read as needed for criterion (a), "the step you are on right now". The fallback cannot see that the task closed, because the turns where it closed are exactly what it does not have.

So the bias is structural rather than a tuning problem: a first-message goal describes the session's opening intent, and a spent-ness judgement needs its current one. It will systematically resolve toward keep.

Why it matters more than it looks

Keeping is the safe direction, so this is not dangerous — but it makes the fallback close to inert while still paying for a content-carrying prompt. That is the worst combination available: the expensive path with the conservative answer. And the fallback is on by default, precisely so the component keeps working on a session's first turn and whenever an entry has gone.

It also means the two paths are not comparable. Any measurement that mixes them will read as "the component sometimes acts and sometimes doesn't" when the real variable is whether the cache read happened.

Options

1. Use the LAST user turn, or the last few, for the fallback's goal — the same shape context: recent already uses (every user turn plus the last N non-tool messages). Cheap, and it removes the structural bias rather than compensating for it.
2. Give the fallback a short recent-turns excerpt rather than a goal string, accepting the fresh tokens for the turns that decide spent-ness. More expensive, closer to what the prefix ask actually sees.
3. Leave it and make the asymmetry visible — a counter distinguishing a fallback keep-all from a prefix-ask keep-all, so a run's numbers cannot silently average the two mechanisms.

(1) looks right and small. (3) is worth doing regardless of which is chosen, because without it the two paths remain indistinguishable in aggregate.

Not attempted here

This is a change to what the model is shown, on a component whose prompt effects are measured, so it is a yield decision rather than a bug fix. Flagged for the author. Reproduced live twice — the same asymmetry appeared in the previous verification round under the older mechanism, so it is not an artefact of the recent candidate-set change.

[amiddavid]

Fixed in 35cbbd5 on #118, taking option 1 with option 3 alongside it.

sweepIntent replaces conversationGoal on the fallback path. Same three parts, ordered current-FIRST and labelled, with the original instruction explicitly marked as possibly already satisfied:

MOST RECENT USER TURN — this is the step the agent is on now: …
THE AGENT'S OWN LAST STATEMENT — any next step it named is an obligation: …
THE SESSION'S ORIGINAL INSTRUCTION — MAY ALREADY BE SATISFIED; treat it as an obligation only
if some part of it is still outstanding: …

The labels map onto the contract's own criteria — (a) current step, (b) unfinished user instruction, (c) a next step the agent stated — rather than arriving as one undifferentiated blob the model has to infer structure from.

The original instruction is kept, not dropped, and that was the part worth thinking about: criterion (b) is an unfinished user instruction, and a standing "…and summarise them at the end" lives in exactly that message. Removing it would trade a bias toward keeping for a bias toward dropping — the direction that loses content the agent still needs. So its position is the fix, not its absence.

conversationGoal is left alone: leading with the task statement is right for extract_llm, which asks whether an output is relevant to the task. It is only wrong for a spent-ness question, so the sweep gets its own renderer rather than the shared one changing under another component.

Option 3 landed too, since I said it was worth doing regardless: sweep_fallback_kept_everything is now split from sweep_kept_everything. A keep-all means different things on each path, and without the split a run reads as "the component sometimes acts and sometimes does not" when the real variable is whether the cache read happened. It is also how this fix gets checked against real traffic rather than argued about — which matters, because the ordering change is a prompt change on a component whose prompt effects are measured, and I have not measured this one.

Revert-verified by swapping back to conversationGoal:

the opening instruction precedes the current step, which is the ordering that made the
fallback keep everything
the opening instruction is not marked as possibly satisfied, so the model has no way to tell
a standing obligation from a finished one

Plus a second test that a transcript whose only user turn IS the opening one does not render it twice under two labels — that would read as two independent obligations pointing at the same text.