expand: an expand permanently un-compacts its content and costs one prefix flip, undocumented and uncounted; in-place replacement was never built

[amiddavid]

What the code actually does today, since it is documented nowhere

Two mechanisms were apparently considered when expand was designed. Only one is in the tree, and its cross-turn behaviour is not what the other's rationale assumed.

The in-turn path. The model calls context_guru_expand; the proxy intercepts it (proxy.go:~1650), and expand.Continuation (expand/response.go:69) appends two messages to the upstream request body — the assistant message carrying the tool_use, then a user message with one tool_result block per call holding the original content — then re-invokes upstream, up to maxExpandRounds (3). The content lands at the end, so the cached prefix is untouched during the turn. That part works as intended.

It does not persist as its own turn. On the intercepted path the client never sees the tool_use/tool_result pair — the proxy answers it and returns only the final assistant response. So the next turn the client re-sends its own transcript without the expanded content. The expansion is transient.

What persists is a flag, not the content. offload.MarkKeptVerbatim writes cg:keep:<content-hash>, and thereafter every offloader skips that content (skipReduce → isKeptVerbatim). So from the next turn the original message goes upstream in full, at its original position.

The consequence, measured

Turn N sent compacted bytes at that position; turn N+1 sends the full original there. That is a change inside the cached prefix, so it costs a cache-write of the suffix — at ~11.5x a cache read, which is the cost the tail gate exists to avoid everywhere else.

Measured directly (mask, keep_recent: 0, min_tokens: 20, one 4,200-token tool output):

turn 1 sent 73 tokens (compacted)
turn 2 sent 4,200 tokens (full)   gates=map[marker_or_kept_verbatim:1]

So "expand does not hurt the cache" is true within the turn and false across it. The flip happens once per expanded content, on the turn after the expand.

That is arguably correct behaviour — re-compacting would loop the agent straight back into another expand, which is exactly what cg:keep: exists to prevent, and one cache-write is cheaper than an unbounded expand loop. But it is a real cost that is currently undocumented, uncounted, and invisible: no counter distinguishes "prefix flipped because the agent expanded" from any other cache-write, so an operator cannot see it and a benchmark cannot attribute it.

The alternative that was considered and does not exist

replace the marker with the content in place, and delete the expand tool call from the earlier messages.

Nothing in the tree does this. I checked expand's whole exported surface — Inject, HasTool, ResponseCalls, Continuation, Marker, HasPlaceholder, ParseMarkers, Resolve, ToolDefRaw, RepairToolResults, Unavailable, AggregateSSE, Unresolved — and there is no in-place replacement and no stripping of prior expand calls.

There is no switch between the two. inject_expand (auto / always / never) controls only whether the tool is advertised, not how a call is answered.

The one path where content does persist

repairExpandErrors (proxy.go:1296). When interception structurally cannot work — a client tool batched alongside expand, maxExpandRounds exhausted, an event stream that will not reconstruct, a non-Anthropic stream, or a bypassed turn carrying older markers — the client did see the call and answered No such tool available. The proxy then repairs that tool_result on the request side on every later turn, so the content does live in the client's transcript.

Worth being explicit: that is a fallback for a failure, not a mode. It is also the only configuration in which the expanded content survives as its own turn.

What is worth deciding

1. Document the cross-turn behaviour, since neither docs/how-to/recover-context.md nor docs/design.md says that an expand permanently un-compacts that content or that it costs one prefix flip.
2. Count the flip. A counter for "prefix flipped because content was expanded" would make it attributable. Note this is the same split-the-ambiguous-counter shape as proxy: an unrecognised usage dialect is indistinguishable from no usage, and degraded silently for 4,015 requests #200: today an expand-induced cache-write is indistinguishable from any other.
3. If in-place replacement is wanted, the gating signals already exist. It was suggested it should apply only on a cache miss or after a serious compaction — ColdCache / TailOnlyCold already detect a turn whose prefix cache has provably expired, and compaction_resets counts turns where the agent's own compaction moved the boundary. So the condition is expressible with machinery that ships; only the behaviour is missing.

How it was found

Answering a question about which of the two designs is live. The measurement above is from a scratch test, not committed. Related: #199 (a different per-content-versus-per-message mismatch in the replay bypass), #200 (the counter-splitting pattern).

[amiddavid]

Examined, not built. Three findings, and the middle one shrinks bullet 2 to almost nothing

Read the tree against this issue's three decisions. The behavioural description holds — I confirmed the flip mechanism at proxy.go:1306 / proxy.go:1658 (MarkKeptVerbatim) → state.go:205 / state.go:319 (isKeptVerbatim) → skipReduce, and there is indeed no in-place replacement anywhere. Three things it does not say.

1. The split bullet 2 asks for ALREADY EXISTS in the tree, at 3 of 11 sites

kept_verbatim_after_expand is already a distinct gate label, raised by extract_sweep.go:296, extract_llm.go:879 and failed_run.go:129. The other eight offloaders raise the conflated marker_or_kept_verbatim for the same condition:

mask.go:86 · cmdfilter.go:151 · collapse.go:123 · linecap.go:123 · extract.go:68 · dedup.go:54 · skeleton.go:157 · readlifecycle.go:171

Those two reasons want opposite readings. "Already carries a marker" is benign — the content is already compacted, nothing flips. "The agent expanded it" is the flip this issue is about. Gates reach /stats per component (metrics.Snapshot.Gates), so this is a published counter that reports both, which makes it the fourth instance of the rule stated in #200 — and the peer's kept_verbatim_after_expand figures (9,478 arm B vs 3,522 arm A, seeds 1–3) are measuring only the three components that already split it.

So bullet 2 is not "add a counter". It is: make skipReduce return which reason it found, and let the eight sites raise the label that already exists. One helper, eight one-line call sites, no new surface.

// today
func skipReduce(c *components.Ctx, content string) bool {
	return expand.HasPlaceholder(content) || isKeptVerbatim(c, contentKey(content))
}

2. The replay path's flip is counted NOWHERE, by any label

reapplyFrozen (state.go:203-206) returns early on kept-verbatim with no rep.Gate at all:

if isKeptVerbatim(c, contentKey(content)) {
	return nil, 0, false // agent expanded this; replaying the collapse would loop
}

The caller then falls through to skipReduce and raises the conflated label, so the event is incidentally recorded — but as a declined new reduction rather than as an established compaction that has been abandoned, which is what actually happened and is what costs the cache-write. Turn N+1's flip is the one an operator would want attributed, and it is attributed to the wrong thing.

3. Bullet 2 has a hard dependency this issue does not state

The flip's cost is a cache-write of the suffix, and cache_write_tokens, cache_read_tokens and fresh_input_tokens all read 0 in every arm-seed of iteration 024 — the exact fields the flip would show up in. So a flip counter can say how often, and nothing can say what it cost, until the usage dialect gap is closed. #200's shape record is the unblocker; #205 lands it. Worth adding to this issue explicitly, because "count the flip" reads as self-sufficient and is not.

Also worth recording: the once-per-content claim in this issue is from a scratch test, not a run, and it could not have been observed in a real run for the reason just given.

Recommendation: split this issue

• Now, small: bullets 1 and 2 — document the cross-turn behaviour, and finish the gate split that is already 3/11 done, including the uncounted reapplyFrozen early return. No new counters, no new config, no behaviour change.
• Separately, a feature: bullet 3's in-place replacement. It should not gate a run, and the gating signals it would need (ColdCache / TailOnlyCold / compaction_resets) are all present, so nothing about deferring it loses ground.

I have not touched either half. #190 is in #204 and #200 in #205; say the word and I will take the small half of this as a third PR.

[amiddavid]

The conflated gate has now been measured, and it is large enough that combining the two labels does not survive

The coref validation session measured both gates per request over five seeds of iteration 024 (recorded in docs/experiments/loca/iter024/results.md, 794baef):

gate	arm A	arm B	B/A
kept_verbatim_after_expand (3 components)	6.63	12.68	1.91
marker_or_kept_verbatim (the other 8)	13.09	20.40	1.56

The conflated gate is roughly twice the size of the clean one. So the ambiguity is not a rounding detail — it is the larger of the two populations, and it is the one nobody can decompose.

A combined figure is not an expansion rate, and this is the part worth recording here rather than only in a session: adding the two rows measures expansion plus already-carries-a-marker, and on this pipeline the second term is almost certainly the dominant one. The mechanism, from the control flow:

• mask, collapse, cmdfilter, skeleton, readlifecycle and agentdiet try reapplyFrozen first and continue on a hit, so a message that component itself compacted never reaches the gate. What does reach it is content compacted by a different component — every turn, for the rest of the session.
• dedup (dedup.go:53), extract (extract.go:67) and linecap (linecap.go:123) have no reapplyFrozen at all, so their gate fires for any placeholder-bearing content whatever produced it, including their own earlier work.

With eleven components in one pipeline, cross-component marker presence is a per-turn, per-message population that grows with the transcript. Expansion is an occasional event. Summing them moves the ratio toward the marker-presence ratio, not toward the truth — which is why the 1.91 from the split gate is the best available estimate of expansion (a floor), and the combined 1.68 should not be read as one.

So the split is load-bearing for measurement, not only for alerting. A published experimental figure has now been corrected twice off this gate — once for being split-only, once for being unnormalised — and it is still a floor for the reapplyFrozen reason above. That is the strongest argument yet for the eight-site change in my previous comment, and it is now measured rather than argued.

One addition to the fix as I described it: because dedup, extract and linecap never consult a frozen decision, splitting skipReduce's return value gives those three a clean kept_verbatim_after_expand for the first time, which is where the largest share of the currently-invisible expansion signal is.

[amiddavid]

Housekeeping on the figures above: the combined 1.68x has since been withdrawn upstream (7f68857 in docs/experiments/loca/iter024/results.md), for the reason given in my previous comment — it summed the clean gate with the conflated one, so it measured expansion plus cross-component marker presence rather than expansion. I referenced it only to reject it, but a reader going to look for it in results.md will no longer find it.

The two rows in the table stand, and 1.91x from kept_verbatim_after_expand is the estimate — still a floor, for the reapplyFrozen early return with no gate at all.

The per-component spread (cmdfilter 3.32→6.17, dedup 3.32→6.33, collapse 3.14→3.27) has also been relabelled upstream as un-attributable rather than as evidence about expansion, which is the right reading: it is variation in the conflated gate, so it cannot be assigned between the two causes — and for dedup it cannot be expansion-driven in any clean sense, since dedup never consults a frozen decision.

Which leaves the argument for the eight-site split in a stronger place than when I filed the first comment, and worth restating plainly: the current counters have produced a real signal nobody can interpret. That is a measurement defect, not a labelling one.