offload: the replay depth-gate bypass is justified per-message but keyed per-content, so a replay can splice at depth into uncompacted content

[amiddavid]

The mismatch

extract_llm's same-session replay bypasses the cache-tail depth gate — the lookup is at extract_llm.go:890 and the gate at :932 — and the bypass is justified in the block's own comment:

SAME-SESSION replay first. This session already sent these compacted bytes on an earlier turn, so the provider's cached prefix holds the COMPACTED form and replaying it is byte-identical — which is why it is safe at any depth, cached prefix included.

That justification is about this message. The lookup is keyed on content (extract.ContentKey), not on position. So the same content appearing at a different position also takes the bypass — and at a position that was never compacted, the provider holds the original, so splicing the compaction there changes bytes inside the cached prefix and forces a cache-write of the suffix. That is the cost the gate exists to prevent, at ~11.5x a cache read.

extract_llm_sweep's replay path has the same shape.

Why the window is narrow, and what widens it

Mostly self-limiting, which is why this is filed as a question rather than a defect:

• The common case is not affected. The client re-sends the original transcript every turn (see collapse.go, failed_run.go, agentdiet.go — "the agent re-sends the original each turn"), so a message compacted last turn arrives unmarked again at the same position, and the replay re-derives the same bytes. The provider holds the compacted form there. Nothing flips. This is the case the bypass is for.
• A genuinely new occurrence arrives in the tail. It is appended, so its index exceeds MaxCachedIdx, and TailOnly (components/component.go:414-419) already permits acting there — the bypass buys nothing. By the time that position is at depth it was compacted on its arrival turn.

So reaching "content at a position that is already at depth and was never compacted there" needs one of:

1. the boundary moved — an agent self-compaction resets the cached-prefix boundary (compaction_resets counts these);
2. the component did not act on the arrival turn — llm_every_n_requests throttle, llm_max_per_request cap, below min_tokens, a model decline, or (after fix(store): give rewind payloads their own reserve, and refuse a removal whose original cannot be stored #188) a refused reserve slot;
3. a different component wrote the record — see offload: extract_llm and extract_llm_sweep share the cg:res: namespace with no component tag, so a depth-gate bypass can rest on another component's decision #197; the reuse is content-addressed and cross-turn, so extract_llm can encounter an id the sweep ruled on at a position extract_llm never touched.

Evidence available, and what it does not show

Iteration 024 measured 364 extract_llm replays in arm B (0 fresh calls, 99.2% cache hit rate, 0 in arm A) — see docs/experiments/loca/iter024/results.md §2. That is the total replay count, not a count of fresh-position replays, and the run cannot separate them:

• an argument that all 364 were fresh-position, on the grounds that the marker skip precedes the lookup so replays land on unmarked content, does not hold — unmarked is the normal steady state at every position on every turn, because the client re-sends originals;
• cache_read_tokens, cache_write_tokens and fresh_input_tokens read 0 in every arm-seed of that build — unpopulated rather than measured — so the direct signal is missing.

So the size of the affected subpopulation is unknown, and could be zero. Filed to get it measured rather than argued.

What would settle it

• Per replay, whether that (content id, position) pair had been compacted on an earlier turn. fix(extract_llm): no log record when an extraction call is made, so its 59s latency and negative net value have no per-request trace #177's per-call record is the closest existing instrument.
• compaction_resets non-zero alongside replays would flag route 1 specifically.
• With the cache-token fields live, arm-B cache-write volume against arm A (where no replays exist) bounds the cost directly.

Possible fixes, if it turns out to be non-empty

1. Key the replay decision on (content, position) rather than content alone — most faithful to the justification, and it gives up cross-position reuse, which is much of what the channel buys.
2. Record, per content id, the positions at which it has been sent compacted, and bypass the gate only for those. Preserves legitimate reuse; costs a small set per id.
3. Let the bypass apply only when compaction_resets has not moved and the component acted on the arrival turn — cheap, and it targets routes 1 and 2 without touching the common case.

How it was found

Reviewing #188. Raised as the residual after the byte-flip concern in #197 was closed by execution — a cross-component replay writes byte-identical text to what the writing component would have written, so sharing is not the hazard; this positional mismatch is what remains, and it belongs to extract_llm's own replay independently of any sharing.

[amiddavid]

Why the sizing recipe cannot be applied retroactively

The section above names what would settle this — per replay, whether that (content id, position) pair had been compacted on an earlier turn. Worth recording that iteration 024's artifacts cannot supply it, so nobody spends time looking:

• the logs carry no content ids at all in that build. That is why the question was approached by reasoning about control flow rather than by measurement, and it is how the invalid inference below arose.
• cache_read_tokens / cache_write_tokens / fresh_input_tokens read 0 in every arm-seed — unpopulated, not measured as zero — so the indirect signal is missing too.

So this needs a fresh run with instrumentation, not a re-read of existing data. #177's per-call record already carries content_key, which is the anchor the recipe needs; the missing half is the position.

An invalid inference, recorded so it is not repeated

docs/experiments/loca/iter024/results.md briefly claimed that all 364 replays landed at fresh positions — reasoning that the marker skip precedes the cache lookup, so replays must be on content the marker skip did not catch. Withdrawn in ae2e5ed.

The step is wrong because unmarked is a property of what the client sent this turn, not of the content's history. The client re-sends its own original transcript every turn, so unmarked is the steady state at every position on every turn — the marker skip only excludes messages where our marker survived in the client's own copy. Re-derivation at the same position is therefore the modal case, it is exactly what the bypass exists for, and there the provider already holds the compacted form, so nothing is re-written.

A cost hypothesis was derived from that claim — that the 57% recovery might be paying a cache-write per replay — and it does not follow. It is withdrawn with the inference.

This matters for reading the issue: the 364 is cross-component replay volume, and the fresh-position subpopulation is unknown and may be empty.