store: a saturated rewind reserve is released only by the TTL, so one busy period can zero out savings process-wide for ~2.8h

[amiddavid]

The remaining gap

#188 bounded the rewind reserve properly — an entry cap, a byte budget (stash_max_bytes), and an evictable floor so the exemptions cannot consume the whole cache — but a slot is still released only by the TTL, and the store is a single process-wide instance (cmd/context-guru-proxy/main.go builds cfg.NewStore() once).

So a busy period can hold the reserve until entries age out. With the default sliding ttl_seconds: 10000 that is up to ~2.8 h after the last touch, and while the reserve is saturated every new removal is refused across every session sharing the process. The failure mode is not broken markers — #188 fixed that — it is savings falling to zero mid-run.

What #188 deliberately did not do

Per-session partitioning was considered and declined. The reasoning, recorded so it can be revisited rather than re-litigated: no session ever reads another's payload (offload.OwnsKey enforces that), so what is shared is the budget, not the data — and a fair-share cap needs a share number as arbitrary as the max_entries default it would replace, plus an ownership rule for a content-hash key two sessions can legitimately both stash. Sizing against the real resource (bytes) was judged the more honest fix for the same failure.

That narrows the window; it does not close it.

How it was found

Raised in the #188 review (finding on store.go:307) and answered there as partially addressed. Filed so the open half survives the PR.

The options, with their trade-offs

1. Age-based last-resort eviction. Under saturation, allow a payload past some age to be evicted, counted as an explicitly reported broken promise (a new counter, or cg_stash_missing_total). Restores liveness at the cost of reintroducing bounded and measured irreversibility. This is the direct opposite of #188's thesis, which is why it needs a decision rather than a patch — but "bounded and reported" is a different thing from #187's "unbounded and silent".
2. Per-session share cap. No session may hold more than a fraction of the reserve; a runaway session refuses for itself while others keep offloading. Bounds the blast radius without any irreversibility. Costs the two design problems named above.
3. A shorter TTL for payloads specifically. A payload's life is only needed while a marker referencing it can appear in a live request, which is a shorter horizon than a frozen decision's. Cheap, but shortens the window in which a long-idle session can still expand.
4. Release on session end. The honest version of (3), but there is no explicit session-end signal — sessions are derived per request — so it reduces to an idle timeout.

What should gate the choice

A re-run of iteration 024 on #188's code, reading stash_refused, stash_live/stash_capacity and stash_bytes/stash_max_bytes. If refusals stay at zero the reserve is correctly sized and this is theoretical; if savings collapse mid-run with refusals climbing, the measurement says which of the four is needed and at what threshold. Picking now would be another guess at a default.

[amiddavid]

A fifth option, from the #188 review: gate exploration on reserve health

Added here rather than decided, because it is the same question this issue asks — what a component should do while the reserve is saturated — and it should be chosen alongside the others with the re-run's numbers in hand.

The problem it solves. #188 made a declined splice record no ratio observation, which is the only one of three choices that does not lie: crediting the achieved ratio authorises calls whose output is discarded, and crediting 0 asserts the workload is incompressible, which is false and would also suppress calls that would land once the reserve frees.

The cost is that r.total then never advances, so ratio() stays pinned to its prior and exploring() keeps granting maxExploreCalls instead of self-terminating once the sample is large enough.

And that cost is not per-session-bounded, which is the easy thing to assume. The store is one process-wide instance, so saturation is correlated rather than independent: when the reserve is full it is full for every session at once. The spend is maxExploreCalls × every concurrent session, repeating for as long as the episode lasts — and with a sliding 10,000 s TTL an episode can run for hours. The deployments paying it are the under-provisioned ones this issue is about, which are the least able to absorb it.

The option. Gate exploring() on reserve health rather than on r.total. The tracker keeps refusing to learn from declined splices, so it stays honest; exploration stops because the reason for non-advancement is known to be refusal rather than absence of evidence. stash_refused already distinguishes exactly that, and stash_missing (added in #188) keeps the distinction clean — a refusal means removals were declined, a missing payload means a marker dangled.

That turns "the tracker cannot advance" from an invisible cause of repeated spend into a condition the component can read and act on.

Needs no new plumbing — both counters exist and are exported. What it needs is a decision about the threshold and about whether a component should self-suppress on reserve health at all, which is why it belongs on this list rather than in #188.

Credit: raised by the reviewer on #188 (thread on extract_llm.go's ratio comment), where the fleet-wide cost of the current behaviour was also worked out.

[amiddavid]

Addressed in #204, with the choice among the five options made from the code rather than from the re-run.

The re-run is not the gate it looked like. Iteration 024 ran on a commit predating #188, so stash_refused has never been observed at all, and this issue's gating plan is circular — the re-run needs #190 either chosen or deliberately left unfixed and instrumented.

Option 3, and the objection recorded against it here is not what the tree does. This issue says a shorter payload TTL "shortens the window in which a long-idle session can still expand". It does not, because a payload — unlike a frozen decision — is re-derivable from the request in flight: every offloader replays its decision on every turn regardless of the cache-tail gate (mask.go:79), and that replay calls commitRefresh, which re-stashes the payload from the message text it just read. So a live marker's payload has its deadline slid every turn, and one already reclaimed is re-created on the request path — before the request goes upstream, and therefore before any expand call in the response could ask for it. The horizon a payload needs is one inter-turn gap, not one session.

That makes the trade-off different from the one weighed here: no irreversibility (unlike option 1), no arbitrary share number (option 2), no session-end signal (option 4). stash_ttl_seconds defaults to 1800 s, capped at ttl_seconds, and the hangover shrinks from ~2.8 h to ~30 min. #204 has the mechanism, the exposure it leaves (a long run of bypassed turns, which run no pipeline and so no refresh), and an end-to-end test that fails if re-derivation ever stops holding.

stash_revived is the measurement this issue wanted, and it arrives on the first ordinary run rather than needing an instrumented arm: revived tracking expired means reclamation is being absorbed at no cost; stash_missing moving is the reserve being genuinely undersized.

Option 5 is deliberately left here — gating exploring() on reserve health is a different cost in a different package and needs its own threshold decision. #204 shortens the episode it is about but does not close it, so this issue should stay open for it after #204 lands.