feat(coref): co-reference-aware compaction

[amiddavid]

Implements co-reference-aware compaction — picking what to drop
at a threshold crossing by looking at back-references rather than at content or age — and measures
it end to end on LOCA.

Status: awaiting rebase, not review-ready as a diff. Much of what this branch built has since
landed on main independently as extract_llm_sweep (#118) and the general-fixes PR (#110). This
branch will be re-cut from main once three follow-up PRs land (#136, #137, and a collapse fix),
leaving only the parts main does not have. Read this for the result and the argument, not the diff.

---

The claim

Selective, reference-aware removal defers the blunt summary, and that deferral pays in latency and
operational cost even when the provider's cache has not expired — the case a purely
expiry-triggered sweep structurally cannot reach.

The evidence — iteration 021, paired, pre-registered

Two arms differing by exactly one inserted component, same frozen binary (cg-proxy-v19, sha256
ecc02f28…, code at fcf78cd), same 15 tasks × 5 seeds, run sequentially.

A  [format, toon, dedup, failed_run, cmdfilter,              extract, cachesplit, summarize]
B  [format, toon, dedup, failed_run, cmdfilter, extract_llm, extract, cachesplit, summarize]

	A (shipped pipeline)	B (+ merged)
wall clock	8,640 s	7,062 s	−18%
per task	115.2 s	94.2 s	−18%
total cost	$224.06	$228.66	+2% (parity)
errored runs	17 / 75	10 / 75	−7
summarize fired on	71%	56%	−15 pts
steps / run	45.7	32.7	−28%
tool calls / run	83.5	61.4	−26%
exact repeats / run	38.6 (46.2% of calls)	20.7 (33.7%)	−46%
requests	3,446	2,466	−28%
unique tokens removed	—	6,545,956
expand restored / unresolved	361 / 1	717 / 0
CG added latency / request	3,817 ms	6,170 ms	+62%

Merged costs ~2.4 s more proxy time per request and the agent still finishes 18% faster, because it
makes 28% fewer requests and summarize runs 58% less.

The repeat result is the one that most directly supports the thesis, and it reverses a worry held
earlier in this work: removing content was expected to make the model re-fetch. It re-fetches less.
The likely mechanism is that blunt summarization on 71% of requests destroys detail the model then goes
back for, while selective removal preserves what is still live.

What iteration 021 did NOT establish

Reward. ITT over all 75 pairs: 14.00 → 15.00 solves, +1.00, p = 1.0000 (task-clustered over 15
clusters, which governs: 2 better, 2 worse, 11 tied; per-pair sensitivity: 7 gained, 6 harmed, 62
unchanged; harm 95% upper bound 15.2%, which does not block).

That is a null, not a refutation. Eleven of fifteen tasks score 0.00 in both arms, so the whole
comparison rests on four tasks with two moving each way — LOCA had almost no room to show a reward
difference for any configuration. Establishing reward needs benchmarks with headroom, which is the next
step rather than a claim made here.

Two further limits, both recorded in the pre-registration before the numbers existed:

• Verdict coverage ended at 61% (2,114 answered of 3,461 offered), and arm B ran below its own
  pre-registered 50% checkpoint for the first third of the run (amendment 2). Arm B's numbers are a
  floor.
• Neither arm carried collapse or mask, so neither is a shipped preset, and summarize ran
  alongside in-place offloaders against the advice in config.go.

Why this branch must keep prefix_econ, given extract_llm_sweep on main

Main's sweep fires only inside a pre-expiry window — remaining = CacheTTLMs − IdleMs, positive and
under preExpiry — so the removal is free: the cache entry was about to be discarded anyway. That is a
better idea than pricing the break-even, because it replaces an unknowable estimate (turns remaining)
with an observation.

But it cannot fire in the case this branch is about. Arm B fired on context pressure — high context pressure on 1,747 requests, moderate pressure with fast growth on 109 — with prefix_econ pricing
each prefix mutation and refusing 385 of them. CacheTTLMs/IdleMs were not in the picture at all,
so sweeping() would have fired on none of iteration 021.

So the two triggers are complementary, and the rebased branch implements both:

trigger	the removal is	accuracy matters
pre-expiry window (on main)	free — the entry was expiring	barely
context pressure + S·T > 11.5·W (this branch)	paid — it forces a suffix re-write	a great deal

That sharpens what coref is for: on the free path a wrong removal costs an entry that was expiring
regardless; on the paid path you have spent a real suffix re-write, so the co-reference evidence is
what makes the paid trigger worth having at all.

What remains unique to this branch

• internal/coref — the tier-1 reference index. main has no index; its sweep carries an
  explicitly empty Evidence seam documented as waiting for one.
• components/offload/coref.go, corefstub.go — the component, and the residue reasoning.
• components/offload/prefix_econ.go — the second trigger's economics (see above).
• The experiment record — docs/experiments/loca/iter007–iter021 and docs/results/*, which is
  the evidence behind every claim here.

Superseded by main and being dropped: extract_llm_merged.go, internal/extract/bulk.go,
expand/restore.go (main's RepairToolResults is better — it has Unavailable() for aged-out stashes),
proxy/expandoffered.go.

The substrate measurement that started this

Reference density is a property of the workload, not a constant — three corpora disagree by a factor
of three, which is why the index is evidence for a decision rather than the decision itself:

	Claude Code (interactive)	UltraHorizon	LOCA-bench
unreferenced	23%	78%	95%
closed	15%	8%	0%
open	60%	13%	4%
…restricted to ≥20 later turns	21%	70%	70%

Measurement defects found along the way

Eight, in one component's measurement path — recorded because every one produced the identical
misleading signal, "the model declines to act", and four of them predate this branch:

defect	effect
coref prefix pre-filter	removed 149,681 candidates, leaving ~1 per call
llm_max_per_request	caps candidates in a one-call design
economic gate	prices each candidate against a whole call; suppressed 1,497 vs 224 through
any pinned floor disables the pressure trigger	fired on 72% of requests regardless of context (#134)
verdicts ÷ calls used as batch size	counts what was ANSWERED, not OFFERED
2048 output-token ceiling	reply cut mid-array on ~70% of calls
tool_choice: none	drove the model into prose — 0 of 6 verdicts (#137)
rig: marker check used the unescaped spelling	has_marker read 0.0% on every arm ever run

The judgement machinery was never the problem. A sampled reply shows the model reasoning correctly under
the criterion and simply saying so in sentences — which the contract itself calls a valid answer.

Next

1. Land feat(schema): validate message SHAPE statically, so the fifth summarize defect is caught offline #136, fix(adjudicate): declare a verdict tool and stop forcing tool_choice:none #137 and the collapse fix on main.
2. Re-cut this branch from main; port coref, fill the Evidence seam, implement the second trigger.
3. Re-run on LOCA and at least one benchmark with reward headroom — the axis iteration 021 could not
   speak to.

[amiddavid]

pls explain this sentence, perhaps add an example

[amiddavid]

Rewritten with a worked example rather than the assertion. It now walks turn 4 reads src/auth.py / turn 5 says "the bug is TOKEN_GRACE_SECONDS" / thirty turns later the agent is on tests — and shows that asked "is the turn-4 output still needed?", conversationGoal can only answer "the task is still about auth", which is true of every output and so decides nothing. The fact that settles it (the one value taken sits in turn 5, and turn 5 isn't going anywhere) is positional and backward-looking, which that signal cannot represent at all.

[amiddavid]

pls explain this paragraph in more details, I find it hard to follow, especially for a proposal document

[amiddavid]

Expanded into a bulleted walk-through of the two existing pieces and what each contributes: deterministic.go's important-key list is already an answer to "which parts of an output would a model carry forward?", and contain.go today checks a shrunken output is a subset of its original. The reusable idea is the second one run backwards — today it asks "is this compacted text contained in the original?", inverted it asks "is this span of the original contained in a later message?", and the same primitive becomes a reference detector. Same test, opposite direction: one validates a rewrite, the other measures reuse.

[amiddavid]

add an example column

[amiddavid]

Added an EXAMPLE column. Same reference at each tier: Tier 1 TOKEN_GRACE_SECONDS = 0 reappearing verbatim in an Edit argument; Tier 2 [{"ms":1200},{"ms":1800}] → "total latency is 3 seconds" (the 3 appears nowhere — it was computed); Tier 3 a directory listing → "as I saw earlier, the tests live beside the source", which is unmistakable to a reader and shares no token at all.

[amiddavid]

I'm not sure I fully agree. consider this scenario:
a tool output returned: { "name": "david", "id": 123, "address": "foobarbaz"}
, { "name": "osher", "id": 235, "address": "banana"} the agent said, I need to remember david 123 address.
the address itself wasn't coref, but the tool output is needed and cannot be removed.

[amiddavid]

i.e. doesnt this contradicts case B?

[amiddavid]

You're right, and this was the most valuable comment on the PR — it found a real bug, not just a wording problem.

I ran your exact example through the index rather than reasoning about it, and it's worse than you flagged. david, 123, foobarbaz are short lowercase words and a 3-digit number — precisely what the precision rules in §2 exclude — so the output yields zero trackable tokens. Zero novel tokens means zero references, which scored unreferenced, which is the class the default config cuts. So the shipped default would have deleted that output while the agent was still asking for the address.

Two separate defects, both now fixed in e7a2623:

1. Your conceptual point. "Any reference is a surviving copy" is too strong. The model referenced an anchor (david, 123) in order to point at a payload (foobarbaz) it never restated. An exact matcher can't distinguish an anchor reference from a payload reference — so closed can't rest on "referenced once, long ago" alone. That is now stated as the reason cut_closed ships off, rather than mere caution. It also inverts my §7 reading of used_frac: a low value is ambiguous, not evidence for case A, because "took the value, rest is chaff" and "took an anchor, still needs the payload" look identical.
2. The concrete one. refs == 0 conflated two opposite states — "introduced 200 identifiers, nobody touched one" (evidence of deadness) and "introduced nothing I can see" (absence of evidence). There's now an opaque class that is never cut at any setting. It is not a corner case: 8% of tool-output mass on interactive traffic, 20% on UltraHorizon, 40% on LOCA-bench — the last being 11 outputs averaging 22k tokens of exactly the record-dump shape you described.

Re-measuring dropped the headline unreferenced figures from 23/78/95% to 13/51/22%, and break-even from 15/30 to 9/30 sessions. Your counter-example is now a test case on both sides of the implementation.

[amiddavid]

I dont see improvement-plan in the docs

[amiddavid]

docs/results/improvement-plan.md does exist on main (verified with git cat-file -e main:docs/results/improvement-plan.md) and is in the mkdocs nav, so the link resolves on the published site. It's just not in this PR's diff, so GitHub can't render it as a clickable target here.

[amiddavid]

can u explain this more, I'm not following

[amiddavid]

Expanded. The short version: Tier 2 is a reference that arrived transformed, so by definition no substring match can find it — what's countable is a symptom. If a model turn states a numeric value appearing nowhere in any earlier message, it computed that number from something, and that something was almost certainly a tool output. 2% of turns look like that on interactive traffic, which is why a zero-LLM first version is viable.

Two caveats now stated, and the second is a self-inflicted one worth knowing: it's a lower bound (only numeric transformations leave this trace — reworded prose is invisible), and tightening the identifier rules also blinded the proxy, since bare numbers now need 5+ digits and most computed values are small. So its 0% on LOCA means "none among tokens the tokenizer still accepts", not "none" — on a corpus with 0% closed and 40% opaque, the honest reading is that Tier-2 references there are common and simply unmeasured.

[amiddavid]

if it a recent turn, it might not had the chance to be referenced, dont we need to guard from this ?

[amiddavid]

Good catch, and no — there was no guard, which was a real gap. An output near the tail has had no chance to be referenced, so scoring it as unused would make a batched pass preferentially cut the most recent context, which is the worst possible choice. mask avoids this with keep_recent; coref had nothing.

Added min_later_turns (default 8): an output with fewer model turns after it is treated as open regardless of everything else. Worth noting what the state was before — the measurement had bounded this bias (LOCA's raw 95% fell to 70% when restricted to outputs with 20+ later turns) but nothing in the component guarded against it. Bounding a bias in a report is not the same as not having it in the code.

[amiddavid]

dont tune it, but still matters?

[amiddavid]

Fair — that phrasing was self-contradictory. Rewritten to what's actually true: closed_dist is load-bearing but flat. Set it to 0 and the closed class stops existing, so it certainly matters; but anywhere in 4–40 gives the same answer within 2–3 points, so there's no return on tuning it. Leave it at the default and spend the effort on open_reps, which moves the answer 18 points across the same kind of range.

[amiddavid]

love this. 💌
though it depends how much is being cut isn't it? and for the determinsitic one, its cheap to calculate and can tell u how much deferring is happening.

I would also appreciate some thought on what it means for larger context windows that are now more and more frequent.... up to 1M

[amiddavid]

Thanks — and both of your points landed in the doc.

On "it depends how much is being cut": yes, and more sharply than I'd written it. The agent-compaction prize is a step function, not a slope — you either drop below the threshold or you don't, and cutting 90% of what was needed to get there is worth nothing. Which argues for sizing the batch against the threshold distance, something min_batch_frac cannot currently express. Noted as a limitation.

On deterministic measurement: agreed, and it's the cheapest real metric available here — compare the API-reported usage against the documented compaction threshold and count the turns of headroom the cut bought. No benchmark scoring, no seeds, no LLM judge. It isn't in the metrics yet; it should be.

On 1M windows — I worked this through and the answer surprised me. Break-even is scale-invariant. Rearranged, S × T > 11.5 × W is T > 11.5 × (W/S) — it depends only on the ratio of rewritten suffix to cut mass, never on absolute size. A 1M transcript with the same density of cuttable mass needs the same T. So a bigger window neither rescues nor damns the token economics; it only moves when the trigger fires. What improves the ratio is cutting a larger share of what lies after the shallowest cut — an argument for cutting deep and rarely, not for cutting more.

Three things do genuinely change, now a table in §7 of the cheat sheet: cache-read becomes the entire bill (so coref is a cost play at 1M rather than a fit play — the strongest argument for it there); the agent's own compaction recedes to ~967k, making that prize rarer but much larger; and the index cost scales linearly, so an incremental per-session index stops being an optimization and becomes a requirement.

[amiddavid]

since I'm not familiar with this repo yet, I would appreciate if you can in a comment explain this more

[amiddavid]

Expanded both from first principles rather than by name.

TailOnly is a helper on Ctx answering "may I safely modify the message at index i?" It returns false for anything the provider has already cached, because editing cached content breaks the prefix hash and forces a cache-write of everything after it. Every other age-based offloader (mask, failed_run, collapse) consults it and declines. coref deliberately ignores it — reaching into the cached prefix is the point, since by the time a session crosses the threshold all the mass is back there — which is exactly why its spend is budgeted rather than forbidden.

repairLostFreeze needs the background first: an offloader freezes its replacement text against the original's content hash and replays it every turn so the bytes stay stable. If the store drops that record (TTL, eviction), it would normally decline to act at depth — but then the message reverts to full text, which is itself a prefix change. So mask and failed_run may re-derive even deep in the prefix: their replacement is a pure function of (content, config), so re-deriving reproduces byte-for-byte what the provider already cached. coref must never do this, because its decision depends on the whole transcript — re-deriving against a longer one can yield a different class and different bytes, the precise flip the repair exists to prevent.