feat(coref): co-reference evidence and an economic trigger for extract_llm_sweep — +8 tasks, 0 worse, p=0.0078 - #186
Open
amiddavid wants to merge 23 commits into
Open
feat(coref): co-reference evidence and an economic trigger for extract_llm_sweep — +8 tasks, 0 worse, p=0.0078#186amiddavid wants to merge 23 commits into
amiddavid wants to merge 23 commits into
Conversation
The branch this replaces carried 97 commits against a merge base main has since moved 295 files past. Three of its general fixes landed separately (#136 schema shape validation, #137 the adjudicate verdict tool, #138 collapse's byte gate), and main absorbed the prefix-ask infrastructure independently as extract_llm_sweep. What is left is the part that was only ever about co-reference, re-cut onto main rather than rebased: internal/coref (the index), the coref component, and prefix_econ (the shared cache-write price). The pre-recut history is preserved on origin at archive/coref-compaction-pre-recut (f31b451) -- the experiment record and the refuted arms are evidence, not clutter, and a force-push must not be the only copy of them. Two things main required that the branch predated: RegisterFields for coref. main added a test asserting every registered component declares exactly its configurable YAML keys, and its motivating story is this branch's own: an account whose extract_llm ran 251 times and acted zero times because the two keys that decided it were not on the settings page. coref had a constructor and no descriptors, so it was that bug waiting to happen. Each hint carries the measurement behind the default rather than restating the field name -- including the two thresholds (closed_dist, open_reps) that are explicitly NOT measured yet, which is why cut_closed ships off. MarkKeptVerbatim lost its session argument. The branch had made keep-verbatim marks session-scoped; main's are global, so a mark written by one session exempts the same bytes in every other session sharing the store -- preferentially on content that recurs across sessions, which is exactly the content most worth cutting. That is a real lost-savings leak, but fixing it is a store key-format change in state.go affecting every offloader plus a read-both-shapes migration for marks already on disk, so it is filed separately rather than smuggled in here. The two tests that prove the leak are SKIPPED with their bodies preserved commented, not adapted: an adapted body would compile, read as a real test, and assert nothing, because both sessions would share one global mark. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…trigger
main left Evidence as an explicitly empty seam ("A SEAM, deliberately empty on
`main`. There is no co-reference index here") and left a note at the candidate
append site saying that when PR #80 brings index-driven selection, "the index's
verdict belongs in the prompt as EVIDENCE for the model to weigh
(extract.AdjudicationItem.Evidence), never as a gate that pre-decides the answer."
This fills that seam on exactly those terms, and adds the second trigger.
THE EVIDENCE IS NOT A FILTER, which is the whole lesson of the
prefix_still_referenced thinner: it removed 149,681 candidates and left about one
per request, silently turning a bulk adjudication arm into the per-output shape
refuted at 6% live-kept -- while the arm reported itself as bulk throughout. It
was self-defeating twice: it starved the comparison, and it meant the model only
ever saw what the index had already judged spent, destroying the veto on the
index's exact-match blind spot that the mechanism exists to provide. So the
record goes into the inventory line and the candidate set is untouched.
The contract gains a paragraph, conditional on an item actually carrying
evidence. main's seam comment names one hazard -- a prompt teaching the model to
read counters the prompt never carries. The converse is equally real: counters
carried with no explanation invite the model to invent a reading of them. Tying
the paragraph to the data avoids both. It frames the index as a fallible witness
in its own words, because the index matches text EXACTLY: reuse in transformed
form is invisible to it, that blind spot is precisely what the model is here to
cover, and a model told the index has decided has nothing left to contribute.
THE SECOND TRIGGER IS THE CLAIM OF THIS BRANCH. main sweeps only in the
pre-expiry window. Neither trigger contains the other: pre-expiry fires on the
clock and cannot fire at all on a session whose cache keeps being refreshed --
the long agent run with the most to save -- while econ fires on mass and cannot
know how much time is left, so it must clear S*T > 11.5*W first. The price lives
in prefix_econ.go, shared with coref rather than restated, because two components
pricing the same cache-write differently would be two answers to one question.
S is an upper bound and this says so: it is the inventory's whole mass, but the
model drops only some of it, and how much is unknown until after the call the
test is deciding whether to make. W leans the other way (an unknown cache
boundary is assumed to be the whole transcript). Neither is calibrated, so
prefix_rewrite_repaid / prefix_rewrite_not_repaid is what makes the split
observable rather than assumed.
Both knobs default OFF, and a test asserts a disabled trigger reports no
decision at all -- a gate that fires when the feature is off reads, on a
dashboard, as a feature that is on and failing.
VACUITY: all five mutations were verified to fail the right test -- reverting the
trigger, reverting the evidence fill, removing only the contract paragraph,
flipping the default on, and INTRODUCING the pre-filter. That last one initially
passed, which is the finding: sweepReqStocked produces `novel=0 refs=0
later_turns=0` for every record, so a filter keyed on references removes nothing
there and the guard could not fail. Hence sweepReqCoref, which contains an output
later turns reuse character-for-character (refs=2) and one never mentioned again
(refs=0). The guard now catches the thinner at 11 candidates against 12.
Signed-off-by: David Amid <david.amid@il.ibm.com>
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
The evidence base for this branch's claim, re-cut with three corrections main's conventions require. LINKS INTO docs/proposals/ ARE NOW PROSE, 26 of them. main excludes proposals/ from the site with a documented rationale -- design rationale addressed to whoever changes the code next, "useful in review, actively confusing on a page a user reaches looking for how to configure something" -- and its own published pages follow that by naming such files in backticks rather than linking them (components/extract_llm_sweep.md does exactly this for sweep-adjudicator.md). Every link from a published page into an excluded one is a `mkdocs build --strict` failure, so these were converted rather than left to CI. Two broken links FIXED rather than carried over: iter014's preregistration linked ../iter013/PREREGISTRATION.md, which does not exist (iteration 013 has only a results page), and iter010's results linked iter011/PREREGISTRATION.md without climbing out of its own directory. Both predate this port; both would fail strict. min-tokens-vs-economic-gate.md updated in place, because its subject moved. It was written when one-call adjudication was a `selection_mode: merged` option on extract_llm, proposed here and not yet on main; it shipped instead as a separate component, extract_llm_sweep (#118), which is now on main. The argument does not change -- extract_llm_sweep makes one call per request, so pricing a candidate against the cost of a whole call is wrong there by the same two to three orders of magnitude -- but the page is no longer a warning about an unmerged option, it is a description of an interaction between two shipped components, and it now says so. The nav gains all 29 experiment pages, not the 8 the pre-recut branch listed. main's nav comment states the invariant ("Every page under docs/ appears exactly once") and iterations 007-021 were reachable only by URL, including iteration 021, the one this branch's claim rests on. Preregistrations are left exactly as written, including the one that references a design this port supersedes. They are dated records of what was predicted before a run; editing them to match what shipped would destroy the only thing they are for. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
They said "filed separately" before the issue existed. A skip reason that names no issue is a skip nobody can act on. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…shipped baseline Written before the run, and before the binary is frozen, which is the control iterations 014, 016 and 018 lacked and the reason none of them can be compared to each other. It opens by stating what iteration 021 did NOT establish, because that decides what this one may claim: 021 was a null on its primary endpoint (clustered p = 1.0000, 14.00 vs 15.00 solves, +2% cost), its pre-registered reading for that outcome was "close merged", and `coref` was in neither of its arms. What 021 did move was operational -- errors 17->10, summarize 71%->56%, requests -28%, 6.5M tokens removed with 0 unresolved. So this iteration measures the operational result and carries reward as a HARM GATE only, with the blocking bound declared in advance per iteration 007's failure. Three arms, alternatives rather than a stack. B turns on the sweep's evidence and econ_trigger -- PR #80's merged design, where the index informs and the model keeps the veto. C adds the coref component, the zero-LLM cutter. They are not combined because coref leaves markers and the sweep skips marked content, so coref upstream would hide its own cuts from the model: structurally the prefix_still_referenced thinner that left about one candidate per request and silently turned a bulk arm into the per-output shape refuted at 6% live-kept. Band moves to 32k. At 128k, 11 of 15 tasks scored zero in BOTH arms, so nothing could have been detected there whatever the configuration. Iteration 008 measured 52.7% at 32k against 33.3% at 64k, and signed-rank power peaks near a 50% base rate. Stage 0 runs seed 42 only, 15 runs per arm, which 008 also validated (52.7% over 75 configs against 53% on state0 alone). The primary endpoint is turns-and-wall-clock SPLIT BY RUN OUTCOME. Iteration 021's -28% requests is the strongest number it produced, but its own text attributes it to fewer runaway sessions and its arm B also errored 7 fewer times -- so the drop may be "failed less" rather than "solved faster", and an end-user latency claim needs exactly that split. 021 could not make it. Two shipped defects are carried deliberately and recorded so they are not mistaken for errors in this file: #134 (pinning min_tokens/trigger makes shouldFire return true unconditionally, so housellm's tail pass fires every request -- and since it leaves markers the sweep skips, it can progressively starve arm B) and #120 (the tail floor is 3000 where the comment above it derives 8000 and calls anything below measurably a loss). Both are identical in all three arms, so neither can bias B-A or C-A, and both become measured endpoints instead of config guesses. The run does not start until a four-part pre-flight passes, including that the sweep is actually offered candidates -- because the venv was rebuilt with --no-deps and because the two rig failures this box has already produced "look like a result rather than a broken rig: tasks run, requests flow, numbers come out." Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…a dead arm The pre-flight did its job and the arm it was gating would have measured nothing. Arm B at 32k reported prefix_rewrite_repaid = 0 and prefix_rewrite_not_repaid = 0. Not a refusal -- the branch's headline trigger was never REACHED. The min_inventory early return sits above the econ decision, so an inventory below the floor short-circuits the trigger before it is evaluated. min_inventory 10 is unreachable on this benchmark, and that is measured rather than argued: across 710 tool outputs the median is 59 tokens, p75 is 1,187, and the shipped floor of 1000 yields 4.0 candidates per request. Dropping the floor to 100 only reaches 5.9. No floor makes 10 -- it was calibrated on production Claude Code traffic, where a tool output is large. So the floor moves to 3 in ALL THREE arms, with min_tokens left at the shipped 1000 so the candidate population is unchanged. Band moves to 64k. Trigger one cannot fire under benchmark load at all (not_in_pre_expiry_window 18/18) because eight workers never leave an idle gap for a cache entry to approach expiry. That is convenient rather than merely tolerable: with pre-expiry inert, arm B isolates the econ trigger cleanly, since econ is only consulted when pre-expiry did not fire. Stage 0's endpoints are operational and mechanism counters, none of which need reward headroom, so 64k costs nothing -- and iteration 008 already prescribed this split. Reward stays a 32k question. The econ arithmetic was then checked OFFLINE, for free, before committing money: over 33 paired requests, with W taken pessimistically as the whole transcript, S must exceed 5,400-9,100 tokens and 25,000-33,655 are available. T lands at 31-43 because per-turn growth is small against a 64k window, and T is what carries the inequality. Three to four times margin. That is feasibility, not yield. Frozen inputs recorded: cg-i022-proxy-v01, sha256 d5bb9558..., commit 143cf73. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…not a copy stage022.sh and run022.sh existed only on the eval box, which is exactly how iterations 014, 016 and 018 became mutually incomparable: the thing that decided what ran was not in the repo. stage022.sh descends from stage-sab.sh with four deliberate differences. ONE BINARY and three configs, because the code is identical across arms here and only the yaml differs, which is what makes B-A attributable. BAND is a parameter that sets the declared window AND LOCA's clearing threshold together -- a proxy told 64k while LOCA clears at 32k measures a pressure curve nothing else shares, and separate knobs invite exactly that. Paths are under ~/ because /tmp is on a 10-day cleaner that already destroyed one iteration's frozen binary. And it refuses to start if npx is missing from PATH, because nvm's shims are absent from a non-interactive ssh PATH and the filesystem MCP server then vanishes silently -- a run that looks like a result rather than a broken rig. run022.sh interleaves BY ARM across three passes of five tasks. Sequential-by-arm would put arm C two to three hours after arm A, so any drift in gateway latency or model behaviour over the night would load entirely onto whichever arm ran last. Round-robin spreads it. It also yields a complete paired comparison on five tasks after pass 1, which is the earliest point an honest direction check is possible. Both kill by PID only. A kill pattern naming the binary also matches the killing shell's own argv, which is not theoretical: it killed one of my own shells earlier today, and the shared box makes a broad pkill unacceptable regardless. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…atency claim does not 45 runs, three arms, 64k, and the first iteration in this series with a clean error column: 0 errored runs and 0 HTML 400s in every arm. That matters because errored runs score 0 in ITT and had been the dominant noise term. THE MECHANISM GATE PASSED, and that is this iteration's real product. The pre-flight had both econ counters at 0/0 -- the trigger was never reached. After amendment 1 it was evaluated 30 times and repaid 30 times, with 0 declines, 30 real adjudications, 0 unparseable replies, and an index record for every candidate. coref acted 391 times, so min_batch_frac clears at 64k. The offline feasibility estimate (3-4x margin) was not merely directionally right. THE LATENCY CLAIM DOES NOT REPLICATE, and this is the endpoint iteration 021 could not produce. Turns per run, split by outcome: arm B is 18.9 against A's 20.7 overall, but among SOLVED runs it is 20.1 -> 19.8, a 1.5% difference inside noise, while among UNSOLVED runs it is 21.1 -> 18.5. B also solved fewer tasks. So the shape of B's efficiency is giving up sooner on tasks it fails, not solving faster. Iteration 021's -28% requests was read as a latency win; its own text attributed it to "fewer runaway sessions" and it could not distinguish the two. Measured with the split it lacked, on a run with ZERO errors, the gain is on the failure path. On wall clock the direction is worse rather than absent: B adds 1,396 ms per request against A's 203 ms -- about 1.2 s on every request -- to buy 1.5% fewer turns on the runs that matter, at 10x its own model spend. Arm C went the other way: +26% turns and 62% more LOCA cost. REWARD IS NULL BOTH WAYS and the pre-registered harm gate blocks both arms: B is -2 solves (p = 0.6250, bound 48.1%), C is +1 (p = 1.0000, bound 31.9%), against a 25% blocking threshold declared before the run. The structural fact this exposes is more useful than either p-value: at n=15 with one seed the best achievable bound is 21.8%, with ZERO worsened pairs. This design can essentially never license a positive claim, which is the argument for five seeds rather than more tasks. One defect filed rather than footnoted: expand_unresolved_missing = 60 in Cp1 against 0 everywhere else -- 60 of that pass's 112 expand calls unresolvable, in the pass that was also arm C's worst. An unresolvable expand is the one failure the reversibility invariant exists to prevent, and the correlation is unexplained. Also recorded, both firsts: extract_llm_sweep as SHIPPED never fires under continuous load (its only trigger needs an idle gap, and 8 workers never leave one), so arm A's sweep is inert and B-A is "a sweep that runs" versus "a sweep that cannot"; and sweep_answered_via_tool is 0 against via_prose 30 -- the verdict tool #137 added is offered every time and never used. The index row for iterations 016-021 is still missing from experiments/README.md; that gap predates this commit and is not addressed here. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
… and why David asked how much summarization was deferred. The answer is that it was deferred completely and not by the treatment: summarize acted 0 times in 986 requests, in all three arms, every record verdict "declined", and it is in top_passthrough in all nine snapshots. So iteration 021's clearest operational result -- summarization falling from 71% to 56% of requests -- cannot be measured in this iteration at all, even though summarize was kept in the baseline specifically to make it measurable. It was not the trigger. 129 of arm A's 311 requests cleared min_request_frac 0.78 and requests reached 228k-453k tokens, so summarize was eligible and declined anyway. The likely cause is collapse, which this iteration added. It acted 195/95/212 times per arm; it is documented as the last-resort catch-all for anything still oversized, it leaves a marker, and summarize skips marked content while protecting keep_last. Iteration 021 had no collapse and summarize fired on 71% of requests. That makes two results reported in this file one intervention rather than two, and the trade is now stated as one: adding collapse bought the clean error column (0 errored runs, 0 400s, against 17 and 10) AND cost the summarize endpoint. collapse was chosen as a shared control because being identical across arms means it cannot bias B-A or C-A. That reasoning holds and is not the mistake. The mistake was not checking whether it left summarize any substrate. Recorded as a HYPOTHESIS, because the component records carry no gate field and "declined" is therefore unattributed in this run's data. Confirming it needs gate-level logging for summarize or a fourth arm identical but for collapse. The opening status line is also amended so the clean-error claim no longer stands without its cost attached. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…treatment attribution David asked whether an arm whose sweep acted 11 times can mean anything. It cannot, and the answer invalidates an attribution this file was making rather than merely qualifying it. The sweep acted 11 times: 30 econ firings, 30 asks, 132 candidates offered, 132 verdicts, 11 asks that removed anything, 187,472 tokens. Eleven removal events over 284 requests and 15 runs cannot carry a -2 solve difference or a turn profile. Meanwhile the same arm carries a 93-event swing that is NOT the treatment: extract_llm acted 108 times in B against 15 in A, 38.0% of requests against 4.8%, saving 1.19M tokens against 0.32M -- on a byte-identical config. Since extract_llm runs before collapse, it is extract_llm and not the sweep that halved collapse (3.10M -> 1.49M truncation savings). So the collapse reduction is real, and it is a deferral of the wanted kind -- blunt truncation displaced by model-guided compaction -- but it is not attributable to evidence or econ_trigger. Why extract_llm swung 7x on an identical config is unexplained. It is not the pressure trigger, which #134 forces on in both arms. So section 2's turn numbers stand as measurements and the outcome split is still the right way to read them, but the claim that B's turn profile is a property of the merged design is withdrawn. A clean B-A needs extract_llm controlled, most simply removed from all arms given #134. What IS attributable is recorded in a new section 2c. Coverage was 132 of 132 verdicts -- 100% at mean batch 4.4 -- against iteration 021's 61% at batch 12, confirming at n=132 candidates the smaller-batch lead that 021's own conclusion named on a 6-of-6 probe. Coverage is a property of the ask, so it does not depend on trajectory. The latency question is also answered: it is the sweep, at 15.2 s per ask. extract_llm_sweep.duration_ms is 454,661 ms against arm A's 35, which is 1,601 ms per request amortised over 284 and accounts for the entire cg_added_ms_avg gap of 203 -> 1,396. Nineteen of the thirty asks returned "keep everything", so ~290 s of it bought nothing. And one anomaly that deserves its own issue: 40% of every ask is paid FRESH -- mean 42,409 cache-read plus 28,764 fresh. A prefix ask appends a question to a byte-identical prefix, so fresh should be little more than the question. The likely cause is self-inflicted, each removal mutating the prefix so the next ask no longer matches the cached one, which would make the mechanism more expensive the more it succeeds. That is the design's central economic premise and it is not behaving as designed. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
… overstated claims David asked to dive into the fresh-token anomaly and fix it, and to say whether the 4.4-vs-12 batch size was configuration or the band. Both answers required correcting this branch's own record. THE BATCH SIZE IS CONFIGURATION, NOT THE BAND. The cap is maxAskItems = 12 here and was 12 in iteration 021, so the gap is candidate availability, and availability is set by the sweep's floor. Measured on arm B's own traffic -- 6,708 tool outputs, 23.6 per request -- min_tokens 1000 yields 4.5 candidates per request (matching the observed 4.4), 300 yields 8.6, and 100 yields 11.6, at the cap. So 64k does supply a batch of twelve and the 1000-token floor excluded it; iteration 021 reached twelve because its extract_llm ran with min_tokens unpinned. This partly confounds the coverage result: batch size and the rest of the configuration moved together, so the honest claim is "coverage was 100% at batch 4.4", not "a smaller batch fixed coverage". Re-measuring coverage at batch ~12 is one config line and is the most informative cheap follow-up available. THE FRESH-TOKEN CLAIM IS WITHDRAWN. An earlier version of results.md called it a defect and proposed that each removal mutates the prefix so the next ask misses cache. Investigation does not support it: prefixask.go already sends the previous turn's SENT body rather than the incoming one for exactly that reason, and sweep_no_prefix was 0 across all 30 asks, so the stash never went missing and the 1.5 MB cap was never hit. Two explanations remain and this run cannot separate them -- the benign one being that Anthropic caches only to the last cache_control breakpoint and cg.cache_boundary saw only 2, consistent with LOCA marking system blocks alone rather than the last content block as Claude Code does, in which case the uncached tail is expected. WHAT IS ACTUALLY DEFECTIVE is that the question was unanswerable at all. cg.sweep.ask is the only record carrying the ask's economics and it had no session, so it could not be joined to the request that produced it. This adds session, max_cached_idx, req_tokens and messages -- the two candidate causes are distinguished by the boundary and the request size, both already in hand at the call site. The test asserts the RENDERED line rather than reading the call site, because a field passed but dropped by the handler is the failure worth guarding. Verified to fail on all four fields when the instrumentation is reverted. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
David asked why lowering min_tokens RAISES the candidate count, expecting the opposite because twelve candidates carry more tokens than four. Both halves are right and the table now shows why they are not in tension. min_tokens is a per-output BAR, not a budget: it asks of each tool output whether that one output is big enough to name. Lowering it can only admit more, and monotonically -- the candidate set at 100 is a strict superset of the set at 1000. Arm B carried 23.6 tool outputs per request and the 1000 floor named 4.5 of them. The mass does rise, as expected, but barely: 38,077 -> 41,001 tokens per request against 4.5 -> 11.6 candidates. +155% items for +8% mass, because the newly admitted outputs are by definition the smallest -- the 2,007 additions between 100 and 1000 average 414 tokens against 8,370 for those already in. That difference in rates is what decides the purpose of a lower floor. In a prefix ask the model reads each output from cache, so naming one more candidate costs one inventory line, ~30 tokens, not the output. A lower floor therefore buys peers to compare against almost free, which is the axis the mechanism lives on -- 6% live-kept at a batch of one against 58% at ~15 -- while buying almost no removable mass, and a dropped 414-token output leaves a descriptor behind so its net recovery is smaller still. This is the same effect docs/results/min-tokens-vs-economic-gate.md records from the economic gate's side. Also corrects a figure I gave earlier: floor 300 reaches a batch of 8.6, not ~12. Matching iteration 021's twelve needs floor 100. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…_tokens doing two jobs min_tokens was deciding two unrelated things at once: which outputs are worth NAMING in the inventory, and whether a drop is worth its cost. Those want opposite settings, and the second was winning. Naming is nearly free. The model reads each output from the cached prefix, so one more candidate costs one inventory line -- on the order of thirty tokens, not the output -- and more candidates is the axis this mechanism lives on: 6% live-kept shown a single output against 58% at ~15. But a high floor was the only thing standing between the model and an expensive drop, so it had to be set for the second job. Measured on iteration 022: the shipped floor of 1000 named 4.5 of the 23.6 tool outputs a request carried, holding the batch at 4.4 against a cap of 12. A floor of 100 reaches 11.6 candidates for 8% more mass, because the outputs it admits average 414 tokens against 8,370 for those already in. SIZE WAS THE WRONG DISCRIMINATOR ANYWAY. A drop's real cost is not its marker -- tryMark already refuses, marker-inclusive, any drop whose replacement would not shrink the message. It is the cache-WRITE that mutating the prefix forces, charged over the span from the EARLIEST dropped index to the cached boundary, once per pass. So a small output dropped after something already being dropped costs its descriptor and nothing more, while the same output dropped earlier than everything else sets W for the whole batch and must repay the entire rewrite alone. Depth relative to the rest of the batch decides, and a size floor cannot express that. selectAffordableDrops walks the votes latest-first, accumulating, and picks the subset maximising S*T - 11.5*W. Maximised rather than thresholded because the objective is not monotonic in k: one early, tiny drop can extend W past what several later ones repay. prefixRewriteNet exposes the terms because choosing a subset is an optimisation, and a bool cannot be maximised. ONLY ON THE ECON PATH. The pre-expiry window fires precisely because the prefix is about to expire, which makes W nearly worthless; pruning there would forgo real savings to protect nothing. Under pre-expiry every vote is applied, as on main. VACUITY: both mutations verified. Making the selector a no-op fails the tiny-early- drop test (kept=3). Removing the !preExpiry guard fails the pre-expiry test with drop_unaffordable_pruned:11 -- but only after that test's fixture was tightened. Its first version used preExpiryCtx's default 1M window, where every drop is affordable, so removing the guard changed nothing and the test passed vacuously. At 40k the window is tight enough that anything surviving does so BECAUSE of the guard. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
Two corrections to iteration 022's arms, both derived rather than tuned, plus the runner. extract_llm IS UNPINNED. Iteration 022 inherited housellm verbatim, which pins both a per-output floor and a trigger threshold. Either sets `explicit`, and shouldFire() then returns true unconditionally, so the warm path fired on every request -- and its pinned floor sat below the 6,250-token break-even the preset's own comment derives, so it fired below cost as well. That is #120 stacked on #134, and the analysis is now posted on #120. Unpinned, the derived pressure trigger governs and pressureFloor derives the per-output floor from pressure instead of from a constant that has to be chosen per workload. It is also the change that makes arm B interpretable. extract_llm acted 15 times in iteration 022's arm A against 108 in arm B on a byte-identical config, dwarfing the sweep's 11 acts. Per session it was concentrated -- 3 of 15 sessions in A, 8 of 15 in B, two sessions holding 67 of B's 108 -- so the aggregate is a high-variance statistic at this n rather than a systematic effect. Unpinning does not remove the variance, but it removes the forcing that guaranteed the component ran every turn. THE SWEEP FLOOR DROPS TO 100. That knob was doing two jobs, and selectAffordableDrops now owns the second, pricing a drop on depth relative to the batch. So the floor is free to serve naming, and naming is what sets batch size: measured on iteration 022's own traffic a floor of 1000 named 4.5 of the 23.6 outputs a request carried, holding the batch at 4.4 against a cap of 12, while 100 reaches 11.6 for 8% more mass. That makes iteration 021's coverage result -- 61% at batch 12 against iteration 022's 100% at 4.4 -- directly testable, which it was not before, because batch size and the rest of the configuration had moved together. Starting at ONE seed to analyse before committing to five. Five is what a reward claim needs: at n=15 the Clopper-Pearson floor is 21.8% with ZERO worsened pairs, against a 25% blocking bound, so 15 pairs can essentially never license a positive result whatever the effect. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…nd hardens Coverage held at 99.8% (643 of 644 verdicts) as the batch grew from 4.4 to 6.7, on 5x the candidates and 2.3x the tokens removed, with 96 asks against 30. That is the question the iteration was launched to answer, answered in the direction the design wanted. The affordability rule added this week also fires -- 5 prunes, only in the arm where the econ trigger runs -- so depth bites on real traffic rather than merely being implemented. But arm B is still confounded and now systematically rather than noisily. extract_llm ran in all three arms on an identical unpinned config and its pressure trigger fired only in arm B: 0 acts in A, 0 in C, 239 acts and 101 extraction calls in B, across 1,135 component runs. Iteration 022 could read its 15-vs-108 swing as variance concentrated in a few long runs; a perfect 0/239/0 segregation is not variance. Unpinning removed the #134 forcing as intended and in doing so exposed a systematic interaction, whose obvious direction is wrong -- the sweep removes content, which should lower pressure and make extract_llm fire LESS. Two candidate mechanisms are recorded and neither is established. Markers plus expand does not survive arm C, which leaves markers too (68 coref acts) and shows zero. The economic gate is the likelier one: it prices candidates by position against cache state, and the sweep mutates the prefix, changing max_cached_idx on later turns. That is testable from this run alone now that cg.sweep.ask carries the boundary. Reward is null both ways again -- B +1 (p=1.0000, bound 40.5%), C 0 (p=1.0000, bound 31.9%) -- both blocked by the 25% harm gate, as iteration 022 predicted any single-seed design must be. Arm A fell from 7.00 to 5.00 because unpinning made extract_llm inert in it, so this baseline is a different configuration and no cross-iteration comparison should be drawn. Recommendation: the mechanism results justify five seeds, the arm design does not yet. Settling the 0/239/0 split costs nothing and is a prerequisite. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…he latency attribution were both wrong Section 3 said arm B was "still confounded" and charged it 5,452 ms per request. Both are withdrawn, and the trace that withdrew them is now in the file. The hypothesis is REFUTED, not merely unconfirmed. It held that the sweep mutates the prefix so content the other arms skip as cached_prefix becomes tail-eligible in B. cg.extract_llm already carries the boundary, so it was directly testable: maxCachedIdx medians are 31.0 / 34.0 / 38.5 across A / B / C, B sits between the other two, and the boundary was never unknown in any arm. If the sweep were moving it or pushing the cache cold, that is where it would show. The trace then found the counters contradicting each other. The same arm and run has components.extract_llm.acted 239 and extract.calls 101, while cg.extract_llm reports cands 0 on ALL 374 records -- 692 across both arms checked, no exceptions. cands is len(cands) after the gate loop, so 0 plausibly means nothing survived, consistent with skip_tail absorbing 7,487. A component with no surviving candidates cannot make 101 calls. The sweep made 96 asks in that arm, close to the 101, and saved_tokens 6,077,421 against unique 597,764 is a 31.58x overcount, the signature of replays counted repeatedly. So the 59s latency and the -$1.162 are probably the SWEEP's figures, which is why attributing 5,452 ms/request first to the sweep and then to extract_llm was wrong in both directions. Filed as #176 (counters disagree) and #177 (no per-call record to arbitrate them). What survives of the 0/239/0 split is only the replay explanation: 2,291 replays, 2,291 calls_avoided, 10,538 cache lookups at a 66.8% hit rate, arm B only. The sweep populates the shared extraction cache and the tail pass replays its verdicts. That is the treatment persisting, not an independent confound. Sections 1 and 2 are unaffected: coverage, batch, prunes and removed tokens all come from cg.sweep.ask, which is per-ask and self-consistent. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…d, and the -$1.162 is neither component's The earlier text guessed the extract nested map "is not scoped to one component" and concluded the 59s latency and the -$1.162 were probably the SWEEP's figures. #178 root-caused it and the guess was too kind. The map was process-GLOBAL: one set of package-level atomics that both extract_llm and extract_sweep wrote every field of, so calls, avg_latency_ms, calls_avoided and the whole net-value block were two components with opposite economics summed under a name that reads as one of them. extraction_cost_usd was wider still, derived from cheapmodel.Usage(), which catches every cheap-model call in the process including summarize and agentdiet, priced through the haiku card even though the sweep asks the request's own frontier model. So the -$1.162 is two components' savings against three components' spend at one component's rates -- neither the sweep's figure nor the tail pass's. Pooling happened at write time, so no re-reading of these logs can repair it. Two corrections to the analysis this file carried. cands: 0 was NOT the wrong side -- a frozen replay continues before its candidate is appended, so cands: 0 on all 692 records and acted: 239 are consistent and both true, and extract_llm acted 239 times without one fresh call. And the 31.58x overcount is not a defect at all; it is the designed cumulative-versus-unique contrast, correctly left alone. One side finding recorded because it has behavioural teeth rather than merely display: tooSlowToExplore read the GLOBAL p50, so the sweep's ~59s asks were braking extract_llm's cheap-model exploration. The tail pass's firing behaviour in this iteration was influenced by the sweep's latency through a channel nobody was watching. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…ttribution The branch needed #178 before iteration 024 could run: a binary built from this branch carried the OLD pooled extraction counters, so the run would have reproduced exactly the unattributable cost and latency numbers iteration 023 spent a day root-causing. components/offload/extract_sweep.go changed on both sides and git auto-merged it. The resolution was verified rather than trusted, because a clean auto-merge of one function can compile and still be wrong. Both sides' work is present and reachable: from main defer foldFallback() with the NAMED result (_ []int, r sweepResult) -- the named result is load-bearing, since a deferred mutation of a local is discarded after the return value is copied, which is what made the obvious version of that fix inert; recordLeg / runFallback booking each fallback leg. from here selectAffordableDrops still called on the econ path only, at the same point between adjudicate() and the freeze/splice loop; evidence and econ_trigger still read; coref.Index still filling the evidence seam. gofmt clean, CGO_ENABLED=1 go vet clean, go test ./... green across 28 packages on the eval box against the merged tree. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…sured for the first time Written after launch but before any number is read, and committed with the frozen inputs and the reading table before the first arm-seed completes. Anything read earlier than this commit is inadmissible. Iterations 022 and 023 established that the mechanism works -- coverage 99.8% at batch 6.7, 96 asks, 426,638 tokens removed, the affordability rule firing. Cost and latency were never measured, because two defects made them unattributable. The extract counters were process-global and both components wrote every field, so latency and net value were two components with opposite economics summed under one name (#178, merged here as be79553). And cheap_model_price_unconfigured fired ONCE PER REQUEST in every arm of 023 -- 333/389/413 -- because the gate reads CHEAP_MODEL_PRICE_*, which the rig never set, so every allow/suppress decision in both iterations was taken against list rates rather than the operator's card. Now exported; a probe confirms the gate at 0 over 6 requests. Two arms, five seeds, 150 runs. The coref cutter arm is dropped -- 5.00/15 twice and no cost story pending makes it the least informative dollar in a run about cost. extract_llm stays IN, reversing 023's plan to remove it. Its 0/239/0 split is not a confound: it runs through the shared extraction result cache, where the sweep's putResult populates entries the tail pass then replays (2,291 replays, 2,291 calls_avoided, 66.8% hit rate, arm B only). That is the treatment's own decisions persisting. With the accounting fixed, acted_fresh vs acted_replay can measure that interaction instead of designing around it. Five seeds is the point rather than a luxury: at n=15 the Clopper-Pearson floor is 21.8% with ZERO worsened pairs, so no single-seed design can clear the 25% harm gate -- both prior iterations were blocked by arithmetic, not evidence. The reading table includes a failure row the earlier iterations lacked: if cost_source reads anything but `component`, the primary endpoint has FAILED and no cost conclusion may be drawn. That is to be checked on the first completed arm-seed, not at the end. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…t, and cost measured honestly 150 runs, 2 arms x 15 tasks x 5 seeds at 64k. REWARD: significant on the governing test with zero tasks regressed. Clustered over 15 tasks, net +2.00, 8 better and 0 worse, p = 0.0078. Per-pair over 75 pairs, net +10, 13 improved against 3 worsened, p = 0.0213. Arm A 36.00/75, arm B 46.00/75. Iteration 021's equivalent was p = 1.0000. The harm gate cleared for a structural reason rather than a lucky one: at n=15 the Clopper-Pearson floor is 21.8% WITH ZERO worsened pairs, so 022 and 023 were blocked by arithmetic, not evidence. Five seeds put n at 75 and the bound at 11.2%. That is what the seeds bought. COST: cost_source reads `component` on every arm-seed, so the pre-registered primary endpoint passed and these are the first attributable figures. The sweep spends $20.26 and books $0.72 of gross value -- 2.4M removed tokens bank at cache-read rates -- for a net of -$19.53. extract_llm earns $11.58 at zero cost. Combined -$7.95, about -$0.11/run or 9% on LOCA's $1.13/run. The mechanism's return does not appear in its own token ledger; it appears in reward. -$19.53 of measured loss bought +10 solves. And two caches turn out to be one: the sweep's putResult writes into the extraction result cache keyed by content id, extract_llm reads that same cache for its own reason, so the sweep's paid decisions become the tail pass's free hits -- 4,749 calls avoided at a 99.2% hit rate, 364 acts, zero fresh calls. That recovers 57% of the spend. Stated carefully: the sharing is measurable, nothing shows the economic consequence was intended, and it was invisible before #178 pooled counters were split. LATENCY: the hypothesis is REFUTED, not merely unsupported. Paired turns are 24.1 vs 29.4, and on the 33 pairs where BOTH arms solved, still 29.3 vs 33.2 -- 13% more turns on comparable work. Composition explains about half the gap and not the rest. The mechanism buys accuracy by letting the agent work longer on a managed context. The pre-registered reading table's row for this outcome is half right and is amended in the file: it treated reward as a harm gate rather than a possible finding, and the honest statement is that the mechanism does not pay in tokens and does pay in reward. Not a claim about the shipped configuration: the arms run min_inventory 3 and sweep min_tokens 100 against shipped 10 and 1000. Not a claim about coref-the-component, which was in neither arm. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…anism, and promote the unresolved expands David asked how the sweep's decisions can feed extract_llm when the tail pass runs BEFORE the sweep within a request. It cannot, and the earlier text asserted a mechanism it had not verified. What stands is the accounting: extract_llm books $11.58 at $0.00 from 0 fresh calls and 364 replays, arm B only, 4,749 calls avoided at a 99.2% hit rate, and the sweep is the only writer to that result cache. The entries come from the sweep across turns. What is withdrawn is the path. Two candidate cross-turn paths were checked and both are closed by the counters. Expand-restore fails because expansion IS far higher in arm B (kept_verbatim_after_expand 9,478 against 3,522) but that gate makes extract_llm SKIP expanded content rather than compact it. The tail/prefix boundary moving fails because cached_prefix dominates at 29,302, so candidates are overwhelmingly skipped as not-tail. So the recovery is real in accounting and unexplained in mechanism -- which matters commercially, since 57% of this configuration's cost recovery rests on a behaviour nobody has identified in a cache neither component documents sharing. And the check surfaced something weightier, now a numbered section rather than a footnote: expand_unresolved_missing is 175 in arm B and 0 in arm A. The agent asked for removed content back 175 times and did not get it -- the one failure the reversibility invariant exists to prevent. Not new either: iteration 023 recorded 60 in its Cp1 pass, against 0 elsewhere, in that iteration's worst-scoring pass. It does not invalidate the reward result -- 8 tasks better, 0 worse, harm bound 11.2% -- but it raises a question that must be answered before any production claim: whether the mechanism wins partly by removing content irrecoverably. A drop the agent can undo and one it cannot are different products, and the reward number cannot tell them apart. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
#187, not refuted An earlier draft called the latency hypothesis "refuted" on the strength of arm B taking 13% more turns on the 33 pairs where both arms solved. That was too strong. Section 2b records 209 unresolved expands in arm B against 0 in arm A -- about 2.8 per run against roughly 33 turns per run. Every one is a turn the agent spent asking for removed content and receiving a placeholder, possibly asking again. That can account for a meaningful share of the 13%, so the turn penalty cannot yet be attributed to the mechanism rather than to the defect now filed as #187. Corrected to "not supported, and partly attributable to a reversibility defect", with the explicit note that no latency claim should be made in either direction until a re-run after the fix separates them. The reward result is unaffected and remains conservative: arm B won 8 tasks to 0 at clustered p = 0.0078 while carrying the same 209 failed expands as a handicap. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
… and it is a shared result cache David asked what happened to the unexplained recovery channel. It is explained, and the answer was in a comment: store.ResultPrefix is documented as "extract_llm's replayed result", and the sweep's drop path writes into it through putResult -- putResult and getResult being shared helpers in components/offload/state.go keyed cg:res:<session>:<content-id>. One cache, two writers, by construction rather than by collision. That resolves the ordering objection an earlier draft foundered on. extract_llm runs BEFORE the sweep within a request, so the sweep cannot feed it in the same turn; the cache is content-addressed rather than positional and persists across turns, so extract_llm need not have seen the content first -- only encounter a content id on a later turn that the sweep already ruled on. And the control flow narrows which ids those are. The marker skip is at extract_llm.go:848 and the cache lookup at :868, so a marker-bearing message never reaches getResult. All 364 replays were on unmarked content, which leaves the same content recurring at a fresh position -- routine agent behaviour, and consistent with a 99.2% hit rate. The two previously-checked candidate paths are recorded so they are not re-proposed: expand-restore dies on kept_verbatim_after_expand (9,478 against 3,522, and that gate SKIPS rather than compacts), and the boundary-moving theory dies on cached_prefix dominating at 29,302. The consequence for cost is not reassuring and is stated: the cg:res: entries are pinned so #187 never touched them, but #188 REFUSES removals when the payload reserve is full, and no removal means no putResult and no replay. The recovery therefore scales with how often the sweep acts and moves in both directions under that fix -- a 5,000-entry store admits more, refusals suppress more. The -$7.95 should not be expected to reproduce. Signed-off-by: David Amid <david.amid@il.ibm.com> Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-cut of the co-reference work onto
main, with the first significant positive reward result this line of work has produced.Supersedes #80, which tracks
feat/coref-compaction— a branch whose merge basemainhas since moved 295 files past. That history is preserved on origin atarchive/coref-compaction-pre-recut(f31b451); nothing is lost, and the experiment record it carried is ported here.What this adds
coref— the co-reference index and component.internal/coref(the index) plus thecorefcomponent: a zero-LLM deterministic cutter that removes tool outputs no later turn literally reuses anything from. Offline, scored against a held-out future,unreferencedalone removed 11.8% at 95% live-kept and was the best discriminator of ten arms, model arms included.The
Evidenceseam, filled.mainshippedextract.AdjudicationItem.Evidenceas an explicitly empty seam, with a note at the candidate append site saying that when this work arrived, "the index's verdict belongs in the prompt as EVIDENCE for the model to weigh… never as a gate that pre-decides the answer." That is exactly how it is filled: the index's record joins each inventory line, the candidate set is untouched, and the contract gains a paragraph — conditional on an item actually carrying evidence — teaching the model to read the counters and to treat the index as a fallible witness rather than a judge.Not filling it as a filter is the whole point. A co-reference pre-filter removed 149,681 candidates and left about one per request, silently turning a bulk arm into the per-output shape refuted at 6% live-kept — while reporting itself as bulk throughout.
A second trigger —
econ_trigger.mainsweeps only inside the pre-expiry window. Neither trigger contains the other: pre-expiry fires on the clock and cannot fire at all on a session whose cache keeps being refreshed — the long run with the most to save — while econ fires on mass and must clearS·T > 11.5·Wfirst. Measured: under benchmark loadnot_in_pre_expiry_windowfires on every request, because eight workers never leave an idle gap, so econ is the only trigger that can fire at all there.selectAffordableDrops— a drop priced on depth, not size.min_tokenswas doing two unrelated jobs: choosing which outputs to name, and standing in for whether a drop is worth its cost. A drop's real cost is the cache-write over the span from the earliest dropped index to the cached boundary, charged once per pass — so a small output dropped after something already being dropped costs its descriptor and nothing more, while the same output dropped earlier setsWfor the whole batch and must repay the entire rewrite alone. Depth relative to the batch decides; a size floor cannot express that. The selector walks votes latest-first and maximisesS·T − 11.5·W(maximised, not thresholded: the objective is not monotonic).All knobs default off. A test asserts a disabled trigger reports no decision at all — a gate that fires when the feature is off reads, on a dashboard, as a feature that is on and failing.
The result — iteration 024, 150 runs
+evidence, +econ_trigger)Iteration 021's equivalent was p = 1.0000. The harm gate cleared for a structural reason rather than a lucky one: at n=15 the Clopper-Pearson floor is 21.8% with zero worsened pairs, so iterations 022 and 023 were blocked by arithmetic, not evidence. Five seeds put n at 75 and the bound at 11.2%.
The mechanism does not pay in tokens and does pay in reward. 2,407,680 removed tokens bank at $0.72, because removal prices at cache-read rates — so −$19.53 of measured "loss" bought +10 solves. Those are different axes, and iteration 024 is the first run able to see both, because per-component attribution only landed with #178.
The latency hypothesis is unsupported — and confounded, so not refuted either. The standing claim from iteration 021's −28% requests was that the mechanism reaches the same reward in fewer turns. On the 33 pairs where both arms solved, arm B takes 13% more turns. But the 209 unresolved expands below are ~2.8 per run against ~33 turns/run, and each is a turn spent asking for content and getting a placeholder — so a meaningful share of that 13% may belong to #187 rather than to the mechanism. No latency claim should be made in either direction until a re-run after that fix.
What this does NOT claim
min_inventory: 3and sweepmin_tokens: 100against shipped 10 and 1000 — iteration 022 measured those as unreachable and batch-starving respectively on this workload. Identical across arms, so neither can bias B−A, but B is nothousellm.coref-the-component. It was in neither arm of iteration 024. Its own question is open.Two open problems, both stated in the write-up
extract_llmbooks $11.58 at $0.00 from 0 fresh calls and 364 replays, recovering 57% of the sweep's spend. The mechanism is now established:store.ResultPrefixis commented as "extract_llm's replayed result", and the sweep writes into it viaputResult— a helper both components share, keyed by content id. The cache is content-addressed and cross-turn, which is why the tail pass running before the sweep is no obstacle; and since the marker skip (extract_llm.go:848) precedes the lookup (:868), all 364 replays were on unmarked content, i.e. the same content recurring at a fresh position. No test pins any of this, and fix(store): give rewind payloads their own reserve, and refuse a removal whose original cannot be stored #188's refusal path can suppress it — so the −$7.95 should not be expected to reproduce.Provenance
Three general fixes were split out of this work and shipped independently rather than riding the branch: #136 (shape validation), #137 (the adjudicate verdict tool), #138 (collapse's byte gate).
mainthen absorbed the prefix-ask infrastructure on its own asextract_llm_sweep(#118). #178 (per-component extraction attribution) was found by this work and shipped separately for the same reason, and is merged in here asbe79553.Also filed from this line of work and deliberately left out of it: #175 (kept-verbatim marks are global), #179–#182.
Full record under
docs/experiments/loca/iter02{2,3,4}/, including the preregistrations and the amendments made after seeing pre-flight data.