Evaluate memory handoffs across coding agents, rules, skills, and prime - #5
Evaluate memory handoffs across coding agents, rules, skills, and prime#5csells wants to merge 7 commits into
Conversation
Add matched ordinary-task experiments across installed coding agents and model strengths. Preserve frozen conditions, actual retained memories, transcripts, independent grading, and provider interruptions. Report capture, retrieval, application, mutation, and unnecessary work separately, with denominators, costs, and practical reliability limits. Keep runtime caches and sensitive process inventory local.
|
Read both result documents end to end. The capture bottleneck replicates here, from a different setup. Our instrument is much smaller: 8-session batches against one frozen endpoint, a seeded store, and an ordinary coding task whose correct solution depends on a fact that only the seeded memory carries. Different corpus, different grader, one model. It agrees with your numbers on the part that matters most for the design. The memory was delivered and read every time, and that bought less than you would expect. Our last batch ( So those three misses were not retrieval failures. The memory arrived, was read, and was not acted on. Your prime study reaches the same place from the opposite direction, by spending 216 sessions making the same instructions more prominent and getting 3 complete initial captures per arm regardless of arm. Both setups point at application rather than delivery. What separated our 1-in-4 arm from the 4-in-4 arm was one sentence naming the destination in the vocabulary of the current task. That difference is in the record's own text. Delivery was identical across both arms, verified per row. The question both documents set aside is the one we have in flight. You write that the runs do not validate the proposed Memory bead type, and that a source-linked memory design deserves its own evaluation. Your ranked proposals then recommend a record that separates approved policy from inference from validation evidence and carries an explicit source, scope and version. That recommendation currently rests on judgment with no measurement under it. Eight paid sessions are running on exactly that question as I write this. Both arms carry identical sentences in identical order, with an identical task, prompt and store. One arm is a single paragraph. The other splits the same words across labelled fields: observation, evidence, practice, evidence_scope, mechanism_class, transfer_claim, source. Only the labels and the line breaks differ. The residual wording delta is three tokens, derived mechanically from the two frozen source texts and recorded before the first call. The reading rule was frozen before any spending. Four replicates per arm cannot separate a true 25 percent difference from noise, so a one-row gap will be reported as not separable rather than as a direction. The result posts either way. If the design has a hole in it, now is the useful time to say so. Two mechanical findings in your traces look worth acting on regardless. 66 rejected Memory search is literal substring matching, and you recorded agents searching a plausible multi-word phrase, getting nothing, and concluding the record did not exist while it sat in the store. An empty result that states the matching rule and offers a shorter query turns a silent miss into a recoverable one. That miss rate is measurable offline against our corpus with no paid sessions, and I am running that measurement now. |
2 participants
Coding agents can finish an implementation while failing to preserve or correctly reuse the agreement behind it. This PR extends the adoption harness to test complete memory handoffs during ordinary issue work across Claude Code, Codex, OpenCode, and zcode, and records where those handoffs still fail.
The seven commits build directly on Stephanie’s
adoption-harness-share, now at the same commit asmain.What is included
Work on <issue ID>and experiment-authored issues contain product requirements and approvals. Memory guidance lives in standing project rules, host instruction shims, and skills. The prime follow-up varies delivery through thin prime, rich prime, or a generic startup briefing.Main findings
These are small synthetic studies with legacy
bdmemory commands. They do not establish production reliability or validate the proposed new Memory bead type. Relevant-memory header selection by prime remains untested. The reports rank the smallest next improvements and separate measured findings from practitioner observations and proposals.Review starting points
Most changed files are retained experimental evidence. Start with:
Validation
Recorded verification includes 617 targeted Python tests for the policy work, 908 TypeScript tests in an isolated Git checkout, and 76 targeted/shared prime checks with Ruff, Black, and strict mypy. All twelve real prime-delivery qualifications passed before scoring. Final audits verify 450 frozen source files and the retained results; completed model trials were not rerun or repaired.
The full Python suite remains non-green on this Mac: 4,814 passed, 44 skipped, 27 failed, and 12 setup errors. Those baseline failures and the exact scope of passing checks are retained in the policy verification and prime verification.
Related proposal: gastownhall/beads#5877
cc @donnabox @quad341 @sjarmak