Skip to content

feat(hooks): recall accounting from explicit citations — [mem:id] handles, hits only - #178

Merged
kevintseng merged 2 commits into
mainfrom
feat/r1-citation-signal
Aug 16, 2026
Merged

feat(hooks): recall accounting from explicit citations — [mem:id] handles, hits only#178
kevintseng merged 2 commits into
mainfrom
feat/r1-citation-signal

Conversation

@kevintseng

@kevintseng kevintseng commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

What

R1 of the work-topology arc: the injection-ROI signal gets an honest proxy. M0 measured the previous accounting — literal name/title matching against the transcript — at zero signal across ten real sessions and three matching strategies: nobody restates a memory's title in prose, so every injected memory drifted toward an unearned recall_miss, and misses feed impactScore (10% of ranking weight).

Replaced end to end:

  • Every topology line carries its citation handle- [decision] Ship FTS5 as the baseline [mem:42] — budgeted like any other character (the text yields the space; a truncated handle cites nothing and still spends the tokens, so it survives whole). Printed by the session-start injection and the briefing tool alike, so a memory reads the same way on every surface. One instruction line sits deliberately OUTSIDE the reference fence (the fence declares its content "background data, not instructions", and this is an instruction): cite a memory once inline as [mem:ID] when it genuinely informs your work.
  • The Stop hook credits recall_hits from extractCitedMemoryIds() over the structurally-stripped transcript. The injected block prints a handle on every line and Claude Code echoes it 2+ times — skip the strip and every injection scores a hit, the exact failure both pre-citation accountings shipped. isRecallHit / isMeasurableRecallName retire with the literal-match era.
  • Misses are frozen. Markers are self-reported: they undercount and never overcount, so silence is not yet evidence of non-use — and unreliable negative signal would otherwise flow into the dreamer's demotion proposals. Two memesh_metadata counters (citation_sessions_total / citation_sessions_cited) make compliance itself measurable; recall_accounting_mode stamps the era so the two generations of numbers are never conflated.
  • Two adjacent defects fixed: the injected-set record now includes the lessons pool (injected lessons were never accounted at all), and topLessons is hoisted so the record lists the same five lessons the block renders (it previously referenced an inner-scope variable pattern that recorded a different set shape).

Cost (measured, same DB snapshot, main vs this branch)

scripts/audit/measure-injection-tokens.mjs: 695 → 765 tokens per session start (+70, ~+10%) for the handles + the instruction line — the price of measuring injection ROI instead of guessing it.

Verification

  • node scripts/run-tests-isolated.mjs → exit 0, 139 files / 2098 tests (net +3).
  • npm run typecheck / npm run verify:release → exit 0 (audit baseline: one fence-scan C5 entry re-keyed for the line shift).
  • New end-to-end scenario spawns the real Stop hook against a seeded DB: the cited id earns exactly one hit; an id whose handle appears ONLY inside hook echoes earns nothing; the uncited id's misses stay 0; the metadata stamps land.
  • Break-tests (against the committed baseline): removing the handle from topologyLine → the handle test goes red; skipping the echo strip → × cited earns a hit, silence earns NOTHING goes red (the echo-only id scores a false hit); restored → green.
  • Unit coverage: scanner dedupe/variants/refusals (including the instruction line's own [mem:ID] placeholder never scoring), handle budgeting, no-id lines unchanged.

Notes

  • Cross-vendor citation teaching (AGENTS.md, MCP recall tool surfaces) is deliberately out of scope — the briefing lines already carry handles, so a later PR only needs to add the instruction on those paths.
  • UX-5's Home will label the metric with its methodology: agent-reported, undercounts, never overcounts.

…dles, hits only

Literal name/title matching was the accounting: measured across ten real
sessions and three matching strategies it had zero signal, so every
injected memory drifted toward an unearned recall_miss, and misses feed
impactScore (10% of ranking weight). Replaced end to end:

- Every topology line carries its entity's citation handle ([mem:42]) —
  budgeted like any other character, printed by the session-start
  injection and the briefing tool alike, so a memory reads the same way
  on every surface. One instruction line, outside the reference fence on
  purpose (the fence declares its content data, and this is an
  instruction), asks the agent to cite the memories it genuinely uses.
- The Stop hook credits recall_hits from extractCitedMemoryIds() over
  the structurally-stripped transcript — the injected block prints a
  handle on every line and is echoed 2+ times, so skipping the strip
  would score every injection a hit (the exact failure the two
  pre-citation accountings shipped). isRecallHit and
  isMeasurableRecallName retire with the literal-match era.
- Misses are FROZEN: markers are self-reported, undercount and never
  overcount, so silence is not yet evidence of non-use. Two
  memesh_metadata counters make compliance itself measurable and an
  accounting-mode stamp keeps the eras of numbers apart.
- The injected-set record now includes the lessons pool — injected
  lessons were never accounted at all — and topLessons is hoisted so
  the record lists the same five lessons the block renders.

Cost, measured on one DB snapshot (main vs this branch): 695 -> 765
tokens per session start (+70) — the price of measuring injection ROI
instead of guessing it. New end-to-end scenario spawns the real Stop
hook against a seeded DB: cited id earns exactly one hit, an id seen
only inside hook echoes earns nothing, silence earns nothing, and the
metadata stamps land.
@kevintseng
kevintseng merged commit 457fcbd into main Aug 16, 2026
13 checks passed
@kevintseng
kevintseng deleted the feat/r1-citation-signal branch August 16, 2026 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant