Conversation
The NLI signal finds contradiction in logical structure: negation, antonyms, numeric conflict, named-entity swaps. A same-slot swap whose only change is an opaque code token has none of that, so NLI reads it as two compatible statements and the stale record survives. Measured on 16 coding-guidance pairs at the shipped 0.62/0.50 thresholds: 14/16 with 0/8 false positives. Both misses were pure code-token swaps. cos 0.945, contra 0.013 `cargo test --workspace` -> `cargo nextest run --workspace` cos 0.839, contra 0.011 "the planning folder" -> "docs/planning" `use_literal_gate=True` extracts code literals (backtick spans, paths, flags) and treats a same-slot pair whose literal sets differ as a supersession. 15/16, still 0/8 false positives. Both-sidedness is load-bearing: a one-sided rule fires on elaborations that merely mention a path. The discriminating negative is a restatement at cos 0.963 whose literals are identical, which correctly stays quiet - so this is not "high cosine means supersede". The second miss stays missed, because one side carries no literal at all and guessing there would cost the zero-false-positive property. Off by default. It was validated only on coding guidance, and the drift and LongMemEval numbers are prose, where the gate is inert because neither side carries a literal. Existing behaviour is unchanged with the flag off. Four tests added, covering the recovered case and the three negatives that keep the gate honest. Full suite green.
The gate in the previous commit was designed on a 16-pair probe where two
command swaps scored NLI contradiction 0.013, and it recovered one of them.
This commit adds benchmarks/technical_memory.py: 36 fresh pairs of coding
guidance, written in one pass before any policy was run against them, sharing
no pair with that probe. For the gate it is a genuine held-out set.
policy recall precision FP FN
cosine-only 0.94 0.77 5 1
nli 0.94 1.00 0 1
nli+literal 0.94 1.00 0 1
The gate is exactly inert. On this set the NLI model fires correctly on every
command swap, contradiction 0.841 to 0.990, so the failure the gate targets
does not generalise. It was a property of two specific sentence pairs, not of
code-token swaps as a class. So memory/sidecar.py, its tests, and the README
go back to what they were; the policy survives only inside the benchmark, as a
documented negative result nobody has to re-derive.
What the benchmark does establish is worth more than the gate was. The shipped
NLI path transfers to technical memory at PERFECT PRECISION: it never once
forgot approved guidance. All five cosine-only false positives are restatements
("Use `pytest -q` to run the unit tests" against "The unit tests run with
`pytest -q`", cosine 0.974), and NLI scores every one at contradiction <= 0.002.
That is a clean domain-transferred demonstration of what the contradiction
signal buys over a density gate.
The one shared false negative is an antonym pair with no code literal at all,
cosine 0.846 and contradiction 0.021. Cosine-only catches it and both NLI
policies miss it. That is the real remaining gap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
benchmarks/technical_memory.py. Net diff against main is that one file: 276 lines, no change tomemory/, no change to the existing tests, no change to the README.Why a separate benchmark
The drift suite measures personal-fact slot updates. Agent memory is increasingly used for standing engineering guidance instead, where the facts are build commands, paths, flags, versions and conventions. That content stresses the write gate differently. "The tests run with
pytest -q" and "The tests run withpytest -n auto" are not logically contradictory sentences. They are two compatible-looking statements that disagree about an opaque token.36 pairs, 18 positive and 18 negative, written in one pass before any policy was run against them. Three policies over one embedder and one fixture, so any gap is the policy's doing.
Result
The shipped NLI path transfers to technical memory at perfect precision. It never once forgot approved guidance. All five cosine-only false positives are restatements, for example "Use
pytest -qto run the unit tests" against "The unit tests run withpytest -q" at cosine 0.974, and NLI scores every one of them at contradiction 0.002 or below. That is a clean domain-transferred demonstration of what the contradiction signal buys over a density gate, on content the drift suite does not cover.The one shared false negative is an antonym pair carrying no code literal at all: "Test fixtures are generated at build time" against "Test fixtures are checked in and never generated", cosine 0.846, contradiction 0.021. Cosine-only catches it and both NLI policies miss it. That is the real remaining gap on this domain.
About the history on this branch
The first commit added an opt-in literal-diff gate to
memory/sidecar.py. I had measured it on a 16-pair probe where two command swaps scored contradiction 0.013, and it recovered one of them.The second commit removes it. On this held-out fixture the NLI model fires correctly on every command swap, contradiction 0.841 to 0.990, so the failure the gate targets does not generalise. It was a property of two specific sentence pairs, not of code-token swaps as a class. The gate is exactly inert here: same recall, same precision, same false negative as plain NLI.
I left both commits rather than force-pushing, since the correction is part of what the benchmark is for. The policy survives only inside the benchmark file, as a documented negative result so nobody re-derives it.
Honest limits
The fixture is self-authored by one person, so it measures transfer to one author's idea of technical memory rather than to a sampled population of real stores. 36 pairs is small. The three policies are evaluated as pure write-gate decisions over sentence pairs, not end to end through
write(), so surprise gating and retention decay are out of scope here.Reproduce with
python -m benchmarks.technical_memory. Existing suite unchanged at 14 passed.