Conversation
The export->import roundtrip was broken: /v1/export payloads could not be imported back (raw 500), so a namespace could not move between instances. - Native 'engram' import format (auto-sniffed by engram_export_version): Memory.import_export() restores facts with their original ids, bi-temporal stamps, supersession chains, and provenance; episodes arrive consolidated so System-2 never re-extracts duplicates; idempotent by id; the target re-embeds locally, which is also the embedder-migration path. - /v1/import returns 400 with the parser's reason on malformed payloads. - MCP streamable-HTTP transport gains a Bearer gate (--http-token / ENGRAM_MCP_HTTP_TOKEN); non-loopback binds without a token fail closed. - ENGRAM_STORAGE env selects the vector backend (memory|lancedb), rejecting unknown values at startup so /health reports the real backend. - Import CLI local mode writes through MemoryService: same digest-backed namespace dirs and locks as HTTP/MCP (a/b no longer lands in a stray dir). - /v1/stats filters by canonical linked identity like every other read path. Docs: API.md migration recipe, agent-adapters --http-token, architecture map + full report entries, and the cross-account memory roadmap.
Several capabilities were written, tested, and then never merged; the memory ledger records them as shipped while main does not have them. Triage each one against today's main by capability rather than by filename, since main evolved 100+ commits independently and often implements the same thing under a different design. Two conclusions worth keeping: main's JSONL persistence layer is stronger than the branch's SQLite snapshot (so that one is dropped, not merged), and five worktrees hold 2000+ lines that were never committed to any branch at all — deleting those worktrees would destroy the work.
…ss that measures it The read path scored every live fact on every query. eval/scaling.py measures what that costs: ms per 1k facts holds constant at ~17-20 from 100 to 10,000 facts -- the signature of O(n) -- and a single query already takes 177ms at 10k facts, past the <100ms read-path target, at a store size that is small next to the 10M-token goal. Bounded retrieval scores a candidate pool instead: a lexical inverted index, the vector store's own search, and the facts hanging off the query's anchor entities, unioned, then completed so every candidate's conflict slot arrives whole (otherwise a superseded fact whose slot head fell outside the pool would survive -- a correctness bug, not a ranking one). Corpus statistics stay tenant-scoped so BM25 scores a subset exactly as a full scan would, and candidates are resolved in store order because rank-based fusion breaks ties by position, so an unordered set would make the same query answer differently run to run. The measured result is negative where it matters, and that is the useful finding: with the semantic channel on, bounding the rest buys nothing (177.03ms vs 177.88ms at 10k), because neither backend has a real ANN index -- the in-memory store brute-forces cosine and sorts, and LanceDB materialises the whole table whenever a Python predicate is supplied, which multi-tenant retrieval always supplies. With that channel off the same code is 14.5x faster at 10k and genuinely sub-linear (ms/1k falls 6.17 -> 1.23). So the candidate-pool design is sound and the blocker is the missing filtered ANN, which is now the P0 on the optimization map. Both flags default off: the published numbers came from the full scan, and dropping the semantic channel would lose the recall the hybrid thesis depends on. A pool at least as large as the live fact count returns bit-identical results to the full scan, which is the test that keeps the two paths honest. A decorator maintains the index so none of the eight existing upsert/delete call sites had to change.
…ng for it Multi-tenant retrieval filters by user on every single query, and the only way to express that was a Python predicate. A backend cannot see inside one, so it had to hand every row to Python before ranking: LanceDB called to_arrow().to_pylist() on the whole table. The consequence is that configuring the scale backend never bought a single sub-linear read -- there was a vector file, not a vector index. Lift user_id out of the opaque JSON payload into a real column and give VectorStore.search() a declarative user_id= alongside the general predicate, so LanceDB can prefilter inside its own index. Measured on a skewed two-tenant table: 4.04ms -> 1.13ms at 500 rows, 75.38ms -> 1.35ms at 10k, and 302.36ms -> 1.83ms at 40k. The pushed-down path barely moves as the table grows 80x while the predicate path is strictly linear -- 165x at the top end. The correctness risk with any filtered ANN is that the filter runs after the search, which silently returns too few rows or none. test_prefilter_finds_hits_beyond_the_unfiltered_neighbourhood is built to fail in exactly that case: the majority tenant fills the query's whole neighbourhood and the tenant being asked for sits far away, so a post-filter finds nothing. Tables written by earlier releases have no such column. _has_tenant_column() probes the real schema rather than assuming, so those keep reading and writing -- falling back to a scan -- instead of failing on a schema mismatch. eval/scaling.py now reports this alongside the retriever benchmark, skipping it when lancedb is absent.
…sh key lookups into LanceDB Every retrieval runs query_entity_ids(), which ends by calling graph_excluded_entity_ids(), which ran two regexes against every entity name in the store looking for "not Lisbon" style constraints. Almost no query contains a negation, so almost every retrieval paid for a full entity scan that found nothing. At 5,000 entities a plain query spent 509ms there; it now spends 0.0009ms. The matcher is anchored to the end of the text preceding an entity mention, so it cannot be run against the whole query directly. What can is a necessary condition: if no cue word appears anywhere, no slice can contain one. Both regexes are now built from a single cue list so they cannot drift. Writing that cue test the obvious way was wrong, and the test caught it. A non-ASCII entity name has no word-boundary guard in _entity_name_mentions, so the preceding slice can end mid-word: in "not上海" the slice "not" ends on a boundary the full string does not have, and \bnot\b misses it -- silently dropping an exclusion. The cue test therefore drops its trailing \b and is deliberately weaker than the real matcher. Over-matching costs a scan that finds nothing; under-matching loses a constraint the user stated. test_exclusion_shortcut checks the implication over every prefix of its samples, so adding a cue to one regex and not the other fails loudly. Separately, LanceDBVectorStore.get() materialised the table and scanned it in Python for one key, which makes any id-at-a-time access quadratic. It now issues a filter-only query so the predicate runs inside LanceDB. (table.query() does not exist in 0.33 and pylance is not installed; search().where() is what works, verified rather than assumed.) Still linear, and now written down: the exclusion scan itself when a query does contain a cue (its matching is substring-based, which Chinese names require, so a token index cannot replace it), and the name/alias matching in query_entity_ids, which is tokenised and could be indexed.
Five worktrees held work that existed in no branch at all, so deleting them would have destroyed it. It is now committed as-is on each worktree's own branch, with the triage recording what each one is worth against today's main. Preserved is not merged: all of it predates 100+ commits of main and needs re-checking before any of it is ported. Two items are not superseded and are worth acting on. segment.py describes a defect that is still live -- the cross-encoder reranks whole ~2000-token sessions at max_length=512, scoring only the first quarter of each one. And the read context is still assembled as a single flat string, so nothing in it can be reused by provider prompt-caching across a multi-turn session. The privacy hold is cleared: the demo corpus uses a pseudonym per the repo owner, and a scan for other identifying data found none.
…arter A cross-encoder reads a bounded window -- 512 tokens for the BGE rerankers -- and a LongMemEval session is around 2000. Handed a whole session it does not fail; it scores the first quarter and discards the rest, so a session whose answer sits late ranks as if it were irrelevant. The project already knew this. lean_context reranks only the fact pool and its comment says reranking whole sessions "truncates to 512 and mis-ranks -- a known _S regression". But retrieve_episodes() still passed whole ep.content to the reranker, and no test covered that path, so the documented defect stayed live in the one place it applied. Sessions are now split on paragraph and sentence boundaries into segments that fit the window, and each session scores as its best segment. Best, not mean: a long session earns retrieval because one passage answers the question, and averaging dilutes exactly the signal being looked for. Short candidates produce a single segment, so nothing changes for them. rerank_long takes any object with .rerank rather than a concrete reranker, so segmentation is testable without loading a cross-encoder -- the zero-setup invariant means the default test path cannot import sentence-transformers. The regression test asserts the precondition first (whole-document scoring picks the decoy) and then the fix, so it fails if the old behaviour ever returns. Measured accuracy is not claimed here: quantifying the recovery needs a keyed LongMemEval run with reranking enabled, which is real API spend. What is verified is that the truncation is gone and that the answer-bearing session survives reranking.
…store
query_entity_ids() runs on every retrieval and walked every entity in the store to work out which ones
the query names. InMemoryGraphStore now keeps a (user_id, stemmed term) -> entity ids index, so only the
query's own terms are looked up.
With distinctive entity names -- people, places, companies, which is what a real graph holds -- the cost
stops tracking the store entirely: 0.004ms whether there are 500 entities or 10,000, against 16.05ms for
the scan at 10k.
The limit is worth stating plainly, because the first benchmark walked straight into it. Synthetic names
of the form "entity number {i}" share two tokens across every entity, so those posting lists are as long
as the store and the index degenerates to a scan with a lookup in front of it -- 1.4x, still linear. That
measurement was the benchmark's fault, not the index's, and it is written up in results/ so nobody
repeats the data and concludes the index does not work.
The alias-anchor filter (length, digits, stop words) tests the term rather than the entity holding it,
which is what lets the same decision come from a term index as from the full walk; it is now a named
predicate used by both paths instead of being inlined in one.
Equivalence is tested against the real retriever: the same entities in a store with the index and in one
with the lookup removed must anchor identically. Graph backends that do not implement the lookup keep
the scan, since that is all the GraphStore interface guarantees.
The architecture states a <50ms write path and a <100ms read path, and the project's discipline is that a number nobody can reproduce does not exist. Neither target was measured on the running service, and the token-saving claim came only from offline benchmark logs -- never from what the service actually served. engram/metrics.py adds sliding-window percentiles per operation, monotonic counters, and token totals. Pure stdlib and bounded memory: the window keeps percentiles describing current behaviour instead of averaging a regression away under months of history, and the process cannot grow by collecting samples. remember, recall, import and close_session are instrumented, plus a remember_degraded counter -- consolidation failures still return success, so without a counter a silently degrading write path looks perfectly healthy. This was rewritten against today's service rather than replayed from the branch it was preserved on, and that surfaced a real bug in the original: the savings ratio divided a full-history total by a context total drawn from a larger set of calls. Only the answer path computes a baseline, so nine cheap recalls plus one measured one reported roughly 1x -- memory appearing to save nothing -- where the comparable call showed 10x. The ratio is now computed from paired samples only, and is absent rather than invented until at least one pair exists. /metrics is unauthenticated like /health, which is a claim that has to hold: the payload is aggregate-only by construction, so it cannot reveal that a tenant exists, let alone what they stored. A test drives real traffic through the app and asserts the namespace and the content are both absent from the response.
…y half lean_context returns one flat string that callers drop into the user turn, so a multi-turn session re-sends and re-processes the parts that did not change -- the profile, the instructions on how to use a retrieved slice -- on every single turn. Splitting it lets the unchanging half sit in the system prompt where provider prompt-caching can reuse it, while only this query's evidence varies. The retrieved evidence is identical either way, so accuracy is unchanged by construction; this is a tokens-and-latency change and should not be expected to move a benchmark score. The whole thing rests on the stable half being query-independent, so that is the test: the same user's different questions must produce a byte-identical stable block, or caching misses every turn and the split costs more than it saves. Measured before claiming anything, and the numbers are more modest than the module this was preserved from implied. The pure split saves 5-9% across long sessions and is a net LOSS below about five turns, because the flat context is small enough that the first turn's extra structure never pays back. Its ceiling is simply the profile's share of the context. The session map costs more than it returns. It is content the flat context does not carry, so the first comparison was really flat-versus-flat-plus-a-new-block: -93% at one turn, breaking even around twenty. It is therefore off by default and opt-in for long sessions or when progressive disclosure is wanted for its own sake. Both non-adoptions are written into the optimization map rather than left implied by a default. Writing the tests surfaced a real wrinkle too: the usage guide told the model to consult a MEMORY MAP that a redacted context had removed, pointing it at something it could never be given. The map sentence is now appended only when a map is actually present. Not wired into lean_context or the OpenAI-compatible proxy: short sessions lose, and the published numbers came from the flat path. Callers choose per session length via Memory.layered_context().
Two different ways the multi-tenant surface was exposed. Without a rate limit one caller can spend the whole process, and once the LLM-backed paths are wired, the whole budget. Without idempotency a client that retries after a network timeout stores the same episode twice and pays to consolidate it twice -- the first request succeeded, only its response was lost. The limit is enforced inside auth() because that is the one place every protected route already passes through to learn who is calling, so a new endpoint cannot forget to be limited. /health and /metrics stay unauthenticated and therefore unlimited, which is deliberate: probes must keep working while a tenant is being throttled, or one busy caller takes the deployment down with it. Rewriting rather than replaying the preserved implementation caught a leak in it: the per-tenant hit map was a defaultdict that was never swept, so every tenant that ever called stayed in memory forever. That is the kind of slow growth that only bites the deployment with the most tenants, which is the one that can least afford it. Entries whose windows have emptied are now pruned, opportunistically so the server still needs no background thread. Two properties are easy to get subtly wrong and are tested directly. A rejected request is not recorded, because counting rejections keeps a retrying client's window permanently full and it never recovers. And the idempotency cache is keyed by tenant as well as key, so two namespaces choosing "retry-1" cannot read each other's response -- that one is a cross-tenant leak, not an inconvenience. Only successful responses are cached; replaying an exception would turn a transient failure into a permanent one. Both are in-process and documented as such: behind several replicas the effective limit is per_min x replicas and a retry routed elsewhere re-runs. Saying so beats shipping something that looks distributed and is not. Rate limiting is off by default, so the zero-setup demo and existing deployments are unchanged.
…in token Authentication was a static env map — edit ENGRAM_API_KEYS and restart to add a tenant — or open mode, where anyone is a tenant. A hosted deployment needs to mint keys while running and revoke them immediately, without ever holding the secret in a form a leaked file would expose. Keys are minted as sk-engram-*, returned exactly once, and only their SHA-256 digest is persisted. The admin surface fails closed: with ENGRAM_ADMIN_TOKEN unset there is no key-management endpoint at all, so an open-mode deployment cannot have tenants minted against it by whoever finds the URL. Resolution order is issued keys, then the static map, then open mode, so a revoked key cannot be resurrected by a stale env entry and existing deployments keep working untouched. Rewriting rather than replaying the preserved implementation caught a data-loss bug in it. A corrupt key file was swallowed and the store started empty — which rejects every issued key, and then the first issue() rewrites the file and destroys records that were only unreadable. It now refuses to load and leaves the file exactly as found: failing to start is recoverable, overwriting is not. Two smaller fixes came with it — last_used_at was mutated outside the lock, and the key file was written with default permissions even though it enumerates the tenants on the deployment. Because this is an auth surface the tests are mostly about refusing: admin absent unless enabled, wrong admin token rejected, revoked key rejected and revocation surviving a restart, an unreadable store answering 503 rather than falling through to another auth path, and neither the secret nor its digest appearing in any listing — publishing the digest would let anyone verify a guessed key offline. Lint caught a weak test on the way: the isolation case issued bob a key and never used it, so it would have passed even if bob's key resolved to alice's namespace. It now reads through the API as bob.
The agent ecosystem is Python-first and the only SDK was TypeScript, so a Python caller had to hand-roll HTTP against the API. This ships inside the core package and speaks over stdlib urllib, keeping the promise that installing Engram pulls in nothing. Method names mirror the TS client so the two read the same way and neither drifts into being the real one, and it covers what the server actually exposes today -- including /metrics, /v1/admin/keys and Idempotency-Key, none of which existed when the preserved version was written. Writing it surfaced a flaw in my own first draft: the transport returned (status, body), so EngramError.retry_after could never be populated -- Retry-After is a header. A field that is always None is worse than no field, because a caller writes a backoff around it and it silently never fires. The transport contract now carries headers, which is also what a custom transport needs in order to be a faithful substitute. The tests drive the SDK against the real application through the injectable transport rather than a mock. A mocked client test only proves the SDK agrees with itself, and agreeing with itself while drifting from the server is precisely how client libraries fail. So the rate-limit test provokes a real 429 and asserts retry_after survives, the idempotency test asserts one episode landed rather than two, and the admin tests check that a tenant key cannot mint tenants.
…tion actually do Three defences shipped over the last rounds and all three are invisible from outside. An operator cannot tell whether throttling has ever fired, whether clients set Idempotency-Key at all, or whether a rising tide of rejected keys means a misconfiguration or someone trying keys. /metrics now carries rate_limited, idempotent_replays, auth_rejected and auth_misconfigured. Deliberately aggregate, with no per-tenant breakdown. The endpoint is unauthenticated, and a counter keyed by tenant would tell any caller which tenants exist on the deployment — the same reason the rest of the payload carries no namespace names. The counters route through a helper that swallows its own failures, because instrumentation must not change behaviour. Building the service can itself fail on a misconfigured backend, and that surfacing from inside an exception handler would replace a precise 401 with a generic 500 — losing exactly the diagnosis the counter exists to support. auth_misconfigured is separate from auth_rejected on purpose: a 503 means the operator broke the key configuration, a 401 means the caller presented the wrong key, and an operator paging on the wrong one of those wastes the outage.
Runtime key issuance, rate limiting, Idempotency-Key, the defence counters and the Python SDK all shipped without a line of documentation. For a project whose pitch is that it is reproducible and self-hostable, a capability nobody can find is a capability that does not exist — and the audience most affected is the operator deploying it, who reads deploy/.env.example and never learns these knobs are there. API.md gains the key lifecycle, retry-safe writes and the metrics payload, each with the operational caveat that matters rather than just the happy path: the admin surface is absent unless ENGRAM_ADMIN_TOKEN is set, the plaintext key exists only in the issuing response, a corrupt key store answers 503 rather than starting empty, and both the limiter and the idempotency cache are in-process so several replicas multiply the effective limit and can re-run a retry. deploy/.env.example gains the new variables, with ENGRAM_ADMIN_TOKEN deliberately left commented out — an example file that hands operators a working admin credential would undo the fail-closed default it is supposed to explain. Both READMEs get the Python SDK, kept in sync as the contributing rules require. The copy sticks to what the code does — no competitor names, no scaling claims a committed log does not back.
The premise written into layered.py was wrong for this surface. The OpenAI-compatible proxy never put
memory in the user turn -- it puts the whole retrieved slice in the system prompt. So the split does not
buy a placement change here; it buys prefix stability. Provider prompt-caching matches on a prefix, and
with the per-query evidence sitting in the system block that prefix changed every single turn, so nothing
was ever reusable.
With the query-independent half moved to the front of the system block and this turn's evidence riding
with the user turn, the system block becomes byte-identical across a session: five distinct blocks over
five turns becomes one. The response reports cacheable_tokens_est so a caller can reason about it rather
than guess.
The honest measurement is that this does not save tokens. Compared like for like it is -1.1%, which is
noise. What it produces is a 61-token stable prefix and a property that did not exist before. Whether 61
tokens is worth having depends on a provider's cache-read pricing, which this measurement does not
model, so it stays opt-in via {"memory": {"layered": true}}: worth it across a session, pointless on a
one-shot call.
I walked into the same measurement trap twice. The first pass counted RECALL_GUIDE and reported +14%,
"the split is more expensive" -- but the guide is content the flat path does not carry, exactly the
flat-versus-flat-plus-a-new-block comparison that made the session map look bad last time. Fixing the
comparison also exposed a real duplication: the proxy already frames memory with _MEMORY_PREAMBLE, so
the library's guide was a second copy of the same instruction, costing 14% and changing nothing. The
proxy now passes guide=False, and the optimization map carries a warning about the trap itself.
Re-running an identical configuration moves 6-10 of 500 answers, so "83.6 to 84.4, we improved" has never been a finding on this benchmark — and a long list of mechanisms was retired on exactly that basis, one expensive run at a time. What was missing is an instrument that decides, before and after, whether a difference is real. eval/significance.py adds McNemar's exact test over paired per-question outcomes, a seeded bootstrap interval on the difference, and — the part that changes what gets run at all — the minimum detectable effect, computed from the item count and how often two systems disagree. Only disagreements carry information, so the usable sample is far smaller than the headline item count: 500 questions at a 22% disagreement rate resolve about 2.94 points, not fractions of one. Two results follow immediately, both in results/significance_headline.md. The published headline holds up: engram_lean 83.6% against full_context 73.2% is 81 wins to 29 over 110 disagreements, p < 0.0001, plausibly +6.4 to +14.4 points. That claim now has a test behind it rather than two percentages side by side. And the gap to the top of the leaderboard is 1.6 points, which is *below this benchmark's resolution at n=500*. It is not "not caught up yet"; it is not decidable with this measurement. Grinding algorithms against it produces noise, not rank. The optimization map now says so up front, with the consequence spelled out: only attempt changes expected to move more than ~3 points, and aim them at the two large weak categories (multi-session and temporal-reasoning, 248 of 500 questions between them), because a +10 in one category is only about +2.5 overall. Pure stdlib, exact binomial rather than a chi-square approximation — the discordant counts here are small enough that the approximation is wrong precisely where the decision is closest.
Everyone here knows the answerer flips "roughly 6-10 of 500" between identical runs. That number is the floor under every accuracy claim the project can make — a mechanism gaining less than it cannot be shown to work — and it has never actually been measured. The three committed lean logs are three different configurations, so none of them can supply it. eval/noise_floor.py takes two or more runs of the same config and reports what changed: how many answers flipped, in which direction, the widest accuracy gap between supposedly identical runs, and the smallest gain that survives as a consequence. Flips are the apparatus, not a regression, so both directions are reported as one phenomenon rather than as one run beating the other. It also warns when two "identical" runs differ by more than chance, because that is not noise — it means the runs were not identical, and averaging it in would bake a configuration difference into the floor. The runbook spells out what the measurement needs (two provider keys and the dataset, neither present in this environment), what it costs from the committed logs rather than a guess (4.78M context tokens and about 8.4 hours per run), and the requirement that the two commands differ only in --out. Before asking anyone to spend on this, the free check: the offline harness gives byte-identical results on this branch and on main (100.0%/70.0% accuracy, 5.4/14.2 context tokens), with lower latency from the negation early-out. So flips measured later are the answerer's, not this branch's.
… that misdirects reproduction Accuracy says how many questions were missed. It does not say whether the system had nothing to offer, was off by one, or answered confidently and wrongly — and those need different mechanisms. Designing against an aggregate means designing against a shape no individual failure has. eval/error_modes.py classifies each miss mechanically (abstained / numeric / wrong value) and reports what fixing each mode would be worth against what this benchmark can resolve. Run free against a committed log, before anyone is asked to pay for a run. Two findings on the headline log, both new relative to what was recorded before. The two large weak categories fail in different ways, so one mechanism cannot serve both: multi-session is 76% numeric errors (counting and cross-session aggregation) while temporal-reasoning is 63% abstentions (declining a question the memory could answer). Only the aggregate split was known before, which hid this. And the numeric errors go both ways — 16 under, 12 over — not the systematic undercount previously believed. That distinction decides the target: a one-sided miss means evidence was not recalled and recall expansion would fix it; a two-sided one means the evidence is largely there and the counting itself fails. The earlier belief would have aimed a mechanism at the wrong problem. Sizing against the 2.94-point floor: attacking either category alone lands right at the edge of measurability (19 questions = +3.8, 15 questions = +3.0). Both together are 34 questions, +6.8 points, comfortably resolvable. So the two lines have to be worked as one experiment. The tool's own first version had the bug it now warns about: it folded `_abs` items into their base category. Those are the benchmark's unanswerable variants, graded by a different judge, where refusing IS the right answer — counting them as abstention failures inflates the target and points the next mechanism at nothing. It now splits them exactly as report.py does, and the category counts agree. Separately, the optimization map pointed at results/headline_500.jsonl as "the current headline raw log". It is not: that file is a different run at 79.0/76.0. Anyone reproducing the published 83.6/73.2 from it would fail to match. Both real logs are now named, with a note that the two numbers come from two runs sharing the answerer, judge and item set.
The obvious reading of 34 abstentions is that retrieval failed to find anything, and the obvious fix is to retrieve more. The committed log says otherwise: the median retrieved context is 9,666 tokens on the questions the system refused and 9,600 on the ones it answered correctly — a one percent difference across every outcome type. The refusals were handed as much evidence as the successes. That relocates the problem. Whatever fails on those 34 questions happens after retrieval: either the right amount of the wrong evidence came back, or the evidence is present in a form that cannot support the reasoning asked for — dates retrieved, but no interval, duration or ordering derived from them. Either way, spending on recall expansion would buy nothing, which is what the previous reading would have recommended. error_modes.py now reports this split, so the check is one command rather than a one-off script. Separating the two remaining explanations needs the retrieved context itself, and the committed logs carry only its size. The harness already computes answer_session_hit; the existing context dumps happen to cover three of these questions, which is not a sample worth drawing from — noted as directional at most. A targeted re-run of just the 64 questions in the two failure modes would settle it at about an eighth of a full run, and that is written up as the one paid step this analysis actually needs.
report.py put two accuracy percentages side by side and left the reader to subtract them. On this benchmark an unchanged configuration moves several answers on its own, so a gap can be smaller than the apparatus that measured it — and nobody reliably remembers to run the test separately before believing a number. Every multi-system log now carries the verdict under the table. It immediately caught a presentation bug in my own code. The difference is computed B-minus-A, so when A was the winner the sentence said "A beats B by 10.0 points, plausibly [-13.4, -6.6]" — a reader cannot tell from that whether the effect is positive or negative. The interval is now reoriented to match whichever system the sentence names first, with a test pinning both directions. report.py runs as a script, so the repo root was not importable and the sibling analysis modules could not be reached; it now puts the root on the path the same way the other eval tools do.
…full detail Three separate signals now agree that retrieval is not what fails. Refusals get as much context as correct answers. The answer-bearing session is retrieved for 79 of 82 failures. And the numeric errors go both ways rather than systematically low. Each ruled out a different retrieval explanation; together they locate the problem after the evidence is in hand. Rank makes it sharper than membership did. The run rendered its top 2 sessions in full and compressed the rest to summaries, and 60 of 82 failures had the answer session inside that full-detail window. The answerer was looking at the raw evidence and got it wrong anyway — 65% of the refusals, 77% of the counting errors. That retires a whole class of work. Anything aimed at retrieving more is worth at most 3 questions, or +0.6 points, against a floor of 2.94 — unmeasurable by construction. The optimization map's lightweight n-hop/PPR expansion is downgraded on that evidence rather than on taste. Widening the full-detail window is also declined, with its own numbers. Going from 2 chunks to 15 would promote 19 questions from summary to detail, a nominal ceiling of +3.8 points. But 60 questions already failed *with* full detail, so the conversion from "shown" to "correct" is demonstrably well under one, and the token cost moves back toward the full-context baseline the headline claim is built against. Buying a smaller-than-stated gain by conceding the thesis is a bad trade. The check itself is LLM-free — session retrieval runs on the local embedder and BM25, and the benchmark labels which session holds the answer — so all of this cost nothing to establish.
…shows half of it The previous conclusion here was wrong, and wrong in a way that would have sent the next mechanism at the wrong layer. It recorded only the rank of the *first* matching answer session and concluded that 73% of failures had the evidence in full detail, so the failure must be after retrieval. Most of these failures are counting questions whose answer spans several sessions. For those, "the best-ranked answer session made the top 2" can mean one of four made it — which is precisely why the count comes out wrong. The right measure is coverage, not first hit. Measured properly: retrieval recall is 86%, but only 14 of 58 multi-session failures (24%) had all their answer sessions inside the full-detail window, at an average coverage of 48%. The evidence is fetched and then half of it is compressed to summaries, because assembly renders the top 2 by relevance while counting needs coverage. That also explains what nothing had explained before: why the numeric errors go both ways. Sessions compressed to a summary get missed (16 undercounts); an ambiguous summary line gets counted twice (12 overcounts). The earlier reading — that arithmetic itself fails — was a layer off. So the mechanism this points at is narrow and testable: for aggregation queries, let the full-detail window cover the evidence set rather than the top N by relevance. The trigger already exists and already works — plan_evidence().aggregation fires on 90% of the counting failures against 79% on the counting successes, so widening the trigger, the obvious next idea, is also ruled out. What it does not do is influence how many chunks get rendered in full. This is deliberately not "widen chunks to 15", which was rejected on cost: aggregation queries are about a fifth of the traffic, so the extra tokens land only where they are needed instead of pushing every query back toward the full-context baseline. Sized against the floor: 44 questions lack full coverage; at even a 50% conversion that is +4.4 points. The conversion rate is the untested premise and is checked next, offline — 60 questions already failed with full detail in view, so it is certainly not 1.
…says why Traced the counting failures to a one-line cause: evidence.py hands aggregation queries n_chunks=1, the lowest non-zero budget there is, while preference, procedural, exact-lookup, multi-hop and duration all get 2. On the committed headline log those failures needed a median of 3 answer sessions, and the planner had already generated a median of 3 subqueries to find them — then asked for one chunk. It looked like a one-line fix. It is not, and measuring it before shipping it is the point. Widening the budget lifts answer-session coverage from 38% to 56% at cap=5 and only 59% at cap=12, while rendered sessions go 2.0 -> 4.4. Raising the budget 2.4x buys three points of coverage, because the subqueries run out of distinct sessions at about four. Best case is +1.2 points, under a conversion of 1 that the same log refutes — below the 2.94-point floor either way, and paid for in tokens that push the lean context back toward the full-history baseline. No-go, recorded as such. The measurement also kills my own sizing. I had estimated +4.4 points from "44 questions lack coverage x 50% conversion", which silently assumed the mechanism would achieve full coverage. It reaches 56%. Estimating a gain from an unverified premise produces a wish, not a prediction. What it does establish is where the real constraint is. 89% of the answer sessions sit in the main query's top-15, and the round-robin selects 59% of them however large the budget. So the binding constraint is which sessions get picked, not how many may be rendered: interleaving subqueries by rank optimises for finding the most relevant session per angle, while a count needs every relevant session. Those two goals conflict at selection time, and that is the next hypothesis to test offline. The knob stays, inert at 0, documented with the numbers that say not to turn it on — a coverage-driven selector would want the budget it provides. eval/coverage_check.py mirrors the real detail_eps loop rather than a top-N simplification, because measuring a selection strategy the code does not have would have reported a gain that evaporates in production; a test pins that ordering. Also in this change, from the parallel-repair verification: the TypeScript client had retryAfterOf() defined and never called, so EngramError.retryAfter was undefined on every response including the 429 it exists for — the same trap the Python SDK hit earlier. Wired up, with tests that provoke a real 429. Removed EngramAdminOptions, whose JSDoc pointed at an EngramClient.admin method that does not exist, and thirteen type imports that no signature used. And eval/locomo10.json (2.8MB, third-party) is now gitignored: the existing data/ rule covered only the fallback path, not the documented one.
Left over from the lint pass. os was imported and never referenced; verified by grep and by running the script, and it is one of the 33 lint findings the CI gate would flag once that gate exists.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured work on the read path, plus the harness that measures it. Every claim traces to a committed log
with a reproduce command, and the results that came out negative are recorded as negative.
The read path was O(n)
eval/scaling.pymeasures it: ms per 1k facts holds constant at ~17–20 from 100 to 10,000 facts, and asingle query already costs 177ms at 10k — past the <100ms target, at a store size that is small next to
the 10M-token goal.
Bounded candidate retrieval (
engram/store/indexed.py, off by default) scores a candidate poolinstead of the whole store: a lexical inverted index, the vector store's search, and the facts hanging
off the query's anchor entities, unioned and slot-completed. With a pool at least as large as the live
fact count it returns bit-identical results to the full scan — the test that keeps both paths honest.
The tenant filter now pushes into the vector index. Multi-tenant retrieval filters by user on every
query, and the only way to express that was a Python predicate — opaque to the backend, so LanceDB
materialised the whole table every time. Configuring the scale backend never bought one sub-linear read.
Lifting
user_idinto a real column takes 75.38ms → 1.35ms at 10k rows and 302.36ms → 1.83ms at 40k,with latency essentially flat as the table grows.
Query anchoring costs the query, not the store.
query_entity_ids()walked every entity on everyretrieval. With a term index it is 0.004ms whether there are 500 entities or 10,000. The limit is stated:
entity names sharing high-frequency tokens degenerate the index to a scan.
Negation no longer triggers an entity scan. 509ms → 0.0009ms at 5,000 entities.
Correctness found on the way
Long sessions were reranked on their opening quarter. A cross-encoder reads ~512 tokens; a session is
~2000. The project had already documented this in
lean_context's comment, butretrieve_episodes()still passed whole sessions and no test covered that path. Sessions are now scored segment by segment,
each taking its best segment.
Observability
/metricsreports live latency percentiles, volume and token totals — the <50ms write and <100ms readtargets were assertions until something measured them. Aggregate-only by construction, so it stays open
like
/health; a test drives real traffic and asserts neither the namespace nor the content appears.What did not work
semantic channel was itself a full scan. That is why the tenant pushdown exists.
about five turns. The session map costs more than it returns until ~20 turns, so it is off by default.
entity. That was the benchmark's fault, and it is written up so nobody repeats the data and concludes
the index does not work.
Verification
458 → 503 tests, all passing; lint unchanged at its baseline; zero-setup check passes. Correctness tests
are built to fail against the old behaviour — the prefilter test hides the wanted tenant outside the
query's neighbourhood, and the rerank test asserts whole-document scoring picks the decoy before
asserting the fix. LanceDB tables written before the tenant column keep working by probing the real
schema instead of assuming it.
Accuracy is not claimed anywhere: quantifying the reranking recovery needs a keyed LongMemEval run.
Reproduce:
python3 eval/scaling.py— evidence inresults/bounded_candidates_scaling.md,results/entity_anchor_index.md,results/layered_context_tokens.md.