feat(memory): the backend defaults to SQLite/FTS5, and an existing memory.json is imported once

[brcampidelli]

What

CHIMERA_MEMORY_BACKEND defaults to sqlite, and an existing memory.json is imported once into the new memory.db so nobody loses a memory. The second half of item 5 of the list audited on 2026-09-16 — the audit note said measure JSON × SQLite recall before flipping the default, and this PR is the measurement first and the flip second.

The measurement (bench/memory_recall, registered before it ran, US$ 0)

4,034 one-sentence facts (this repository's test names — one project's vocabulary, heavily shared words), 100 targets × 4 query styles (sentence control, needle, noisy, partial) × 3 seeds × 3 store sizes, both arms behind the production MemoryManager.search, k = 5, paired per query.

	recall@5 (worst cell)	disagreements @5	median search, N = 200 / 1,000 / 4,034
json	0.990	0–1 of 100 per cell	3.2 / 16.4 / 70 ms
sqlite	0.990	0–1 of 100 per cell	0.13 / 0.16 / 0.18 ms

All three registered predictions held: the control is 1.000 on both; |Δ recall@5| ≤ 0.01 on every cell (registered ≤ 0.03) — they rank the same quantity, an IDF sum, and disagree on ties; the JSON median is linear in N because the path re-tokenizes every stored fact and recomputes IDF per query, and recall runs on every turn. The registered decision rule — (a) sqlite recall@5 ≥ json − 0.02 everywhere, (b) sqlite latency ≤ json at N = 1,000, (c) the flip ships with an import — held on (a) and (b), so (c) is in this PR. RESULTS.md has the full table and what the bench cannot show.

Why a module and not a one-line default

There was no JSON→SQLite import in the product. The one-line flip would have left every existing memory.json unread — a memory that silently vanishes, the family of defect this project keeps a file about. chimera/memory/backend.py::open_memory_store:

1. Resolves — an explicit choice as written (json/sqlite, case-insensitive; a typo is the default and logged). The default is sqlite where this Python's SQLite has FTS5 and json where it does not: the SQLite store's LIKE fallback has no ranking, and a default must not trade ranked recall for unranked. Asking for sqlite by name on such a build gets the documented degradation.
2. Imports once — a memory.db that does not exist yet, beside a memory.json that does: every fact copied with id, kind, key, source, metadata, provenance, project and age intact (a tainted fact stays tainted, an old fact stays ageless), written to a sibling .importing file and renamed into place so a crash mid-import leaves no half store the next boot would mistake for a finished one. The JSON file is not touched — a copy, not a move.
3. Refuses to shadow — a memory.json that exists but did not load (MemoryStore.stale, a format from another version) stays the store in use; an empty database beside it would turn "these facts did not load" into "these facts are gone".

Both files existing means the owner switched before the default did: the database wins, the JSON is left alone. build_memory_manager, the migrate command, memory_key (the boot-vs-live store check) and the config/doctor readers all go through the resolver, so the Settings and Doctor screens name the store the turns actually read.

Found on the way

The SQLite store dropped created_at. MemoryItem has carried the age since test_a_memory_had_no_age.py, the JSON store round-trips it, and SqliteMemoryStore had no column — a fact written through SQLite came back ageless, on the field whose docstring says an old fact must never be given a plausible age. Added with a rebuild migration in the same shape as _migrate_project; old rows read None; the round-trip is pinned on the SQLite side now.

Tests

tests/test_the_memory_default_is_sqlite_and_nobody_loses_a_memory.py (10, skipped where FTS5 is absent): default resolves to sqlite with FTS5 and json without; explicit choice honoured, through kwargs and the environment; typo is the default; an existing JSON store is imported once with every field equal (== on the pydantic items), the JSON bytes unchanged, no .importing leftovers, a second open does not import again and a fact added after stays; a fresh install gets an empty database; both files → the database wins and nothing is re-imported; an unreadable JSON store is not shadowed (no memory.db created); a crash at the third add leaves no memory.db and no temp file, and the next boot imports all four; created_at round-trips through SQLite; a database built by hand with the pre-age schema migrates and reads old rows as ageless while keeping provenance and project; memory_key and read_config report the resolved store.

Sabotage: _import replaced by imported = 0 → two tests fail (the import test and the crash test); restored. Full gate on a clean copy in WSL: ruff clean, mypy clean (359 files), 6526 passed, 18 skipped, 10 xfailed in 182 s.

Operational note

On the VPS the next deploy will import /opt/data/…/memory.json into memory.db on first boot (logged: memory: imported N fact(s)), unless its .env sets CHIMERA_MEMORY_BACKEND=json explicitly, in which case nothing changes there. The JSON file stays where it is either way.

🤖 Generated with Claude Code