feat: reindex builds the new vector index beside the old one (generation swap)

[kevintseng]

Reindex stops deleting your embeddings before it has replacements

The last item on the UX arc, and the one with real user money in it.

Rebuilding the vector index dropped every stored embedding first and refilled
afterwards. The DROP committed inside openDatabase, before the refill loop
started, so a run that died part way — rate limit, dropped connection, Ctrl-C,
a killed process — left the graph half unsearchable, and on a paid API the
embeddings already produced had to be bought a second time.

Now the new index is built beside the old one. A staging generation is filled
while the live index keeps answering every query, and replaces it in a single
transaction only once every entity that should have a vector has one and nothing
failed.

• A failed run publishes nothing. The live index is what it was — never a
  half-new, half-old mix whose distances no longer compare against each other or
  against the dedup threshold.
• The embeddings already produced are kept. A second memesh reindex resumes
  and asks the provider only for what the first run missed. Measured in the test:
  two requests on the failing run, one on the resume.
• A generation started by a different provider or width is discarded rather
  than resumed. Vectors from two embedding spaces must not share an index.
• Switching provider needs no flag. Change the config, run memesh reindex.

Three sqlite-vec facts, measured before the design

Not assumed — each was probed against sqlite-vec v0.1.9 first, and one of them
killed the obvious design:

1. ALTER TABLE ... RENAME does not work on a vec0 table. It reports success
   and leaves the table unreadable: vec0 keeps four shadow tables (_chunks,
   _info, _rowids, _vector_chunks00) and the rename touches none of them.
   The first read fails with no such table: main.<new>_rowids. So the swap
   copies rows; it cannot rename.
2. Two vec0 tables of different widths coexist. This is what makes
   build-beside possible at all.
3. DROP + CREATE + copy inside one transaction really does roll back. With
   the swap forced to fail before COMMIT, a fresh connection still read the
   original table at its original width with every row present.

Fact 3 is also where this review caught itself: the first probe checked
sqlite_master for the table name after a rollback and concluded "transactional".
A table name returning proves nothing about the vectors. The real probe asserts
row count and vector width through a fresh connection on a file-backed database.

What was removed

The destructive branch is gone from the database-open path entirely — a dimension
disagreement now keeps the working index and records that a rebuild is owed. So
--vectors is retired rather than kept as a no-op: its only purpose was
consenting to that drop. The consent machinery it needed
(allowVectorIndexRebuild, the path-scoped one-shot grant, ~55 lines) is deleted
with it.

The pre-flight probe survives for a different reason: a provider that cannot
produce a vector at the configured width fills nothing, and finding that out one
entity at a time wastes the whole run.

Namespace-scoped reindex still writes in place. A staging table holding one
namespace would drop every other namespace's vectors when swapped in, and
in-place is safe for the reason a full rebuild was not: each row's old vector
survives until its replacement exists.

Provider requests are bounded (first commit, stands alone)

The two provider calls used a bare fetch — no timeout, no retry, no backoff. A
provider that accepted the connection and never answered hung a whole rebuild
indefinitely, and a 429 was indistinguishable from a 500 or a 401, so a rate
limit produced the same silent null as a bad key while the run kept spending
entities that would all fail identically.

Now: 30s per request; up to three attempts for 429/5xx honouring Retry-After;
immediate stop naming the status for 401/403/404, which is configuration rather
than weather.

Tests

The swap now has the crash-injection case this path never had. A trigger
refuses the metadata write at the last step — after the drop and the copy, the
worst possible moment — and the previous index is asserted intact by row width
and dimension stamp, not by table existence. The technique already existed in
this repo (migration-atomicity.test.ts) but had never been applied to the
vector path, despite db.ts's own comment describing the historical bug a
mid-transaction kill caused.

Three suites moved from the old contract to the new one. The partial-run test now
states the stronger guarantee: a partial run leaves the live index untouched,
rather than merely not corrupting it row by row.

node scripts/run-tests-isolated.mjs   exit=0   Test Files 152 passed (152) / Tests 2220 passed (2220); no "Errors" line
npm run verify:release                exit=0

One process note

Three times in this session I read output that could not have shown me
otherwise: a grep | head whose window was filled by unrelated matches (twice),
and a CLI test suite that spawns from dist/ and therefore passed against the
build from before the edit — printing the exact error message that had just been
deleted. Rebuilding turned 1 green into 3 red. Recorded so the next session
suspects the observation before the conclusion.

---

Second round: what an eight-lane review found, and what changed because of it

The first two commits were reviewed by eight independent lanes — a structured
pass, six checklist specialists, a red team, and two adversarial passes. The
mechanism held up; the semantics did not. Everything below is fixed in
23ae9434 and da2412b8.

The mechanism was confirmed, empirically

• A child process SIGKILLed inside the swap transaction (busy-loop, not
  sleep, after INSERT…SELECT) left the original index readable from a fresh
  connection with every row present and its original width.
• DROP TABLE on a vec0 table tears down all four shadow tables plus both
  autoindexes, and re-CREATE at a different width leaves no residue — which is
  why rename was correctly rejected.

One root cause with several faces

The swap installed exactly the staging index, so any state that existed only
in the live index was treated as non-existent.

• A memory captured mid-rebuild lost its vector. Every writer except the
  rebuild targets the live index, and the rebuild works from an entity list
  snapshotted before it started. Worse silently: a memory edited mid-rebuild
  had its fresh vector replaced by the older staged one, which the missing-vector
  count cannot detect because the row is present. Now: still-active rows absent
  from staging are carried across — same width only, since during a width change
  a concurrent write is refused as a dimension mismatch.
• A memory deleted mid-rebuild could come back. forget clears the live row
  and knows nothing about a staging index. Now: staged rows for non-active
  entities are pruned before the copy.
• memesh doctor could report health over a graph owed vectors. The swap
  deleted pending_reindex itself, pre-empting the post-rebuild measurement, so
  a rebuild that finished with vectors missing printed "the reindex-needed flag
  was left set" and then left nothing set. The marker now has one owner.

Also fixed

• Two runtime messages told the user to run --vectors, the flag this PR
  deletes, in the exact situation the mechanism exists to handle. A new test
  scans every advice-printing source file for a memesh <cmd> --flag the CLI
  does not register; it caught an older second instance on its first run,
  memesh doctor --verbose, a flag that never existed.
• Semantic search being off during a width change is now reported. A
  width-mismatched query raised inside sqlite-vec, the error was swallowed into
  an empty result, and empty is indistinguishable from "found nothing" — so
  recall claimed mode: "hybrid", degraded: false for a vector side that
  answered nothing, for the entire window until a rebuild finished. Now
  mode: "fts", degraded: true.
• BUSY_TIMEOUT_MS 5000 → 30_000. The swap's row copy is the atomicity
  guarantee and is O(rows): measured 5.4s at 20,000 vectors, 9.1s at 30,000,
  so past ~16,500 vectors a rebuild made concurrent writers fail rather than
  wait — reproduced, a hook losing its capture after 5213ms.
• providerFetch now refuses redirects. A 307 forwards the POST body — the
  user's memory text — and OLLAMA_HOST is an unvalidated env var.
• generationRowIds no longer reports a failed read as "nothing staged", a
  number that decides promotion and what a resume re-buys. startedAt survives a
  resume. Both generation entry points reject a non-positive-integer width before
  it reaches DDL.

Cleared by the review — stated so nobody re-spends the time

No SQL injection path in any added interpolation (all are a module constant or a
value from a fixed integer map); no secret in any new log line, and the URL is
never logged; Retry-After bounded against every hostile form including
HTTP-date and negatives, worst-case sleep 60s; no timer or listener leak
(AbortSignal.timeout is unref'd); timeout and bad key are distinguishable. And
embedder.model cannot mix two embedding spaces — embedWithProvider has
exactly one caller and it hardcodes model: undefined. Two lanes reported that
last one as critical; it is not.

Known and deliberately not in this PR

Each needs its own change and its own test:

1. A resumed run trusts a staged row with no content fingerprint, so an entity
   edited between an interrupted run and its resume keeps the older vector.
2. The reindex loop has no circuit breaker: a provider failing at entity 50 of
   20,000 still attempts the rest (worst case ~91.5s per entity), and the backoff
   is linear rather than exponential.
3. res.json() is awaited outside providerFetch, so a body-phase timeout is
   not retried and is not named as a timeout.
4. An abandoned staging generation is never reclaimed and is invisible to
   doctor.

Verification

node scripts/run-tests-isolated.mjs   exit=0   Test Files 153 passed (153) / Tests 2226 passed (2226); no "Errors" line
npm run typecheck                     exit=0
npm run verify:release                exit=0
node scripts/audit/verification-audit.mjs  exit=0  (C1 new=0 C3 new=0 C4 new=0 C5 new=0 C6 new=0 C7 new=0)

Break-test of the three swap fixes, mutation applied then written back and
re-verified — carry-forward removed → KILLED, staging prune removed →
KILLED, swap deletes pending_reindex → KILLED.

Codex contributed nothing to this review: both passes returned
You've hit your usage limit … try again at Aug 20th. The auth probe reports
ready because credentials are valid — it cannot see a spent quota. The
cross-vendor lane was substituted with independent Sonnet passes, which is a
different model but the same family, and weaker than genuine cross-vendor
review.