ISS-10812: seed a worktree's code index from the main checkout's (PLN-2070 PR 1)

[wongk]

Campaign iss-10812-20260921, chunk C1 of 2: seed a worktree's index from the main checkout's.

This PR builds PLN-2070 PR 1, for ISS-10812: code search in a git worktree answers from the main checkout's index.

What this lands

A linked worktree's code index is no longer built from scratch, and no longer left as the partial index that edit-triggered reindexes create today. The first time an engine opens for a worktree, its index is seeded as a copy-on-write clone of the main checkout's index. One incremental refresh then brings it in line with the worktree:

• Seeding (infra/code_intel/worktree_seed.py). The clone is taken under main's index lock and SQLite's own write lock, after a PASSIVE checkpoint, and swapped into place by atomic rename. The copy is marked with a seeded_from stamp and a repo_id_alias:<worktree id> row, so it keeps main's repo_id and shares its disk pages.
• The engine adopts the alias in CodeContextEngine.__init__. Nothing else in the engine computes repo_id.
• Re-seed, never rebuild. A missing, partial or stale-format worktree index is replaced by a fresh seed. Any forced rebuild of a seeded index is refused, and lc code index --reindex in a worktree re-seeds. A seeded index is never vacuumed.
• Main busy. If main's index is busy, the call reports index_state: rebuilding with a seeding note, and the next call seeds.
• Lifecycle. The daemon's engine cache retires a worktree engine once the worktree's .git file is gone, or after code_context.worktree_engine_idle_s (default 1800 s) without a request. Retiring stops its autosync and frees its cached scoped capabilities. Main-checkout engines are never retired.
• Zoekt. A seeded engine takes candidate files from main's Zoekt server. HEAD refresh now works in a daemon launched inside a worktree, where .git is a file.
• The CLI. lc code index checkpoints the index WALs after each reindex, so a seed usually finds a small WAL.

This PR changes no routing. Sessions still default to the main checkout until C2. Calls that already pass repo_root=<worktree> (175 in the transcripts ISS-10812 cites) now get a complete index.

Acceptance ledger

8 criteria: 7 met, 1 met differently, 0 not met.

Criterion	Verdict	Evidence
AC-1.1 seed by clone, no full build	met	factory → ensure_seeded → seed_worktree_index; test plus counterfactual
AC-1.2 first refresh re-extracts only what differs	met	extractions = the 2 differing files of 7; the rest get an mtime update only
AC-1.3 first answer ≤ 5 s, worktree-correct ≤ 60 s	met	symphony-alpha copy (17,870 files): open 0.108 s, first answer 0.897 s, worktree-correct 5.088 s
AC-1.4 partial or stale index re-seeded, never rebuilt or vacuumed	met	_seed_reason covers missing, stale and partial; the engine refuses the rebuild and skips VACUUM
AC-1.5 disk per worktree ≤ 200 MB	met differently	worst case 190.8 MiB (200.06 decimal MB), after the seed, the first refresh and 20 edits; see deviations
AC-1.6 RSS measured, idle default set, removed engine stops	met	main 145 MB; +1 engine +140 MiB, then about 13 MiB each; git worktree remove retired the engine on the next sweep
AC-1.7 Zoekt from main, HEAD refresh in a worktree	met	unit tests plus counterfactuals (live Zoekt is off on the measuring host)
AC-1.8 main-checkout engines unchanged	met	repo_id, engine_state and Zoekt root unchanged; main is never tracked for retirement

Measurements (a local copy of symphony-alpha; the real repo was never touched)

• Time: a seed takes 0.05–0.14 s to clone and 0.002–0.008 s to checkpoint. Today's live WALs would take 5.55 s to checkpoint (fts: 808 MB, 195,566 frames), which is why lc code index now checkpoints after each reindex.

• Disk, as the df -k delta on a dedicated APFS volume, cumulative KiB:

  Worktree	After seed	After first refresh	After 20 edits
  wt1	1,292	31,956	195,372
  wt6	172	3,644	193,868
  wt7	184	3,652	149,280

  Most of it comes from the 20 edits, through the engine's normal index writes.

• Memory (RSS):

  • Main alone: 145 MB.
  • Main plus 5 seeded engines: +190 MiB. The first engine costs about 140 MiB and each one after about 13 MiB.
  • After retiring one of the five: 327 MB, down from 340 MB.
  • Reopening an engine takes about 1 s, so the 1800 s idle default stands.

Deviations from the plan

1. The writer audit failed. code_context.sqlite and intel.sqlite are written at query time without the index flock. The plan's fallback, SQLite's backup API, would write a full private 5.8 GB copy, so the seed instead holds SQLite's write lock (BEGIN IMMEDIATE) on each main database while cloning. That gives the same guarantee that no commit is in flight, and it keeps the clone copy-on-write.
2. WAL handling. A -wal is cloned only when main's checkpoint could not fold it in, and it is folded into the staged copy before the swap. Each database is replaced by one atomic rename. The clone's retrieval cache is emptied, and its index_version is bumped past the index it replaces.
3. Seeding applies to any linked worktree, found through the worktree's own .git file, so a daemon launched inside a worktree seeds too.
4. Every forced rebuild of a seeded index is refused, not only the one a format mismatch triggers.
5. Retirement checks the worktree's .git file, not its directory. Measured: git worktree remove raced the first-refresh subprocess, which recreated the directory.
6. _retire_code_engine also drops cached scoped capabilities that hold the retired engine. Otherwise retiring frees no memory.
7. The WAL checkpoint moved off the seed's critical path, into lc code index, because it measured 5.55 s against the plan's 5 s threshold. engine.py is unchanged for this.
8. Disk was measured on a dedicated APFS volume. On the shared volume, df moved 548 MB during a seed that added 1.3 MB.
9. AC-1.5's 200 MB holds as MiB. It is 200.06 MB in decimal units.

Review

The deep review (5 reviewers, run cr-19457, at d78c751c) was NEEDS_ATTENTION, with 5 verified findings: 1 HIGH and 4 MEDIUM. It proved it read this worktree at that commit.

Fixed:

• HIGH, import cycle between worktree_seed.py and engine.py: the indexer semantics version moved into infra/code_intel/store.py (52a97bac).
• MEDIUM, two threads racing a worktree's first seed: check-and-seed now runs under one process-wide lock, marked lc-debt with a per-worktree upgrade path, and has a regression test (526da0f2).
• Found while pruning comments: the --reindex help and docstring now say what it does in a linked worktree (768c583e).

Declined, with the reasons verbatim:

• _flock duplicates the engine's index-write lock: "Both acquirers take flock(LOCK_EX) on the same .indexlock file, and that is the whole protocol: flock excludes one holder from the other whatever their poll loops look like, so the two cannot diverge on who holds the lock. What differs is each caller's wait policy, and that is deliberate. A seed runs on a code tool's request path and must give up after about 2 s so the caller gets IndexRebuilding and retries; an indexer waits LEMONCROW_INDEX_LOCK_TIMEOUT_S (10 s by default) and logs while it waits. The cited steal handling is a documented no-op ("retained for API compatibility but never bypasses a live holder"), so there is no engine-side behaviour the seed is missing. A shared helper would mean rewriting CodeContextEngine._index_write_lock, the lock every indexer in every process takes, to save about 20 lines. Not worth that risk now."
• Three .git-file parsers: "The duplication is real, but the validation differences are deliberate because the copies answer different questions. main_root_of has to reject submodules and bare repositories because it needs a main checkout to seed from. zoekt/server.py's _git_dirs has to accept them because it only reads the checkout's own HEAD: a submodule's gitdir holds its HEAD and has no commondir file, which is exactly the fallback _git_dirs takes. Unifying the parsers on main_root_of's validation, as recommended, would stop Zoekt reading HEAD in a submodule."

Re-review: the gate fired because 4 production files changed. The re-review (cr-31110, --since-last-review, at 768c583e) was APPROVED with 0 findings and proved it read that commit.

Noted, not changed: linked_worktree_of was moved unchanged from mcp_server.py. It resolves a relative gitdir: (from git worktree add --relative-paths) against the process cwd, so it returns None and fails safe to the workspace root. That behavior predates this PR.

Verification

• tests/infra/code_intel: 399 passed. This includes the unedited test_reads_during_reindex.py and 13 new seed tests.
• tests/core/test_code_context.py: 98 passed.
• The gateway code-tool suites (cache invalidation, code tools during reindex, edit handler): 96 passed.
• tests/gateway/cli/test_code_client.py: 5 passed.
• The mypyc compile-safety test: 3 passed.
• mypy on the 9 changed source files: clean. ruff: clean.
• Counterfactuals: 19 of 20 single-line mutations turned their test red. The 20th went red once paired with the branch that masks it.
• An independent check run by the campaign orchestrator on a throwaway repo, where the worktree deletes, renames and moves code and adds a new symbol:
  • after seeding and one refresh, the worktree index reflects all four changes;
  • the main checkout's index rows are unchanged.
• Not run here: the full suite, which CI runs, and live Zoekt, which is off on the measuring host.

What C2 still owes

PLN-2070 PR 2 routes each session's calls to its own worktree:

• a hook records the session's cwd;
• a per-request session root follows the precedence explicit repo_root → paths inside a worktree → the recorded session cwd → per-session bash cwd → workspace root;
• results name the checkout they came from;
• edit's "resolved against worktree" note appears only when true.

Until C2 lands, sessions still search the main checkout by default.