fix(code-index): delete a reindex's stale rows in one pass per FTS table

[wongk]

Problem

An incremental reindex of symphony-alpha ran for more than 10 minutes and never finished.

The FTS5 tables (file_line_fts with 12.4M rows, symbol_fts, symbol_trigram, file_path_trigram) are keyed by rowid only; file_path and symbol_id are UNINDEXED columns. So _delete_file_index, run once per changed file, did a full scan of each table for every file: about 2.2 s per file. After this morning's pulls, 556 files were stale, which is about 20 minutes of scanning.

The autosync subprocess is killed at 600 s. Every run died before committing, the next run started over, and the live index had not committed in hours.

Fix

• _delete_files_index removes a whole batch's rows with one statement per table (… IN (SELECT value FROM json_each(?))). That is one scan per FTS table per run. The regular tables are indexed on these columns, so batching costs them nothing. All three call sites use it.
• _excluded no longer resolves each path when there are no exclude patterns (2.3 s per run).
• The incremental loop resolves each directory once, not every component of every file (_dir_cached_relpath). Its output is identical to _safe_relpath on all 19,749 files of two real repos, and a symlink case is tested separately.

Results

symphony-alpha, measured on a copy-on-write clone of the live index. The machine was under heavy load, so ranges are given.

files changed	before	after
0	~5 s	5–8 s
5	~16 s	9–12 s
50	~2 min	13–15 s
~830 (after pulls)	never finished (killed at 600 s)	72–91 s

Tests

• A structural test traces the SQL of an incremental run with 7 stale files and asserts one delete per FTS table. On main it fails with "7 stale files cost 28 FTS scans".
• A correctness test checks that no table keeps rows for a changed or removed file, and that an untouched file is unchanged.
• The core and code-intel suites pass. The one failure, test_index_pool_does_not_fork_live_parent_state, fails on main too (macOS uses spawn).

Related

One of three independent fixes for code search failing on symphony-alpha: fix/code-engine-autosync-leak, fix/incremental-reindex-batched-deletes, fix/search-reads-during-reindex. Each is based on main, and all three merge cleanly in any order (checked with git merge-tree, and their new tests pass together on an octopus merge).