indexer: file-level delta indexing keyed on (path, content_sha) (#104)

[IceRhymers]

Refs #104

Summary

Makes indexing incremental at the file level. Today any HEAD move on a branch
rewrites every file, deletes/reinserts every symbol and edge, and re-embeds
every chunk. This adds a per-file classification inside index_repo's
existing transaction — unchanged (write nothing), membership-only (a
row with this (path, content_sha) already exists under another branch of
this repo; union the branch array in), or changed/new (today's full write
path) — gated on the branch's stored index_semantics_version already
matching INDEX_SEMANTICS_VERSION. indexer/job.py gains a matching advisory
read so the embedder is only ever called for files the delta path will
actually rewrite.

Executed from the critic-approved plan at .omc/plans/issue-104/approved-plan.md
(not checked in — .omc/ is gitignored); this PR body summarizes its outcome.

What changed

• indexer/store.py: read_repo_content_shas / read_indexed_shas
  (statements 3a/3b), the provenance gate (statement 4,
  _repo_is_wholly_at_current_version), three-way per-file classification,
  batched membership-only UPDATE … RETURNING (_union_membership), the
  delta write set … INFO line, and a base-case fix: a run that parses zero
  files no longer advances index_semantics_version (only
  last_indexed_commit) — without this, a zero-parse run straddling a
  version bump could manufacture a false "current version" base case and
  serve stale extraction under delta forever.
• indexer/job.py: an injected shas_fn seam (default
  read_indexed_shas), called on a short-lived connection before embedding,
  gated on the same stamp check; narrows _precompute_chunk_writer's input
  to not-already-carried files; a covered-set guard on the chunk_writer
  closure (warn-and-skip rather than delete an uncovered path's chunks); a
  BranchOutcome.semantic_degraded flag and an aggregate run-completion
  WARNING naming every branch whose chunk precompute failed this run (no
  longer self-healing on the next run under delta indexing).
• Tests: new tests/unit/test_store_delta.py (statement-inventory tests
  against a fake connection) and tests/integration/test_store_delta.py
  (row-identity proof, small delta, multi-branch dedup, semantics-version
  mismatch, the provenance gate, both directions of the branches/
  repo_branches membership invariant, the zero-parse stamp fix, and an
  EXPLAIN proof the pre-read is index-served); tests/unit/test_job.py gains
  the shas_fn seam/embed-narrowing tests and the degraded-branch WARNING
  tests; six existing tests/integration/test_store.py re-index tests
  updated for the delta gate (four value updates; two deliberately re-force
  the gate CLOSED via UPDATE repo_branches SET index_semantics_version = NULL rather than relaxing their assertions, since they prove the
  unconditional-delete guard the delta path would otherwise skip);
  tests/integration/test_store_chunk_writer.py gains the chunk-side delta
  cases (Lakebase-deferred, see below).
• Docs: docs/runbooks/indexing-parallelism.md gains the delta write set … line, a correction to the force-reindex SQL (repos →
  repo_branches — the skip seam has only ever read repo_branches, so the
  previously-documented UPDATE repos SET index_semantics_version = NULL was
  a no-op), and the two accepted regressions below with their remedies;
  docs/runbooks/semantic-enablement.md documents that re-enabling
  semantic.enabled no longer backfills chunks on its own; the
  INDEX_SEMANTICS_VERSION bump obligation (app/db/models.py) is extended
  to the embedding model / SEMANTIC_EMBEDDING_DIM, and
  app/config.py's semantic_max_chunks_per_repo comment records its new
  per-run scope.

Accepted regressions (documented in-code, in the runbook, and here)

1. A branch whose chunk precompute fails keeps partial chunk coverage
   indefinitely, not just until the next successful run — only a changed
   file re-embeds now. Mitigated with the aggregate WARNING above; remedy is
   clearing that branch's semantics stamp.
2. semantic_max_chunks_per_repo is now a per-run bound, not a
   per-branch-corpus bound — re-enforced in full on every first index and
   every semantics bump.

Neither is a defect in the mechanism; both are consequences of "unchanged
files are never rewritten" that weren't true before.

Scope discipline

• indexer/languages.py, indexer/symbols.py, indexer/parse.py (the
  semantics-tripwire-watched files) are untouched. INDEX_SEMANTICS_VERSION
  stays 4.
• git diff --name-only origin/integration/indexer-performance...HEAD — verified,
  no watched file appears.
• No Alembic migration, no new dependency, no TEMP TABLE, no change to the
  transaction shape, the CAS baseline, StaleIndexError, the sweep, or
  reconciliation.
• Filed #113
  for the pre-existing gap this work surfaced (the semantics tripwire doesn't
  actually run in CI — the unit job's shallow checkout can't resolve
  origin/master) — out of scope for this PR, linked rather than fixed here.
• Issue indexer: file-level delta indexing keyed on (path, content_sha) #104 and epic [Epic] Indexer performance: delta indexing, batched writes, streaming ingest #110 bodies updated to record the provenance gate, the
  zero-parse stamp fix (a narrow, deliberate carve-out from indexer: file-level delta indexing keyed on (path, content_sha) #104's declared
  "Out of scope: … branch-level stamp semantics"), the two accepted
  regressions, and that extraction is not skipped (epic items about total
  wall-clock time restated as db-write + embed time, proportional to the
  delta — item 4's db-write half is what this issue delivers; indexer: single-pass in-memory tarball ingestion (drop extract-to-disk) #106/indexer: process-pool symbol/edge extraction #108 own
  the parse-side cost).

Gates (fresh, this session, against codesearch-pg / pgvector:pg16)

Gate	Result
make lint	green — ruff check, 136 files formatted, mypy: no issues in 36 source files
make test	green — 1187 passed, 260 deselected
make test-integration (local Postgres)	204 passed, 8 failed, 43 errors

The local integration gate is not "all green" — this repo has no local
Postgres image with the Lakebase-only extensions (lakebase_vector,
lakebase_ann, lakebase_bm25) or SET ROLE grant fixtures six modules
need, and that's pre-existing, not introduced here. Every failure/error is
one of:

• test_semantic_rrf.py (12), test_reconcile.py (17), test_migrations.py
  (11), test_webui_semantic.py (2), test_commit_search.py (2),
  test_mcp_server.py (1) — identical, by name, to the pre-existing
  baseline recorded before this branch's work started (190 passed / 8
  failed / 41 errors on integration/indexer-performance @ 649ba80).
• test_store_chunk_writer.py — 6 errors, not the baseline's 4: the
  +2 are this PR's own new chunk-side delta tests, which need the same
  lakebase_vector fixture as the rest of that module and are Lakebase-
  deferred by design (see that module's docstring) — reasoned through, never
  locally verified.

No other module regressed. tests/integration/test_store_delta.py (new, 12
tests) runs and passes locally in full, including the EXPLAIN access-path
proof (ix_files_branches_gin for the branch-carried read;
Index Only Scan + Heap Fetches: 0 on uq_files_repo_path_sha for the
repo-wide read — the pre-committed downgrade path for an unattainable
Heap Fetches: 0 was not needed).

Review

An independent code-reviewer pass (separate context, full diff against the
plan) returned no CRITICAL/HIGH findings after actively trying to break
the delta-skip induction, the zero-parse fix, the provenance gate, and the
advisory/authoritative read divergence handling — all held. It found 2
MEDIUM + 8 LOW issues, all fixed in the "Address review pass 1" commit: a
test that the delta gate had silently neutered (re-indexing byte-identical
content at the same head_sha made the file classify unchanged, so the edge
write path it was meant to test never ran), vacuous EXPLAIN assertions
(missing VERBOSE, so Postgres never populated the field being asserted on),
a semantic re-enablement documentation gap, and several doc/comment
precision nits. Gates re-run clean after the fixes (see table above).

Measurement (issue AC5)

Semantic/embed phase can't run locally (no embedder configured), so per the
plan's fallback this measures the index_repo db-write phase directly —
iter_source_files + extract_file over a real, actively-developed corpus
(this repo's own working tree), run twice: once as a first index, once at
identical content and a new head_sha (the real-world "HEAD moved, nothing
changed" case).

Before (indexer/store.py at 86e64d7, pre-#104, 205 files):

run1 (full):      wall=1.084s counts=IndexCounts(files=205, symbols=2122, swept=0, edges=12021)
run2 (unchanged): wall=1.133s counts=IndexCounts(files=205, symbols=2122, swept=0, edges=12021)

After (this branch, 218 files):

run1 (full):      wall=1.157s counts=IndexCounts(files=218, symbols=2207, swept=0, edges=12681)
run2 (unchanged): wall=0.008s counts=IndexCounts(files=218, symbols=0, swept=0, edges=0)

Before this change, a re-run at unchanged content costs the same as a full
index (no measurable improvement — 1.084s → 1.133s). After, the unchanged
re-run's db-write cost drops ~145x (1.157s → 0.008s) and writes zero
symbol/edge rows, confirming AC1. Extraction/parse cost is unaffected either
way, as documented (#104 does not skip parsing — see the epic-body note
above).

Blockers

None. Ready for review.