Indexing performance integration arc (#110)

[IceRhymers]

Indexer performance integration arc

Tracks #110. This is the protected integration branch handoff; it remains draft and must be reviewed and merged into master by a human.

Delivered child PRs

Issue	Delivery	Integration merge
#103	#112 — per-phase timing instrumentation	86e64d7
#104	#114 — file-level delta indexing	17aeb4f
#105	#118 — batch per-file DB statements	86f8e8b
#106	#115 — streaming ingest, #116 — ingest semantics tripwire	ad04748, 732a7d7
#107	#117 — order-preserving concurrent embeddings	5f290c6
#108	#119 — process-pool symbol/edge extraction	b6379db
#109	#120 — re-derived semantic worker limit	a68d509

All child PRs target integration/indexer-performance and use Refs #<issue> rather than closing issues before this integration PR reaches master.

Validation

• Combined integration SHA: a68d509a16540b3f763005d5b0ec27a297c4d2af
• Combined GitHub CI: green — lint, unit, webui; integration-lakebase is intentionally skipped.
• Fresh local bootstrap: uv sync --all-groups --all-extras and npm ci completed.
• Local combined gates: lint passed; frontend tests passed (209 tests); frontend production build and committed-dist freshness passed.
• Local Python unit/observability suite: 1318 passed, 1 expected inherited failure. The failure is the known semantics-tripwire comparison against origin/master: indexer/ingest.py was added within this integration arc and is deliberately guarded by test: watch indexer/ingest.py in the semantics tripwire (#106) #116 without an INDEX_SEMANTICS_VERSION bump. Child and combined GitHub unit checks are green with the correct integration comparison base.
• Fresh independent final review of master...integration/indexer-performance: APPROVE.

Known, accepted limitations

• indexer: process-pool symbol/edge extraction #108’s real-tarball measurement reached 2.72× combined speedup at four processes, below its literal ≥3×/four-process target; it exceeded 3× at eight processes. The indexer: process-pool symbol/edge extraction #108 exception was explicitly approved before merge.
• infra: re-derive worker, disk, and memory limits after streaming + delta #109’s clamp change is grounded in five semantic-index measurements on the DEFAULT-profile dev workspace. The dev schedule remained paused; the temporary measurement corpus/config was restored after the protocol.
• Lakebase integration is skipped in GitHub and cannot be fully reproduced by local vanilla pgvector because Lakebase-only extensions are unavailable there.

Exit criteria

• Every planned child is merged into this branch with verified green required CI.
• Combined integration CI and local mandatory checks were exercised; the sole local tripwire failure is documented above and proven inherited.
• Independent final integration review approved the combined diff.
• This draft records the delivered PRs, merge SHAs, validation, and limitations.
• Human review and merge of this draft into master.

After human promotion, manually close #103–#110 with this integration PR as provenance; they remain open today by design because child PRs use Refs.