indexer: raise semantic worker clamp from 2 to 4 (#109)

[IceRhymers]

Summary

• Re-derives the memory model behind indexer/repo_config.py's semantic effective_workers clamp and raises it from 2 to 4, based on measured per-source-byte materialization coefficients, resident vector cost, indexer: process-pool symbol/edge extraction #108's per-process RSS, and a measured container memory ceiling — all showing N=4 clears 0.7x the container budget with margin.
• Confirmed empirically against the live dev job: two "before" (clamp=2) and two "after" (clamp=4) runs, each pair mutually consistent. Peak self+children RSS at N=4 landed at ~83% of the 0.7×M budget — actually more comfortable than N=2's own ~90–91% — with wall clock down ~11%. No OOMs, no degraded-coverage warnings, no 429s across all 5 semantic-index runs this session (1 priming + 2 Arm A + 2 Arm B).
• index_concurrency's default stays 4: ingest thread-scaling (scripts/measure_ingest_threads.py) shows no real gain at 4 threads (1.15–1.20x, short of the 2.0x threshold), and the clamp raise alone makes effective_workers == index_concurrency at the shipped default — no separate change needed.
• Both derived byte limits from the same model (MAX_EXTRACTED_BYTES, the per-repo chunk cap) are documented in code comments and the runbooks but left unchanged: no repo in the measurement corpus approached either the current or the derived value, so this run gives no empirical signal to justify moving a constant whose breach closes the whole run's reconciliation checkpoint.
• New indexer/job.py peak-RSS instrumentation (RUSAGE_SELF/RUSAGE_CHILDREN, logged once per run) made the live-job before/after comparison possible — there was no RSS instrumentation anywhere in the codebase before this.
• New test (tests/unit/test_job.py::test_shas_fn_connection_closes_before_embedding_starts) pins that the advisory shas_fn connection closes before embedding starts, since pool_size == effective_workers depends on each worker holding at most one connection at a time.
• Full derivation, every coefficient's provenance, the container-memory-ceiling instability finding, and all four live-job runs are recorded in the new docs/perf/issue-109-measurements.md.

Notes

• The dev workspace corpus was temporarily enlarged (3 extra large repos, per-repo chunk-cap overrides) to get a meaningful memory-pressure signal for this measurement. It has been fully restored: workspace config.yaml reverted to the pre-PR live config, the 3 extra repos retired via a normal (non-mass) reconciliation purge in a separate clean run. The tracked config.yaml's connections: block is untouched (comment-only changes elsewhere in the file).
• The dev job's schedule was PAUSED for the entire session and is still PAUSED. max_concurrent_runs: 1 and queue.enabled: true are untouched.
• tests/unit/test_semantics_version_tripwire.py::test_semantics_change_bumps_the_index_semantics_version fails on this branch. This is an expected, pre-documented false positive (indexer/ingest.py is in SEMANTICS_PATHS and postdates origin/master locally, per the test's own module docstring) — not a real extraction-semantics change from this PR, and not a stop condition.

Refs #109

Test plan

• uv run pytest tests/unit -q --ignore=tests/unit/test_webui_main.py — 1281 passed, 1 expected pre-documented failure (semantics tripwire, see above). test_webui_main.py is skipped locally for a pre-existing missing fastapi dependency, unrelated to this change (confirmed present on master/base too).
• make lint (ruff check, ruff format --check, mypy app indexer webui) clean, apart from the same pre-existing local fastapi mypy error (confirmed present on the base branch too, CI installs fastapi).
• Two live Databricks dev job runs at clamp=2 (today's shipped code) and two at clamp=4 (this PR's code), against a superset measurement corpus, with the new peak-RSS instrumentation — see docs/perf/issue-109-measurements.md for full numbers.
• Independent code-reviewer pass (fresh agent, no self-approval) — all findings addressed (CI lint gate, several doc/comment consistency issues, a measurement-script residency bug, added test coverage for the no-op-clamp-at-default case).