infra: re-derive worker, disk, and memory limits after streaming + delta

[IceRhymers]

Summary

Re-derive the indexer's concurrency, disk, and memory limits after the arc's structural changes land. Three load-bearing constants stop being true:

• Disk: REQUIRED_FREE_BYTES = 2.5 GB/worker assumes tarball + extracted tree coexist. After streaming ingest, peak is the tarball alone (~0.5 GB) — the runbook §3 table ("8 workers costs 20 GB") and the assert_disk_headroom guard are 5x too conservative.
• Memory / the semantic clamp: effective_workers clamps to 2 when semantic is on because embedding materializes a whole repo's chunks per worker. After delta indexing, only changed files' chunks materialize on recurring runs; after embedding concurrency, in-flight batches add a bounded, configurable increment. The clamp's premise needs re-measuring, and likely relaxing to the general index_concurrency.
• CPU: the "returns at the ceiling are sublinear (0.95x at 4 threads)" argument against index_concurrency: 8 dies with process-pool extraction; the remaining per-worker costs (network, DB) parallelize fine.

Approach

• Measure a full production run with the instrumented phases at current limits, then at proposed new limits, on the real corpus.
• Update: effective_workers (clamp logic), REQUIRED_FREE_BYTES / MAX_TARBALL_BYTES+MAX_EXTRACTED_BYTES framing in indexer/fetch.py, config.yaml comments (the 2.5 GB/worker arithmetic and the clamp note), and docs/runbooks/indexing-parallelism.md §3's tables and trade-off narrative.
• The engine pool derivation (pool_size == workers, max_overflow=0) follows automatically — verify the leak-detection property survives.
• Keep max_concurrent_runs: 1 and all single-writer invariants untouched — this issue tunes within-run limits only (the reconciliation coverage boundary in runbook §1.1 stays as written).

Acceptance criteria

• New limits derived from measurements, not extrapolation; before/after run times recorded in the issue.
• Runbook §3 rewritten to the new arithmetic; the disk guard error message's numbers match reality.
• Semantic clamp either removed (with measured memory evidence) or re-justified with the new bound.
• No change to run-level invariants (max_concurrent_runs, per-repo branch sequencing, reconciliation gate).

Depends on

• File-level delta indexing, streaming tarball ingestion, and embedding concurrency (the three changes that invalidate the current bounds).

[IceRhymers]

PR #120 re-derives the memory model behind the semantic effective_workers clamp and raises it from 2 to 4, confirmed both by a measured P_worst model and by two live-job runs each at clamp=2 (before) and clamp=4 (after) on the dev job, against a superset measurement corpus.

AC1 before/after (wall clock, live dev job, this corpus)

	Arm A (clamp=2, best-of)	Arm B (clamp=4, best-of)
wall	342.4s	305.5s (−10.8%)
peak self+children RSS	8116.7 MiB (90.6% of 0.7×M budget)	7475.5 MiB (83.4% of 0.7×M budget)

Both arms ran twice; each pair agreed within ~1.6 percentage points of budget utilization, so neither result is a fluke. No OOMs, no degraded semantic coverage warnings, no 429s across any of the 5 semantic-index runs this session.

index_concurrency's default stays 4 — ingest thread-scaling showed no real gain at 4 threads, so the clamp raise alone makes effective_workers == index_concurrency at the shipped default with no separate change needed.

Full derivation (coefficients, container-memory-ceiling measurement, both derived byte limits and why they were left unchanged) is in docs/perf/issue-109-measurements.md on the PR.