perf(#190): the pooled reader's scope, measured on all three rows — plus the save-path attribution and the stride1 scaling curve

[yifanxuaaa]

This branch now carries two lanes: the #190/#205 pooled-reader work it was opened for, and the #209 commit-time RCA (ledger L54–L62). Its base was retargeted from codex/190-corpus-phase-declaration — whose content is already on main — so this PR shows the branch's whole 16-commit diff against main.

Closes #209.

---

#209 — what the multi-writer regression actually was

The three causes, in order of size

1. The locator LIMIT (63fa15c49). Save-scoped publication means objects can hold up to SAVE_SLOTS rows per object_id, so the locator query gained ORDER BY o.object_id,o.save_id LIMIT ? to guard a duplicate case this corpus never has. The scoping itself is cheaper than the previous model's query (3.488 vs 4.559 µs); the LIMIT cost 15.5 µs per call × 380,380 calls. Removing it recovered 6.65 s. Fixed and kept.
2. The cadence multiplies how often the same bytes are written. A pack grows on every append, so sqlite3BtreeInsert compiles to OP_Delete + OP_Insert and rewrites the whole overflow chain every time. Inside a ~40-append transaction 39 of every 40 rewrites are superseded in the page cache; with one commit per append every one is flushed. The old model's commit was 5.5× cheaper per transaction (204.5 vs 37.3 µs) and still 7.67× more expensive per append.
3. The fixed per-transaction cost, paid 42× more often — BEGIN IMMEDIATE 9.7 µs (≈3.8 µs of it the eager RESERVED file lock), the COMMIT statement's setup, and the per-step statement parses ≈ 25 µs × 48,446 ≈ 1.2 s.

The cadence is not a tuning choice. With busy_timeout = 0 by declared profile, any overlap refuses the second writer rather than delaying it: COMMIT_EVERY = 8, 64 and 100000 all fail in round 0 with CleanupFailed { OwnershipUnavailable }. Widening the step does not slow the second writer, it loses its save.

The commit path, read with SQLite's own instruments (73e0b961c)

• What a step writes. 27.839 pages per commit on the product's own connection (SQLITE_DBSTATUS_CACHE_WRITE = 1,348,771 pages over 48,446 commits): 24.433 the pack body it grew, 4.74 the structural pages the growth forces (page 1's change counter, the objects primary-key btree, the objects_save index, the free-list trunk, the pack's own leaf — ≈245,000 predicted against 229,788 measured). Zero spills. COMMIT executes 3 VDBE opcodes and costs 56.1 µs, so its price is the page writes and nothing else.
• No statement does more than its rows and indexes require. FULLSCAN_STEP, SORT, AUTOINDEX and REPREPARE read 0 in all 21 buckets; every plan is a seek; the EXISTS subquery is a coroutine evaluated once for the one matched row. The 4 MiB pack cache is not thrashing either: 97.33 % hit rate over 227 distinct packs.
• BEGIN IMMEDIATE is 9.7 µs of every step and 0.47 s of the run for 5 VDBE opcodes; a deferred-BEGIN arm prices ≈3.8 µs of it as the eager RESERVED lock. Not removable — the watermark read that opens the step must happen under the write lock.
• Kept: the per-step policy write removal (704580673) — 48,191 of 48,446 watermark statements and 45,794 ceiling statements gone, Store byte-identical, commit_ns −5.9 % / −5.0 % in two windows, with tests/pack_watermark.rs pinning it from outside the crate (both cases fail on a structurally deferred watermark).
• Withdrawn: the per-step statement cache (341926632) — the bucket moved, the wall clock did not.

The format: measured at −52.8 % on commit_ns, then reverted by owner decision (f876242b2)

Reserving each lane's pack directory area at a fixed offset, recording the assembled length in the header and pre-allocating the row in reservation steps makes an append keep the payload the same size, so SQLite takes btreeOverwriteCell and its content comparison and dirties only the pages whose bytes changed: 3.353 pages/commit against 34.158 in a four-arm design probe. Matched pair, one sample per arm, one window: operation 26.021 → 24.720 s (−1.300 s, −5.0 %), commit_ns 2.938 → 1.388 s (−1.550 s, −52.8 %), per append 64.16 → 30.30 µs, every other bucket inside ±0.11 s. The prediction was beaten on direction and missed on level (1.388 s against ≤ 1.0 s); the residual is recorded as unattributed.

It is not in this PR. It was reverted by owner direction and retained whole at docs/roadmap/0.1/0.1.7/evidence/stage-6-history-209-format-20260920T222118Z/scratch/format.diff (21 files, +258 −82), so it can be re-applied without re-deriving it. It moves the Store hash and needs a format-profile and schema bump, which is why it was always an owner decision.

A measurement-protocol finding that outlives the issue

The machine's level moves by 55 % and the protocol's quiet preflight does not predict it. The archived, unchanged profile-1 binary read 16.739 s in L57's window, 17.550 s in the confirmation window and 26.021 s in the last window — same executable, same 45,794 appends, same 48,446 commits. All six rows compared passed the same gate (no named cargo/rustc/fs-bench competitor, ≥ 70 % idle) and the fastest window carried the highest load (8.38 against 5.42). Every future round must re-derive its absolute bar in-window.

What is not done

resolve_ns — the locator is 2.5 s of engine time over 297,082 save-path calls (8.48 µs/call against 3.488 µs isolated) and pack_bytes a further 1.1 s over 19,771 whole-body reads at 56.1 µs each; the cache hypotheses are dead and the gap is not explained. filesystem (6.7 s) has its own round. #190, #205 and #208 remain open.

Production LOC, #209 commits

commit	production LOC
63fa15c49 locator fix	25391 → 25388 (delta −3)
704580673 the step's policy write	25391 → 25403 (delta +12)
every other #209 commit (evidence, ledger, handoffs, pre-registrations)	unchanged, delta 0

Same counter (tools/production_loc.py) and scope over each commit's exact first-parent and committed crates + core/crates snapshots. The branch's product content against main is 5 files, +159 −11: the locator fix, the step's policy write, and tests/pack_watermark.rs.

---

---

The #190 / #205 half of the branch (unchanged)

One production change, retained and measured; the rest is the evidence, the attribution and a correction of an earlier attribution.

1. The product change (one commit, d57d6f9aa)

Resolver::resolve_charged built a PoolReader once per resolved inode leaf, so its pack cache (4 MiB) and decoded value cache (512 KiB) died with every leaf. The reader's lifetime is now its calling operation's: ReadSession owns one for the whole read operation (as it already owned the operation-scoped decoded-group cache), the save's own pool_reader — already shared by the pooled lane and the depth walk, already released on every pack write — serves read_batch, resolve_location and the selection input, and Store::read_batch keeps a reader for its one wave. No bound moves, no retained-bytes growth, no instrumentation, no other cache change, single construction worker untouched. PoolReadCounters::since keeps each chain's reported work its own.

The mechanism was confirmed before any product line changed, from the retained campaign's own counters: 99.7 % of stride10's 79,784 pack fetches re-read a pack the same state had already read, 7.99 GB copied for 45.3 MB of pack space (176×), and the operation-scoped decoded-group cache already served 95.1 % of record calls. Pre-registered in PREREGISTRATION.md before the first sample.

2. Measured, matched arms, one sample each

	pre-fix	post-fix	delta
stride10 operation	19.500 s	16.746 s	−2.754 s (−14.1 %)
stride3 operation	48.813 s	37.066 s	−11.747 s (−24.1 %)
stride1 operation	146.178 s	105.726 s	−40.45 s (−27.7 %)
ns per changed MB, 17 / 53 / 157 versions	52.3 / 68.6 / 85.0 M	44.8 / 51.6 / 61.5 M	slope 14.71 → 7.49 M per e-fold (−49 %)

The whole saving is the read/update path (filesystem −2,909 / −11,712 ms); storage.accept_loop is unchanged (+0.030 / −0.085 s). Counters moved exactly as pre-registered: pack_fetches 79,784 → 250 and 427,384 → 2,104; pack_bytes 7.99 GB → 16.6 MB and 25.11 GB → 80 MB — while physical_record_calls, physical_group_decodes, physical_group_cache_hits, chain_edges and leaf_requests are bit-for-bit unchanged. Retained against the pre-registered ≥1 s stride10 bar with no stride3 regression.

Equivalence on all three rows: every state root and content counter equal between arms, and both Stores hash to a recorded constant — stride10 4af37932…, stride3 f5c7ff5a…, and stride1 1635cf7b…, the last recomputed from the retained campaign's own stride1 Store. Sampled identity-matched verification per row, inside the 60 s budget: 1,083 / 3,377 / 9,996 path-states, 0 mismatches, 0 missing, 0 unexpected on every arm.

3. The attribution rounds (harness-only and evidence)

• fe17ac75a publishes the save's own counters per state from the SaveOutcome the product already returns (29 figures), so storage.accept_loop can be read as the work that happened.
• L50 sizes the write pattern the Store's shape implied: 44,141 whole-file payloads in groups of exactly one, 45,791 pack BLOB rewrites = 4.43 GB for 45.3 MB stored (97.8×), forced commits every ~40 appends — bounded by a synthetic replay at 1.03 s. Decision: no write-path treatment, because the best-sized candidate was around, not clearly above, the one-second bar.
• L51 measures stride1 and finds the residual depth term: at matched changed volume the ratio barely moves (1.54 / 1.84 / 1.54 / 1.32×), while per-resolution chain work and resolution counts grow.
• L52 corrects L50/L51: save.chain.* is MutationOwner::chain_total, which accumulates selection acquisitions and reuse resolutions (membership.rs::stored_canonical → owner.resolve_location). Per object written early → late at stride1: reuse resolutions 0.112 → 1.032 (9.21×), trials 0.677 → 0.745 (1.10×), chain objects per event 2.59 → 3.56.

4. Production LOC

commit	production LOC
fcfacdce3 mechanism + pre-registration (docs)	85725 → 85725 (delta 0)
d57d6f9aa the scope treatment (product)	85725 → 85778 (delta +53); reference 65417 unchanged, core 20308 → 20361
43f06aef7, eae3d839c, 67163427c, faec3a83b evidence (docs)	85778 → 85778 (delta 0)
fe17ac75a save counters (harness only)	85778 → 85778 (delta 0)

Same counter (tools/production_loc.py) and scope over each commit's exact first-parent and committed crates + core/crates snapshots, runtime SQL included, tests/harness/docs/tools/generated excluded.

5. Checks

On the treatment commit: core cargo test --locked 494 passed / 0 failed (72 binaries), clippy --all-targets clean, fmt --all --check clean, check_product_boundary.py PASS (122 files) with self-tests OK (6 ran). Harness: cargo test --locked 117 passed / 0 failed, release build PASS with the inherited unused_mut warning recorded, not fixed. Harness format is not clean and not claimed: ops/history.rs carries 20 pre-existing rustfmt hunks plus 9 in the added code — reformatting moves the built binary (embedded panic! locations), which would leave the measured receipts describing a source no longer in the tree, so the measured source is kept and the deviation stated. No CI, no tools/preflight.sh.

6. What this does not claim

Every history.* row is a diagnostic: admission INELIGIBLE, O3 INCOMPLETE, complete commands 35.8 / 61.9 / 158.5 s against the 15 s and 25 s classes, so all budget classes are NOT_RUN. stride1 runs the ordinary recording (the driver refuses per-state phase nodes above 53 states), so its spans sit inside the state total — the pre/post points on each row are computed the same way, which is what keeps the curve comparison valid. The v0.1.6 time comparison is untouched. Two stride10 rows reconcile INCOMPLETE with a measured out-of-clock cause (first execution of a freshly built executable, 0.7–2.0 s) plus one further unidentified out-of-clock cost, both outside the child's clock and neither affecting any operation figure. No pin was written, no budget class changed, no cap promoted, no selection shrunk, no cold claim invented, no case re-run for a better number.

7. Follow-up

The remaining work is the save path, which is now 50–55 % of the operation and has no time attribution at all — no span for selection, resolution, compression, placement or commit, so compression/chunking/CAS have no cost counter. That is filed as #205 with the instrument to build and the three candidate levers. #190 stays open for its owner-blocked qualification gaps (D1–D5) and the v0.1.6 reconciliation.