v0.1.7: the store path caps throughput at a few hundred MiB/s — write-once packs, page size or fewer stored bytes

[yifanxuaaa]

Status: Proposal for the next optimization round; not an implemented capability and not a performance claim. A Store-format change needs an owner ruling before it is implemented — see "Boundaries".

What the measurements say

The store half, not construction and not the wire, is what caps throughput. All numbers below are from docs/roadmap/0.1/0.1.7/evidence/, one sample per arm, on a host that was not idle (load ~6–7 of 14 cores).

Product phase split, measure_ingest, 512 MiB incompressible:

phase	time	rate	share
content construct (C1)	0.829 s	617.66 MiB/s	19%
accept (C2 save)	3.095 s	165.43 MiB/s	71%
publication commit	0.419 s	—	10%
end to end	4.349 s	117.73 MiB/s	100%

Counters: inserted=27845 packs_created=2229 pack_appends=11682 commits=13913, Store 520.7 MiB for a 512 MiB payload (~5.2 appends per pack).

The same code path with content that deduplicates stores 0.3 MiB and runs accept at 2213 MiB/s and the save at 1778 MiB/s, with construction unchanged at 641.99 MiB/s. The save half is proportional to stored bytes, not to CPU.

External SQLite mimic, same declared profile (MEMORY journal, synchronous=OFF, temp_store=MEMORY, busy_timeout=0), no product code, 512 MiB stored per arm:

pattern	rate
sequential 250 KiB BLOB inserts, 4 KiB pages	209.3 MiB/s
sequential 1 MiB blobs	242.3 MiB/s
sequential 250 KiB blobs, 16 KiB pages	272.9 MiB/s
one append to a growing blob	150.3 MiB/s
aggressive grow-to-2x (~5 appends)	46.2 MiB/s
same, commit cadence 16x coarser	50.9 MiB/s

So the engine's ceiling for this shape is a few hundred MiB/s, rewriting a growing BLOB costs 2–5x, a larger page buys ~30%, and commit cadence only ~10%. The product's 165 MiB/s sits inside that band, which is why this is a placement/format question rather than a layer-above question.

Levers, with prices

1. Write-once packs — never append to a placed pack; one INSERT per sealed pack, or one pack per sealed group. In the mimic this is 46 -> 209 MiB/s. Price: physical layout and the Store hash change, so it is an owner ruling. The #209 format round produced a related change (−1.55 s commit_ns, −1.30 s operation on stride10, every check green) and it was reverted by owner decision; that ruling is the precedent this issue must resolve first.
2. Larger pages (16 KiB) — ~30% in the mimic (209 -> 273 MiB/s). Also a Store-format change.
3. Fewer stored bytes — the dedup/delta path already exploits this; the lever is real (60x less storage, ~60x faster save), so representation selection and reuse coverage remain live work.
4. v0.1.7: the multi-writer transaction cadence costs the single-writer save path 2.06x on stride10 (42x more commits) #209's open remainder — stride10 is 24.72 s against 16.36 s for the pre-multi-writer model; the commit cadence is load-bearing for the second writer, so this is not a revert.

Refuted levers, so nobody retries them

• More writers: the budget ladder plateaus at 2–4 and falls at 8 (125.29 / 142.68 / 140.99 / 132.76 MiB/s at 1/2/4/8).
• A faster AEAD: the transport is 802 MiB/s (one stream) and 1556 MiB/s (two) against a ~100 MiB/s route — already 5–8x faster than what it feeds.
• Faster construction: 617 MiB/s against a 165 MiB/s save.

The instrument this round needs first

The product reports no pages-written-per-commit or write-amplification counter, so any placement change would be pre-registered blind: the mimic bounds the engine, but the product's own amplification is only inferable from pack_appends / packs_created. Before a format ruling is requested, add that observation (pages written per commit, and stored-bytes written per stored byte) so the treatment's mechanism is measurable in-product.

Boundaries

• Do not retune or relabel historical arms and receipts; #209 stays its own issue with its own evidence.
• Preserve authentication, bounds, failure/visibility semantics and the single construction producer per ordinary operation.
• No new durability, WAL or fsync work; no third-party patching.
• Every measurement row states its cache state, its interference and its gaps; one sample per case per arm.

Acceptance for this issue

• The instrument exists and reports amplification on a real save, with the counters named and bounded.
• An owner ruling records which of levers 1–3 is authorized, with the format-compatibility consequence stated (including what an existing schema-8 Store does).
• The authorized lever is implemented with matched arms: identical canonical results and refusal behaviour, committed Store accounting, and a measured change in the instrument's own units.
• Every non-passing row, every unrun route and every budget class is reported as plainly as a PASS.

[yifanxuaaa]

Cross-link from the #219 RCA handoff (705f1d961): the reference-era payload-create-100m row gives a phase split that this issue's store-path attribution predicts.

docs/roadmap/0.1/0.1.4/issue91-campaign/terminal-r26/campaign/r26-report-repair/report/report.md (line 241): Time 662.003 ms = Exec 282.778 ms + Commit 361.895 ms (the rest is unassigned). The 500 MB row is 2701.168 = 1215.713 + 1468.741. So ~55% of the create path is the database commit, on a route that also runs FUSE in a container.

That is the same term this issue's evidence isolates from the other direction: construct 617 MiB/s against save 165 MiB/s for a 512 MiB write, a dedup control that runs the save at 1778 MiB/s when only 0.3 MiB is stored, and a plain-SQLite mimic where the growing-BLOB append pattern costs 46–150 MiB/s against 209–273 MiB/s for sequential inserts.

Consequence for the levers here: squad S3 of the #219 handoff (commit attribution, one difference per arm) is the evidence this issue needs before any format lever is authorized. Read S3's output against the mimic baselines before pre-registering lever 1 (write-once packs) or lever 2 (page size); the write-amplification instrument in this issue's acceptance list should be designed to make S3's mechanism visible in-product rather than inferred from pack_appends / packs_created.

[yifanxuaaa]

Store-path finding from namespace-10000 (#219) — the cost is stored bytes, and the anchor bypasses admission

Cross-reference, not a closure. Full evidence in #219's S0/S1/S2 comment and docs/roadmap/0.1/0.1.7/evidence/issue219-s2-attribution-20260921T031259Z/.

Measured on a clean sealed build (b0260df3a, product seal b3e3cb7453…), two rows with identical file count (10,000) and identical scanned bytes (300 MB), differing only in content compressibility:

	pseudorandom	-text-v1
layerstack_init_ns	944.881 ms	708.991 ms
store_growth_bytes	304.7 MB	8.4 MB (36×)
store_database_bytes	304.8 MB	8.5 MB
user + system CPU	1788.8 ms	1531.9 ms
system CPU	752.0 ms	381.6 ms
initialize_admission_transactions	73	73
initialize_max_transaction_bytes	4.15 MB	4.16 MB

Two things for the store path specifically:

1. The initialization cost tracks bytes the Store must durably hold, not file traversal. The admission geometry is identical in both rows (73 transactions, same 4.15 MB maximum), while store growth differs 36×; the 235.890 ms wall delta is carried by system CPU (+370.5 ms) while user CPU actually falls 113.6 ms. The marginal rate implied by the contrast is ≈0.786 ms per MB of incompressible content — derived from a compressibility difference, not an isolated byte-path measurement.
2. The 100 MB anchor does not flow through the admission-transaction path. initialize_max_transaction_bytes is 4.15 MB, i.e. 24× smaller than the anchor alone, and the mean transaction across 73 is 4.14 MB (≈137 files each). Whatever carries the anchor into the Store, it is not the admission transactions that carry the 9,999 small files.

Host, not container: 1788.8 ms of host process CPU (getrusage(RUSAGE_SELF), including the initialization worker threads — CPU/wall 1.89×) against 12.0 ms of container command-window CPU. Disk read during the timed phase was 0.73 MB against 300 MB scanned, so this is not a storage-read cost — consistent with the row's declared reused-first-sample-uncontrolled cache state.

Caveat, stated plainly: the anchor's share of the timer and the per-file/per-byte split are NOT_MEASURED. No registered case varies anchor size at fixed file count or bytes at fixed file count, so the treatments that would separate them are pre-registered but unrun. No optimization was implemented; #219's plan forbids it before the gap is established, and its reference arm turned out not to exist as a distinct product (v0.1.6 and current main share 216 production files with zero content differences).

[yifanxuaaa]

Store-path finding: 17,378 commits for 25,245 objects — and half the gap is parallelism

Cross-reference from #219, not a closure. Full handoff: docs/roadmap/0.1/0.1.7/issue219-namespace-3x8-gap-rca-handoff.md.

A new v0.1.7 row now performs namespace-10000's work end to end — 10,000 files / 100 directories, ~300 MB persisted including a 100 MB anchor — and passes (gates=11, 14/14 pinned counters). Two store-path observations fall out of it.

1. Commit cadence differs by 238x, and it is the prime suspect for the CPU term

	v0.1.6 init_namespace	v0.1.7 pipeline-namespace-10000
objects	25,158	25,245
canonical bytes	302,182,831	301,171,810
transactions	73 admission transactions	17,378 commits
CPU (user+system)	1,788.8 ms	3,354.4 ms

Object counts agree within 0.35 % and bytes within 0.33 %, and v0.1.7 does strictly less inside its timer — v0.1.6's layerstack_init_ns includes constructing the 300 MB, while v0.1.7's excludes it under the C2 supplied-object rule — yet v0.1.7 spends 1.88x the CPU.

The mechanism that moved the counter is already attributed for another row: 7075f338d ("replace the fixed two-save model with a configured per-Store writer budget") made every step commit before releasing the arbitration lock (cas/lifecycle.rs:166-175). The same commit moved pipeline-filesystem-build's pipeline.commits from 1 to 43 and forced a pin re-baseline.

This is a hypothesis, not a finding. The handoff pre-registers the one-difference arm that would test it (hold objects and bytes fixed, vary only commit cadence) and requires it be run after the cheaper questions below.

2. SaveOutcome carries no byte field

Worth knowing before anyone writes another byte gate: SaveOutcome has inserted/reused object counts, packs_created, pack_appends, commits, statements, presence_queries — and no canonical-bytes field. Its chain member is ChainCounters, i.e. delta-base acquisition work, not bytes. A gate that reads outcome.chain.canonical_bytes gets a hard zero and looks like a save that stored nothing.

The new row therefore counts bytes from the objects on the way in (object.canonical_len()), reporting 301,171,810 against 300,000,000 declared. If a store-side byte total is wanted as a cross-check, the Store's own space accounting is the place it lives — pipeline-filesystem-build already reports canonical_bytes_total that way.

3. Ordering backing is required per batch, not per operation

MAXIMUM_WALK_ENTRIES = 4_096 forces a 10,101-binding tree into 3 operations, and each one needs its own FileBacking (fs::batch_backings, one scratch directory per batch per phase, created before the timer — FileBacking::new scans its directory for run tokens, which is harness work). Omitting it gives ResourceUnavailable { what: "ordering backing" }. The rejected batches also need backings, or the row fails in its oracle rather than its measured region.

Caveats

One sample per arm; two different workspaces (crates/ vs core/) so seals differ by construction; cache declarations differ and are not pooled; the 3-way split is forced by the product's own ceiling and its cost is NOT_MEASURED. Roughly half the 3.8x wall gap is a thread-count difference (CPU/wall 1.89x vs 0.94x), which is AGENTS.md §3.8's init_namespace multi-worker exception — and raising worker counts to move a number is forbidden. No optimization was implemented.