v0.1.7: retained-history read optimizations and remaining qualification gaps

[yifanxuaaa]

Retained-history performance: current progress and next-agent handoff

Status: five improvements merged; the read-path attribution round is complete and its one treatment is retained. Qualification is the next round, and its first two items are owner decisions. Latest product checkpoint: 4049e28b6 with the LOC record in 3cec2f8da, PR #202. Timing results are single-sample diagnostics, not release admission. Keep this issue open.

Start here

Handoff published in PR #201, commit 9fe8eb290072d09594cde3ba67c3bbd1963d5e56.

Next-agent handoff: data access, reuse boundaries and execution pipeline. It contains source/corpus pins, raw evidence, rejected directions, required measurement and correctness proofs, reproduction guidance and lock protocols.

That handoff has now been executed for its Priority A: the read path was attributed, one treatment was selected from the attribution and retained, and the pre-registered range-read candidate was not selected for a measured reason. Priorities B (reuse boundaries) and C (pipeline) were reviewed and remain unimplemented proposals.

Next-agent handoff: admission qualification for the retained-history lane. It carries the diagnosis of admission INELIGIBLE and g1.o3-pinned-counters = INCOMPLETE with exact sources, names the three structural blockers in severity order, separates the two owner decisions from the engineering, and fixes what must not be done to make the labels move.

Qualification status — what is still missing, and who can supply it

Neither label is a product defect, neither is caused by any retained change, and both were present in every earlier round.

Gap	What it actually is	What closes it
g1.o3-pinned-counters = INCOMPLETE	The harness's verdict that no frozen oracle exists for the history.* rows: tests/golden/expected.tsv pins 217 case ids, and src/families/history.rs:1-9 places this group deliberately outside the 217, --smoke and --lane full. Not drift, not a miss.	An owner decision that these rows carry a frozen oracle, a qualifying baseline run through the runner, an invariance argument per pinned counter, and regeneration via shared/pin_expected.py.
admission INELIGIBLE	A declared stance, not a gate — no INELIGIBLE record exists in any trace. The rows declare CacheState::CreatedInSample, and the only enforced cache contract (shared/cold.py) applies solely to init_namespace/namespace-100000.	An owner decision on admission mode and budget for these lanes, plus a declared and verified cache stance using the existing residency/disk_read_bytes primitives.

Three blockers, in severity order: (1) there is no admission path for these lanes — runner.py:136-147 grants ADMISSION_MODE only to --lane full, and the history.* lanes are not in full, so a history invocation is always sampled and INCOMPLETE whatever else is fixed; (2) the measured complete commands (38.1 s stride10, 73.1 s stride3) exceed even the 25 s declared exception ceiling; (3) the cache stance is a modelling question, because the workload is a same-process write-then-read replay whose Store is created in-sample, so the open question is whether CreatedInSample can be verified, not whether a cold claim can be manufactured.

Explicitly not to be done: adding the history rows to registry::cases()/FROZEN_CARDINALITY, changing HISTORY_LANES membership in full without the owner, hand-editing a pin, inventing a cold claim, moving reads into setup, shrinking a selection to fit, or promoting the 120/240 s diagnostic caps to admission rows.

The owner considers one second worthwhile, and accepts a small allocation overage for good time reduction. Preserve all measured misses; no unspecified tolerance or unlimited storage waiver is implied. The owner also explicitly stopped cross-task coordination messages once the lock mismatch was fixed: use the shared locks directly, without contacting #192 or arranging resource windows.

Completed and retained

PR	Improvement	Production LOC delta
#194	Batched directory-parent lookups and bounded authenticated base-record reuse	+49
#195	Pooled physical-group telemetry and reuse through the existing 512 KiB cache	+141
#196	Existing prepared-statement cache for catalogue queries	−1
#199	Group compression level 19 → 1; payload level 3 unchanged	0
#202	Ordinal-ordered pooled-leaf resolution: one catalogue statement per distinct covering group instead of one per covering-group change	+3

Current product total at this checkpoint: 85,725 LOC, comprising reference 65,417 and core 20,308. Source-size accounting excludes tests, comments, harness and documentation and includes runtime SQL. The improvements are separate campaigns; do not compound their percentages.

PR #197 tested and reverted the zero_count_serials New-row filter: operation difference 156,679,583 ns, target-phase difference 53,181,962 ns. It did not meet the revised one-second criterion. PR #198 completed the legacy source comparison that motivated #199; its proposed codec experiment is now completed, not still pending.

Latest matched measurement: group level 19 versus 1

Quantity	Stride10 baseline	Stride10 retained candidate	Stride3 baseline	Stride3 retained candidate
Operation ns	22,506,420,003	20,914,489,418	58,410,516,121	56,577,520,957
Operation reduction ns	—	1,591,930,585	—	1,832,995,164
Store begin + accept + finish ns	11,050,102,332	9,364,846,704	22,624,366,586	20,161,248,171
Apparent Store B	49,053,696	49,324,032	61,767,680	62,152,704
Allocated Store B	49,651,712	50,249,728	62,783,488	62,152,704
Verification work ns	5,617,963,792	5,405,493,708	18,824,407,917	18,904,055,292

Observed operation reductions: 7.073228815545978% / 3.138125265325286%, one sample per case/arm. Save reductions are partly offset by increased read work. Exclusive codec CPU remains NOT_MEASURED. The large stride10 complete-wall reduction included unrelated untimed corpus-read variation and is not attributed to compression.

Stride10 numeric allocation misses are 307,200 / 905,216 B above 49,344,512 B; matched growth is 598,016 B, accepted under the owner's time/space ruling. Apparent growth is 270,336 / 385,024 B. Both stride3 allocated readings meet 64,024,576 B. Its allocation decrease despite larger apparent content is filesystem variation, not an algorithmic storage reduction. Candidate stride3 verification is 79,647,375 ns slower, still within 20 seconds.

All 70 roots and complete sorted ObjectId/role/canonical-length inventories match. Independent review rehashed both measured binaries and all four Stores and reproduced inventories/arithmetic. Physical Store bytes intentionally differ. All four identity-matched verifiers pass 10/20-second targets with 1,083 / 3,377 sampled paths per arm and zero mismatch/missing/unexpected; verification is not exhaustive.

494 core tests, 117 harness tests, 12 example targets, core Clippy/fmt, the 122-file boundary guard, six guard self-tests and locked builds PASS. Inherited unchanged harness Clippy/fmt failures remain unresolved; they were not rerun or relabelled as passes. No CI or retired preflight. Measured binary/source identities remain distinct from the comment-corrected final build.

Retained measurement: ordinal-ordered pooled-leaf catalogue resolution

Campaign stage-6-history-190-read-20260920T042617Z, base 9fe8eb290072d09594cde3ba67c3bbd1963d5e56, PR #202. One sample per case/arm, matched instrumented baseline versus candidate with identical instrumentation.

Quantity	Stride10 baseline	Stride10 retained	Stride3 baseline	Stride3 retained
Operation ns	21,905,900,168	19,888,424,711	56,597,343,506	49,501,013,748
Operation reduction ns	—	2,017,475,457 (9.209735%)	—	7,096,329,758 (12.538274%)
Filesystem ns	10,008,188,499	8,228,260,706	32,620,645,790	25,336,780,793
Provider ns (nested)	9,019,067,220	7,254,356,455	31,150,817,381	23,865,118,886
Catalogue statements	278,927	65,337	1,334,277	368,074
Catalogue interval ns	2,180,465,764	584,740,322	10,443,893,588	3,315,711,491
Complete command ns	40,220,718,250	38,101,144,250	80,483,422,875	73,127,405,000
Verification work ns	5,593,199,541	4,606,757,792	19,864,253,708	16,126,274,334

The attribution put 2,180,465,764 ns of the stride10 provider's 9,019,067,220 ns in that catalogue lookup, at 7,817 ns per statement. The mechanism is a false premise in the code: the covering-group memo assumed a leaf's rows are in ordinal order, but they are ordered by serial, so a leaf issued 24.8 / 31.8 statements against only 5.80 / 8.77 distinct covering groups. The retained treatment visits the rows in ordinal order; the statement count became exactly one per distinct group.

All 17 / 53 state roots and the canonical and value-group inventories match, and the saved Stores are byte-identical between the arms and equal the retained L40 level-1 candidate Stores. Stride10 allocated 50,249,728 B still misses the historical 49,344,512 B target by 905,216 B, inherited from L40 because this treatment changes no stored byte; it is not relabelled. A first treatment shape (one statement per leaf ordinal span) was measured and rejected: stride10 −2,298,597,419 ns but stride3 +1,884,341,075 ns. Operation and region figures are summed over all selected states including the first, matching phases-perf.operation_ns and the L40 table; an earlier revision of the campaign's derived JSON excluded the first state, moving the reductions by at most 4,543,959 ns, and is corrected in the campaign's SUPERSEDED-ANALYSIS.md rather than hidden. The stride10 provider's unattributed remainder is 1,305,940,532 ns.

Admission remains INELIGIBLE, every O3 pinned-counter row INCOMPLETE, and the historical v0.1.6 comparison unresolved. Retained validation: 494 core tests, 117 harness tests, 12 example targets, core Clippy/fmt, 122-file boundary guard, six guard self-tests PASS.

Remaining gap: tree algorithm aligned, execution details differ

Latest stride10 operation partitions exactly into:

Disjoint interval	ns
File-content construction	1,607,234,583
Filesystem update	9,569,275,834
Store begin + accept + finish	9,364,846,704
Other child work/bookkeeping	373,132,297
Operation sum	20,914,489,418

Authenticated provider reads occupy 8,609,430,602 ns inside the filesystem interval. They are nested, not additional time. Latest counters show 79,784 pooled pack fetches / 7,985,771,449 application bytes, 65,337 value-group decompressions and 2,723 physical-group decompressions. Application bytes are not disk I/O, and provider time is not a measured excess over legacy.

Area	Established difference	What remains unmeasured
Data access	Legacy reads pack control data plus a selected group range; current pooled reads copy whole pack BLOBs. Current full-directory validation must survive any change.	Exclusive acquisition/copy/parse cost and attainable selected-range saving.
Reuse boundaries	Legacy shares a bounded pool reader across targets in a metadata group; current pool readers are per leaf, with an already-shared physical-group cache.	Repeated value/pack reconstruction attributable to lifetime boundaries, and safe bounded reuse benefit.
Execution pipeline	Legacy pipelines producers/admission; the current harness stages construction then admission and copying.	Removable copying/staging cost under the unchanged single-worker and C1/C2 contracts.

Both tree implementations already apply incremental sorted changes and reuse untouched subtrees. Reviewed stride10 validation read zero base-directory pages in every state; a whole-directory scan is not an established cause of this lane. Legacy's 342,355,542 ns namespace counter excludes dirty-directory work charged to Content. It cannot be compared directly with our entire filesystem phase. Both implementations rewrite whole open-pack BLOBs.

Historical Commit is 11,370,679,212 ns. The current 9,543,810,206 ns difference (1.8393351028606961×) is an unmatched historical comparison: timer boundaries, effective workers, cache/machine state and canonical object streams differ or are unrecorded. The full causal gap remains unresolved.

Next-agent work

Done in the read-path round (PR #202, ledger L42):

• Attribute pack acquisition/copy, complete directory parsing, physical/value-group decoding and reconstruction, with named residuals and real read counters.
• Select one justified bounded treatment from that attribution. Selected-group BLOB reads were not selected, for the measured reason recorded in the campaign.
• One matched stride10 pair, a stride3 confirmation, separate identity-matched verification, an independent re-derivation and exact LOC.

Open, in priority order:

• Qualification, and it starts with two owner decisions. See the qualification handoff: whether the history.* rows carry a frozen oracle at all, and whether admission mode plus a longer frozen budget extend to these lanes.
• Review within-operation reuse separately (an operation-scoped pooled pack cache is a real 2.1–3.9 s target but needs an append-invalidation contract). Do not move a cache hit before required validation without a precise authenticated lifetime argument.
• Consider pipeline changes only after measuring copying/ownership cost; distinguish a harness change from a product improvement. No helper-worker workaround.
• The pre-registered selected-group BLOB range-read screen is still NOT_RUN; it needs its own fetch/parse attribution before implementation.

Do not repeat the completed codec experiment or the rejected zero-count filter, do not resurrect the measured-and-rejected leaf-span catalogue shape, and do not start subtree summaries, a wider persistent cache or a schema/pipeline rewrite from a source diagram alone. These are continuation plans, not proof of another saving.

Evidence and remaining qualification

Qualification handoff · Read-path campaign · Data-access handoff · L40 report · Results · Inventory · Independent review · Legacy comparison · Active ledger.

History O3 remains INCOMPLETE and cold/performance admission remains INELIGIBLE, both now diagnosed with exact sources in the qualification section above; historical attribution remains open. Large Store/binary payloads are local-only with explicit custody, while traces/receipts/source patches are committed. Earlier receipts, misses and issue comments remain evidence for their own campaigns. This status update supersedes old next-action instructions, not the recorded measurements.

[yifanxuaaa]

The handoff prompt is written, and it starts from data rather than from scratch

Prompt: docs/roadmap/0.1/0.1.7/component-decoupling/history-slowness-root-cause-prompt.md (9f35c49ad).

It states the issue exactly, hands over everything already measured, names the confounds, ranks the hypotheses with the measurement that settles each one, and authorizes a parallel subagent campaign.

The head start — and the finding that shapes the whole investigation

The per-state child body has only four named spans in the entire run:

span	Σ	share of the 32.566 s operation
content (C1 construction)	1,115.2 ms	3.4 %
storage.finish	843.3 ms	2.6 %
storage.begin	8.6 ms	0.0 %
store.create	2.5 ms	0.0 % (state 1 only)
unnamed inside the children	~30.6 s	94.0 %

94 % of the measured operation sits inside the per-state child body and outside every named span, so no attribution is possible until that hole is closed. Splitting it is the campaign's critical path.

Per-state cost tracks the base tree, not the state's change: r = 0.9586 against oracle entries, 0.9605 against logical bytes, 320.9 µs per oracle entry (the 17-row per-state table is in the prompt). Each state changes a small delta yet costs more as the history grows — the shape of a walk over the base tree. A clue, not a finding.

What the prompt hands over

The phase receipt (operation 32.566 s, root 44.832 s, harness work inside the timer 12.265 s, CPU 39.66 s, RSS 264,536,064 B), the Store and its save counters, the three tiers and where our side is thin (stride3/stride1 have wall only; stride1 has no receipt at all), the v0.1.6 record with its container boundary, the five confounds (composition, worker rule, cache state, machine state, selection), and the noise bound measured across five behaviour-identical runs (per-case median 16.6 % relative range; lane total 75.4 → 79.2 s).

Hypotheses, each with its decisive measurement

H1 the tree update walks the base tree per state · H2 composition (our timer brackets work their Commit did not) · H3 codec level 19 and the 16 MiB workspace · H4 the single-worker rule · H5 the persisted index and chunk cursor · H6 save/transaction cadence. Falsification counts as a result.

Subagent campaign

Five parallel squads with non-overlapping scopes and one document each — S1 span attribution (the critical path), S2 tree-update path (source-linked), S3 C2 cost as matched-pair diagnostics, S4 the v0.1.6 Commit boundary reconstruction, S5 the matched-pair protocol and the unmatched-comparison register — then a synthesis that must close the arithmetic with residual 0, and a read-only adversarial reviewer that re-derives the headline numbers and tries to falsify them.

Boundaries the prompt sets

No product change is authorized (harness and documents only); no third-party patching or vendoring; one sample per case per arm, no n3, no best-of; diagnostics labelled and reported beside the gate sample; history-stride1 never optimized; the measurement lock respected; receipts append-only with a fresh --out every time; no timing claim becomes a gate; and NOT_RUN with the reason is required behaviour, not a failure — inventing a mechanism to close the issue is the failure.

[yifanxuaaa]

Execution kickoff — steps 1–5 now requested

The owner has now explicitly requested the subagent optimization campaign, including product batching/bounded reuse, correctness proof, stride3 confirmation, issue updates after each step and production LOC reporting. This supersedes the earlier investigation-only restriction for these changes.

• Subagents own harness read-work instrumentation, product batching/reuse, and external correctness/work-reduction tests. One coordinator serializes builds and measurement under the shared locks.
• First freeze an instrumented baseline; product edits wait until its binary and runs are retained. Same harness/corpus/switches across arms, one sample per case/arm, no stride1.
• Prospectively freeze raw diagnostic execution ceilings at 120 s (stride10), 240 s (stride3); verification hard wall remains60 s and its work targets remain10/20 s. These execution ceilings do not relax ordinary gates or turn the uncontrolled-cache diagnostics into admission evidence.
• Actual provider read waves/objects/returned bytes and elapsed read work will be recorded separately from logical demand counters. No per-read telemetry-node explosion.
• Initial production LOC: reference65,417 + core20,116 =85,533. Method: python3 tools/production_loc.py --json, excluding tests/docs/tools/harness and legacy inline tests, including runtime SQL.

Campaign: docs/roadmap/0.1/0.1.7/evidence/stage-6-history-190-opt-20260919T232858Z/. No optimization or new timing result is claimed yet. No format, codec, worker-count or validation relaxation is planned.

[yifanxuaaa]

Step 1 — fresh stride10 phase/read-work diagnostic retained

One baseline sample completed with frozen instrumentation and unchanged product source. Raw diagnostic only: cache residency is uncontrolled, admission/cold-performance comparison remains INELIGIBLE.

Interval/counter	Exact value
Sum of17 state operations	47,161,768,126 ns
Filesystem interval	34,686,235,752 ns
Validation	3,429,981,622 ns
Directories	11,506,307,751 ns
Inodes	4,470,610,374 ns
Filesystem residual outside six named children	15,278,643,093 ns
Provider-read elapsed, nested inside filesystem	33,559,231,582 ns
Filesystem provider waves	110,715
Requested/returned canonical objects	117,533
Returned canonical bytes	286,247,942 B
Provider connection opens / group decodes	16 /513
Complete command wall	66,657,423,500 ns

Provider time overlaps filesystem phases and must not be added again. It includes local Store/SQLite acquisition, reconstruction and authentication; it is not pure disk or codec CPU. This is a fresh baseline, not a rerun that replaces the original32.566-second historical receipt.

The new public-API structural regression test fails on the unchanged product as intended: for five parents in one leaf, omitted parent metadata requires15 acquisitions vs10 for explicit identical metadata (batch64), with equal canonical roots. Two accompanying correctness tests pass. Read-work observer test passes after correcting its initial missing-docs compile failure; all attempts retained.

The repeated-parent batching/reuse proposal is now supported by observed read work and the deterministic regression. Baseline stride3 collection is in progress; a constrained ordering-quota matrix will be run on baseline before product edits, because earlier value insertion could change spill order.

Production LOC unchanged: reference65,417 + core20,116 =85,533 (delta0). Baseline binary SHA256 c5826d20f217a73ed4d814b82c3f8d8cdcaaa7af94ead18c0291918c88d4a855.

Evidence: .../stage-6-history-190-opt-20260919T232858Z/runs/baseline-history-stride10/; exact commands/identity/pre/post machine observations/timeout are retained. Raw O3 pinned-counters gate remains INCOMPLETE; no release claim.

[yifanxuaaa]

Steps 2–3 checkpoint — first implementation rejected by quota parity

Implemented bounded parent batching plus immediate reuse of fetched inode values in one product file. The focused shared-leaf case improved from15 to6 object acquisitions, and canonical/correctness assertions passed. However, comparing all56 quota cells against baseline exposed 8 previously successful configurations now refusing with ObjectLimitExceeded:

• pending1, ordering bytes864/960, batch1/64;
• pending2, ordering bytes960/1152, batch1/64.

The change inserted directory values into the reducer earlier, increasing transient spill demand. This candidate is rejected before any performance sample. No ceiling was increased. The generic matrix test passed because it allowed explicit resource refusals; baseline-supported cells are now being locked into regression assertions.

Revision in progress: preserve original global reducer insertion order and contents map, batch both parent acquisition passes, retain only a bounded final parent-record window for reuse. This intentionally permits batched rereads beyond the retained window; it does not add an unbounded all-parent cache.

Rejected-v1 production LOC: reference65,417 unchanged; core20,116→20,132 (+16); combined85,533→85,549 (+16). Method tools/production_loc.py, tests/harness/docs excluded. This is an intermediate rejected snapshot, not the final LOC delta.

Baseline stride3 completed: operation125,277,281,254ns; filesystem99,950,581,082ns; provider reads98,406,450,952ns,210,380waves/230,323objects. Baseline read-back: stride10 work8,124,985,000ns meets10s; stride3 work29,241,424,750ns is TARGET_MISS against20s. Sampled read-back and absent pinned counters do not yield admission.

Evidence includes candidate-v1-rejected.patch, candidate-v1-quota-differential.json, all check logs and baseline raw files under stage-6-history-190-opt-20260919T232858Z/. Independent reviewer confirmed the quota regression. No candidate performance run yet.

[yifanxuaaa]

Step 2 — bounded lookup batching implemented; Step 3 — bounded record reuse implemented

The revised product change is frozen in core/crates/layerfs-content/src/filesystem/update.rs; architecture section5.7 is updated.

Step2: use existing lookup_many for ordered parent batches, limited by existing base_read_batch and the existing4096-demand ceiling. The final metadata-overlay pass also batches missing records. Validated parent order already guarantees uniqueness, so no redundant sort/dedup layer is added.

Step3: retain only the final nonempty authenticated parent window for reuse. Earlier omitted records are reread in bounded batches; there is no all-parent cache. Preserve caller metadata precedence, original global reducer insertion order, contents map, validation and existing quotas. At most two record windows coexist. No format, public API, dependency, codec or worker change.

Shared-leaf test	batch1	batch3	batch64
Baseline omitted metadata	23	17	15
Candidate omitted metadata	22	10	6
Baseline explicit metadata	18	12	10
Candidate explicit metadata	18	9	6

All4 focused tests pass. All56 quota outcomes match baseline exactly, including success roots and error type/limit/actual. The8 baseline-supported cells now have explicit regression assertions. Independent reviewer parsed the logs and confirmed parity. The rejected first implementation remains retained; it was never performance-sampled.

Production LOC: changed file 405→454 (delta+49); only this product file changes. This implies core20,116→20,165 (+49), reference65,417 unchanged, combined85,533→85,582 (+49); the full-scope recount follows workspace checks. Counter/method: tools/production_loc.py, production-only Rust/runtimeSQL, no tests/docs/harness.

Step4 is running: full core checks, candidate build, then one candidate stride10 and one stride3 diagnostic with the frozen baseline harness. No measured candidate speedup is claimed yet.

[yifanxuaaa]

Step 4 — correctness, work reduction and stride3 confirmation complete

One frozen-harness diagnostic sample per arm/case; uncontrolled cache, no cold/release admission claim.

Selection	Baseline operation ns	Candidate operation ns	Difference ns	Read waves before→after	Objects before→after
history-stride10	47,161,768,126	27,386,866,665	-19,774,901,461	110,715→66,616	117,533→74,279
history-stride3	125,277,281,254	79,607,320,336	-45,669,960,918	210,380→111,853	230,323→135,296

All17 and53 state roots match baseline. Both final Store files are byte-for-byte identical to their respective baselines; all retained save decision counters and transaction counts match (1,149/1,470 commits). Non-provider filesystem work counters are unchanged. This isolates the delivered change to reduced repeated parent acquisition, not skipped canonical/validation work or altered stored content.

Correctness:490 core tests and117 harness tests pass; examples compile/test, core Clippyalltargets-Dwarnings, core fmt, boundary guard and self-tests pass. Four focused parent tests pass; all56 constrained quota outcomes exactly equal baseline. Earlier rejected-v1 regression and obsolete work-count assertion failure remain retained. The materialization expectation changed107→104waves and303→300objects because one repeated three-page parent path is eliminated; widebatch/inodepage/goldenroot assertions are unchanged.

Sampled read-back: stride10 baseline8,124,985,000ns / candidate8,000,840,000ns,1,083 compared each, zero mismatches/missing/unexpected. Stride3 baseline29,241,424,750ns / candidate29,978,286,084ns,3,377 compared each, zero disagreements. Both stride3 verifier rows remain TARGET_MISS against20s; all fit60s hard wall. This is sampled, not exhaustive read-back.

Storage: apparent bytes unchanged49,053,696 and61,767,680. Baseline stride10 allocated49,688,576 exceeds49,344,512 (FAIL); candidate49,192,960 is below. Baseline/candidate stride3 allocated62,402,560/62,103,552 are below64,024,576. Since paired database bytes are identical, allocated differences are not claimed as an algorithmic storage saving. Raw O3 pinned-counters remains INCOMPLETE on all four performance rows.

Final production LOC: reference65,417→65,417(0); core20,116→20,165(+49); combined85,533→85,582(+49). Same production-only counter, runtimeSQL included; tests/docs/harness excluded. One product file changes, no new dependency/format/worker policy. No commit yet.

Raw/derived evidence, identity manifests and per-state CSVs: docs/roadmap/0.1/0.1.7/evidence/stage-6-history-190-opt-20260919T232858Z/. Independent reviewer has reproduced both performance comparisons; final custody review follows.

[yifanxuaaa]

Step 5 — deeper subtree/streaming changes considered and deferred

The measured batching/bounded-reuse change is sufficient to deliver the requested targeted optimization; no additional product mechanism is added in this campaign.

Subtree summaries: the candidate inode spans are4,482,754,167ns (stride10) and15,739,979,668ns (stride3), but include lazy reference processing as well as sorted inode work. They do not isolate avoidable sibling reads. Both versions already acquire siblings before pruning; current parents lack all authenticated child facts needed for fill/count/rebalancing. The evidence does not yet justify a format/trust-contract change. Keep existing validation.

Save streaming: candidate accept-loop time remains9,448,569,166ns /19,247,494,540ns, but those intervals include actual Store admission, encoding/index work and transactions. The measured object-clone handoff is only49,171,738ns /87,294,823ns. That does not justify assuming streaming removes the multi-second save cost. No added workers, enlarged buffers, reordered persistence or shifted timer boundary.

Remaining filesystem residual is explicitly retained:5,210,486,416ns /15,511,671,619ns. Its read/zero-count/release components need finer evidence before another optimization claim. We do not assign it to a full scan by guesswork.

Independent reviewer reproduced both raw comparisons, all70 roots, exact56 quota parity, identical Store hashes across each pair, source/binary/lock custody and production LOC85,533→85,582(+49). Issue stays open for its remaining qualification gaps (stride3 verification TARGET_MISS, O3 pins missing, uncontrolled cache/no admission). Steps1–4 deliver measured diagnostics and validated code; step5 is the recorded decision to defer unsupported deeper changes. No commit/push or release claim.

[yifanxuaaa]

Publication requested; next optimization squads launched

The owner requested committing/merging the completed parent-lookup optimization and continuing with subagents. The completed change retains its measured improvements (stride10 operation47.161768126→27.386866665s; stride3 125.277281254→79.607320336s), identical70state roots/paired Store bytes and production LOC85,533→85,582(+49). It does not meet the historical time tripwire or stride3 verification target; O3 pins/cache admission and existing harness lint/format failures remain open.

The publication branch is isolated from unrelated local architecture work. Exact first-parent/staged-tree LOC and source identities are being checked before PR creation/merge. Large immutable Stores/executables remain local per the repository archive policy; committed receipt manifests retain original paths, bytes and hashes.

Three subagents have begun the next-target investigation:

1. Provider reads: confirmed fresh PoolReader per inode leaf and two record() extractions per pooled chain element (dependency discovery and reverse reconstruction). Compressed physical groups can be decompressed twice. Existing ordinary group_decodes/pack/edge counters omit this pooled branch, so they cannot price it. First target: measure pooled work explicitly and test bounded decoded-group reuse without changing chain-work refusal semantics. No speedup claimed.
2. Filesystem residual: three coarse scopes can isolate metadata overlay, zero-count lookup and release within the existing1024-node bound. Lazy reference processing remains inside inodes. Filtering ignored base lookups for newly allocated rows is a candidate only after error/corruption parity proof.
3. Store admission: fallback-index selection can compute the same content signature again during FULL admission. A lazy per-selection signature reuse is the smallest secondary candidate; full-pack write coalescing is riskier because it affects visibility, memory and transaction cadence.

Next research is separate from the change being merged. Reports are under docs/roadmap/0.1/0.1.7/evidence/stage-6-history-190-next-20260920T003441Z/; no next-phase product edit or measurement has occurred yet.

[yifanxuaaa]

Merged the completed parent-lookup optimization into main via PR #194, commit 9d82685f3. The issue body now links the committed report, independent review, exact LOC and local-artifact custody manifests.

• Stride10 operation: 47,161,768,126 → 27,386,866,665 ns (41.93% reduction).
• Stride3 operation: 125,277,281,254 → 79,607,320,336 ns (36.46% reduction).
• 70 state roots and paired Store bytes unchanged; 56 quota outcomes identical. These remain single-sample diagnostics with uncontrolled cache state, not release admission.
• Production LOC: 85533 -> 85582 (delta +49); reference 65417 unchanged, core 20116 -> 20165. Verified the merged tree and first parent exactly match the counted snapshots.

Not achieved: historical operation-time parity; stride3 verification target (29.978286084 s versus 20 s); history O3 pins; eligible cold-cache admission. Existing harness Clippy/format failures and baseline stride10 allocation miss remain recorded. Issue stays open.

Three next-target subagents completed source analysis: pooled inode-leaf decoding, filesystem residual/reference reads, and save signature/pack work. Priority is instrumenting the pooled read branch whose decompressions are missing from current ordinary counters, then testing bounded decoded-group reuse with unchanged error/resource refusal semantics. Their reports identify duplicate chain-record extraction, ignored new-row base demands and duplicate fallback/FULL signature scans. None is presented as a measured next-phase saving. No subtree-summary or pack-streaming policy change is justified yet.

[yifanxuaaa]

Proceeding with the next #190 optimization from merged PR #194: pooled physical-group decoding. Three subagents own production telemetry/bounded reuse, external regression tests, and independent correctness review. No secondary optimization is included.

First freeze instrumentation in both arms and count actual pooled decompressions/decoded bytes, record extractions, value materializations and pack BLOB acquisition. Then test minimal bounded reuse while preserving validation, visibility, memory limits and decoded-work refusals. One sample per arm on stride10, followed by stride3 confirmation if correctness and reduced work hold. Same120/240s diagnostic limits,60s verification hard limit,10/20s verification targets, one construction worker, fresh outputs and serialized locks. Cache remains uncontrolled and admission-INELIGIBLE. No speedup is claimed before measurement.

Baseline production LOC85582 (reference65417, core20165). Instrumentation and treatment deltas will be reported separately. Subtree summaries, signature reuse and pack-write coalescing remain outside this experiment.