v0.1.7 pair 2: implement portable history, staging and conditional publication

[yifanxuaaa]

Status: Proposal; target LayerFS v0.1.7; not a released contract.

Merged implementation; qualification remains open — 2026-09-21

Pair 2 is merged into main at 152b9c3a2. GitHub reports #211 (design), #212 (implementation) and #213 (review remediation and bridge consolidation) as MERGED. Original source branches and worktrees are retained. The newer main storage changes and #214/#215 additions are preserved.

M0–M3, the remediation packages P0–P7 and M5 are complete at the implemented scope. M4 remains incomplete, so #210 remains OPEN under the owner's instruction to close it only when all work is finished:

• H04: real overlapping C2-save lifetimes with reverse completion/failure and retained stage are unverified.
• H06: independent-stack upload overlap is NOT_RUN.
• H08: real failures at all four boundaries and identical-root failed/unknown versus successful writer schedules are NOT_RUN.
• H14: unchanged full history-consumer substitution across distinct compatible C1-only/C2-only/combined revisions remains UNVERIFIED. The earlier C1/C2-only component harness is narrower proof.

Final integration verification used the existing core/.cargo/config.toml profile, with commands run from core/, CARGO_TARGET_DIR=/Users/yifanxu/.codex/worktrees/b995/layerfs/target/remediation-core, LAYERFS_CONSTRUCTION_WORKERS=1, toolchain +1.85.1 and offline/locked dependencies:

• cargo +1.85.1 test --manifest-path Cargo.toml --offline --locked --workspace: 605 passed, no failures, 3 declared ignored entries. The test tree was e21549050; subsequent main integration changed only documentation and a measurement example, which the final examples build/Clippy covers.
• cargo +1.85.1 build --manifest-path Cargo.toml --offline --locked --workspace --examples --bins: PASS.
• cargo +1.85.1 fmt --manifest-path Cargo.toml --all -- --check: PASS.
• cargo +1.85.1 clippy --manifest-path Cargo.toml --offline --locked --workspace --all-targets -- -D warnings: PASS.
• cargo +1.85.1 build --manifest-path Cargo.toml --offline --locked -p layerfs-history --no-default-features: PASS.
• python3 tools/check_product_boundary.py: PASS, 194 production files; python3 -m unittest discover -s tools -p 'test_*.py': 6 passed.
• Fresh production host route: 14 PASS. Fresh Linux Docker daemon → rebuilt macOS service route: 14 PASS, cleanup PASS. The Linux image layerfs-history-bridge:7a6db4c2 (sha256:53a9b1bfd21b68cffbebb8bb4ab7a3dcf94a1910ed69c47316ab81d9e53bac58) was reused after exact daemon/bridge/telemetry source, manifest/lock and core-profile equality checks; the service includes main's newer storage source.

Initial attempts are retained, not promoted to passing evidence: running Cargo from the repository root skipped the core profile and produced a host/image cipher mismatch, so initial Linux routes failed before the first reply. Rebuilding from core/ made the unchanged standard route pass. The initial root-profile test run also hit an unchanged telemetry retention NotFound; the native-feature rerun and complete correct-profile run passed. A default-feature telemetry-only run did not exercise native tests and is not evidence for that case. No product, worker, timeout, cache or dependency change was made for these diagnostics; no benchmark or release claim.

First-parent production LOC for the merge commits:

Commit	Core	Reference	Combined
4ae8e668c design merge	25403 → 25403 (+0)	65417 → 65417 (+0)	90820 → 90820 (+0)
e21549050 implementation/remediation merge	25403 → 30860 (+5457)	65417 → 65417 (+0)	90820 → 96277 (+5457)
152b9c3a2 concurrent-main integration	30860 → 30860 (+0)	65417 → 65417 (+0)	96277 → 96277 (+0)

Method: tools/production_loc.py --root <exact git-archive snapshot> --detail --json, first parent versus final staged tree, with committed tree equality verified. First-party Rust/runtime SQL only; comments/blanks, inline legacy tests, external tests, examples, docs, tooling, manifests and artifacts excluded. Counter SHA-256 c6e853c2280e2ee96caa6cafff4b221fa9201e1f5bf40d12e8251f70387210fc. No relocation/retirement. The combined total includes 9 production lines already landed independently on main; Pair 2's net addition is 5457 and final bridge production LOC is 3658.

The existing repaired-boundary handoff retains the earlier source-pinned evidence and qualification details. Integration raw logs and exact source/count manifests are retained in the task's target/pair2-main-merge/ directory.

Historical review and remediation checklist — 2026-09-21

Original review-time disposition: not ready; remediation required. The current merged disposition and remaining qualifications are recorded above. Independent review pinned PR #212 at 92e56635ae4559d175fe3cd455f36f9fe6b5b498 with a clean tree before and after the review. The governing implementation specification and pre-publication audit remain pinned at c85cf6b69b3809d860caaad764a09e86a54ece9a.

• Remediation specification — committed as 35740836f84b9ef687da2f70d785c9e6b2ed2fd2 on codex/pair2-history-remediation-plan.
• Independent counterexamples and raw verification evidence — preserves the failed initial reviewer harness attempt, subsequent reproductions, SQL checks, host-route receipt and LOC comparisons.

The existing workspace checks passed at the reviewed head (580 tests; 97 test executables plus 6 doc-test invocations), as did the 13-case host daemon/service route. Independent probes nevertheless reproduced contract violations. Earlier milestone/coverage claims in PR #212 or its handoff must not be used as evidence that these defects are fixed.

Remediation checklist

• P0: freeze repaired profile-2 typed failure/context, root-descriptor and cursor capability/format boundaries while preserving legacy profile-1 bytes.
• P1 / R01, R07: preserve expected/actual and known exact-stage context; handle unknown C5 outcomes without guessed rollback or reusing an uncertain writer.
• P2 / R03, R12–R14: repair legal minimum record widths, enforce 16 KiB reply budgets, correctly classify content-writing history commands, and restore legacy mutation ResultData rejection.
• P3 / R02, R04, R05: prevent rehashed sibling-ancestry cursor forgery, support all legal names within the cursor cap, and resume anchored pages after the actual subject advances.
• P4 / R06, R08: initialize every manifest directory, accept valid interleaved parent declarations, and validate/read the complete filesystem root descriptor including root serial.
• P5 / R09, R10: validate frozen schema/binding on open and enforce stale-base publication precedence even with a refreshed expected stack head.
• P6 / R11: make catalog reservation success and checked terminal endpoints agree before allocation exposure.
• P7: add regressions and source-accurate documentation, preserve historical receipts, run fresh current-head checks/routes, report exact per-commit production LOC, and obtain independent review of the repaired head. Record R15 as a pre-existing baseline observation rather than a new reachable-panic defect.

Qualification remains separate

H04 real overlapping-save/reverse-failure observation, H06 independent-stack upload overlap, H08 actual boundary/identical-root failure schedules, the Linux Docker route, and H14 component substitution are not established by the host route or code inspection. H01/H10/H13 PASS labels are contradicted by the review's manifest/allocation/page counterexamples and require fresh evidence. During review, Linux targets were installed for toolchain 1.85.1; the former missing-target explanation must not be repeated as a current observation. Record any genuinely unavailable schedule/capability as NOT_RUN/UNVERIFIED with its exact reason; no fabricated product hooks, sleeps or implied PASS.

A separate Codex task is being launched to execute this specification in an isolated checkout. Keep the original reviewed worktree untouched; use a separate remediation branch and a reviewable follow-up PR. No merge/closure is authorized here. No benchmark/performance campaign, worker/timeout/cache retuning, dependency patch/vendor/fork, CI or retired preflight. Preserve W=2, Q=0, exact-finish staging, metadata-only save exclusion and the selected continuity envelope.

This planning/evidence commit changes no production source: Production LOC: 95665 -> 95665 (delta 0); core 30248 unchanged, reference 65417 unchanged. It does not claim implementation of any remediation item or close #179/#180/#192/#193/#209.

Implementation sub-issue of #180. Pair 3 and its schema-7 multi-writer implementation have landed. Implement pair 2 independently through the existing service; a non-FUSE client supplies the first consumer, and later pair 1 (#179) consumes this contract.

Specification: Implementation specification
Pre-publication audit: Three-subagent audit and dispositions

Outcome

Deliver layerfs-history, a separate versioned catalog, real service initialization/staging and history operations through the existing bridge. Preserve C1/C2 canonical content, physical encoding and public save contracts. No FUSE/Workspace implementation is required for this issue.

Inspected baseline: a02168adbb1b02571941654919cefca12dbc1f42, including multi-writer landing eb319aaa9. C2 stays schema 7 with six tables and two private saves per Store; service admission stays W=2, Q=0, one construction producer per ordinary operation; namespace init retains its existing exception.

Operations

• init_layerstack: Empty or bounded logical namespace manifest -> genesis Layer/stack and root descriptor, through production C1/C2 code.
• fork: from a Layer or an authorized existing Commit -> Branch sharing the selected root/ancestry.
• read_branch: coherent stack/base/head/root/profile/scope/root-serial descriptor.
• stage_changes: validated existing-inode prepared changes -> saved candidate and exact stage token.
• commit_staged: atomic immutable Commit + conditional Branch advancement + exact stage removal.
• commit: convenience composition of stage_changes and commit_staged, one service admission, separate persistence boundaries.
• add_layer: reuse Commit root, insert Layer and conditionally advance LayerStack; Branch base/head unchanged.
• discard_stage: delete only the exact matching stage; no content deletion or serial recycling.
• reserve_inodes: checked scope-wide range under the continuing-authority contract.
• Typed get/list/Commit-history/Layer-history/stage inspection with bounded pages.

Tables

Create seven tables in C5 catalog schema 1, separate from C2:

Table	Contents / main invariant
history_meta	Catalog/incarnation, stable Store/authority binding, identity-format version, monotonic stage-token allocator
layer_stacks	Name, scope/profile, conditional head Layer
layers	Immutable stack/parent/root/source; unique genesis, child and published source
branches	Stack/name/base Layer/optional head Commit; coherent conditional head/base
commits	Immutable root/parent Commit/base Layer; ancestry shared by forks
workspace_stages	Workspace incarnation/token, candidate root, frozen expected head/base/root and construction context
scope_allocator	One checked high-water authority per shared inode scope; no exposed-range recycling

C2's saves is physical ownership, not a history stage or Commit. No foreign key or transaction crosses databases; service validates root provenance and binding. A Workspace ID is opaque: no runtime workspaces table or live Workspace dependency.

Relationships

LayerStack S.head -> L0(root R0)
                     ^
                     | base
Branch A.head -> K2(root R2, base L0) -> parent K1 -> None

Stage W/token17:
  branch=A, expected=(K2,L0,R2), candidate=R3

commit_staged(17):
  insert K3(root R3,parent K2,base L0)
  CAS A.head K2 -> K3; delete exact stage17

add_layer(A,K3,expected=L0):
  insert L1(root R3,parent L0,source A/K3)
  CAS S.head L0 -> L1
  A.head=K3 and A.base=L0 remain unchanged

Fork-from-Commit uses that selected Commit's base, not the source Branch's current base. Layer publication reuses the same root without content copying/re-encoding. Same-Branch concurrent changes get one state-changing winner and a retained losing stage, whether file edits overlap or not. No automatic rebase or conflict resolution.

File and LOC plan

core/crates/layerfs-history/
  src/{lib,identity,records,catalog,error}.rs
  src/sqlite/{mod,open,rows,branch,staging,commit,layerstack,allocation,query}.rs
  sql/schema-v1.sql
  tests/{lifecycle,conditional_updates,allocation,reopen,history_pages}.rs

core/crates/layerfs-service/
  src/operation/{history,history_bootstrap}.rs
  extend owner, dispatch/failure, native configuration/startup
  tests/history.rs

core/crates/layerfs-bridge/
  src/contract/history.rs
  extend request/outcome, metadata/response/state codecs and client matching
  tests/history_protocol.rs

core/crates/layerfs-daemon/tests/history_route.py

Production scope	Central planning LOC
History Rust + runtime SQL	2,505
Service bootstrap/history/validation/wiring	900
Bridge contract/codecs/classification	650
Total	4,055; expected 3,500–5,000

These are estimates, not measured counts or caps. External tests/helpers/drivers are another 1,800–3,000 LOC, excluded from production. The revised estimate includes bootstrap, role validation and permission/protocol work exposed by audit. Each commit still requires exact first-parent/staged/committed production LOC, reference/core subtotals and counting method. Product files <=999 physical lines; lib.rs/mod.rs <=200 and declarations/delegation only; external tests.

Integration decisions and explicit limits

• Keep C1/C2 in the service. Stage only after that exact validated operation's known C2 finish. Root presence, an authenticated wrong-role object or another writer publishing identical bytes does not prove success.
• Initial staging requires construction base = captured Branch root and intended Commit base = captured Branch base. If the head races during saving, retain the candidate stage; commit_staged enforces the authoritative CAS.
• C5 holds only short metadata transactions with immediate Busy refusal; no catalog lock across upload/construction/C2 finish, no nested cross-database locks.
• Add HistoryQuery/HistoryCommand as grouped opcodes 6/7 under operation profile 2. Preserve legacy profile1/opcodes1–5 and mask31; explicit Store-wide history grants, exhaustive mutation classification, separate metadata-only dispatch and checked response matching.
• Native initialization supports Empty or at most 128 logical manifest entries within 32 KiB metadata, assigning fresh scoped serials in the service. Attributes/symlinks are saved before filesystem construction if retained roots are required. Host scanning/arbitrary imported roots are not included.
• Initial stage_changes remains existing-inode only. Live new-inode/metadata extensions for complete writable Workspace are later pair-1 work; allocation alone does not enable create/mkdir.
• Fresh writable catalog creation occurs inside its owning service process. One continuing authority writes; read-only reopen works. Arbitrary writable process restart/restore/copy fails ContinuityUnavailable until a trustworthy continuity capability is implemented. No clean-flag/root-scan proof or added sync/WAL.
• Portable C5 contract; initial native C1/C2 composition is Linux/macOS. No native Windows/WASM/cloud qualification is implied.

Milestones / acceptance

• M0: freeze IDs/DDL, catalog application ID, wire tags/grants, bootstrap manifest, bounded cursor grammar and writable-authority lifecycle.
• M1: real catalog/IDs/schema validation, genesis/Branch/fork/allocation/queries and external public tests.
• M2: exact staging/discard, Commit/Branch CAS, Layer/stack CAS, no-change and prior-publication precedence tests.
• M3: production bootstrap and trusted save-to-stage composition; role validation, authorization, profile/codecs and non-FUSE client lifecycle.
• M4: actual service/daemon route plus H01–H14 from the spec, with W2 isolation, stale-head retention, failure/unknown outcomes, bounded resources and continuity refusals.
• M5: independent implementation audit, exact checks/LOC, source-backed architecture update and pair-1 handoff.

H04's reverse-completion proof must observe overlapping real C2 save lifetimes. Delayed metadata/END_INPUT or submission order alone is not that proof. If unavailable, retain the exact gap as NOT_RUN alongside separate C2 W2 and actual service/history evidence; no product test hooks/delays or implied PASS.

Run explicit core-manifest locked workspace tests, examples, fmt, warning-denying Clippy, boundary guard and its self-tests. Add no-native feature and actual daemon-route checks. No CI/aggregate preflight; no worker, cache, timeout or workload change to pass a gate. Runtime/benchmark acceptance is NOT_RUN at issue creation.

Out of scope: full FUSE/Workspace, diff/conflict resolution (#164), GC, force-reset, automatic replay/rebase, arbitrary host/root import, transparent migration, cross-store crash atomicity and stronger durability. This issue does not close #180, #179, #192, #193 or #209 by implication.

Pre-publication review

Three subagents audited reference history semantics, landed C1/C2 multiwriter integration, and service/bridge portability. The spec incorporates their corrections for construction-base binding, typed-root provenance, production bootstrap, opcode/grant capacity, metadata-only dispatch, continuity, anchored cursors and proof quality. The linked audit records findings/dispositions. Review is not implementation verification.

[yifanxuaaa]

Implementation status — pair 2 (history) landed on a branch, PR open

Implementation PR: #212.
Branch codex/pair2-history-implementation, tip 4cbb8aecb, based on a02168adb (the audited
baseline). Reviewed specification and audit read at c85cf6b69b3809d860caaad764a09e86a54ece9a
(PR #211, not merged).

Committed record with every identity, command and result:
docs/roadmap/0.1/0.1.7/evidence/issue210-pair2-history-20260920T210753Z/HANDOFF.md.
Independent-review prompt: REVIEW-PROMPT.md
in the same folder.

This comment closes nothing. #179, #180, #192, #193 and #209 remain open and are not implied by
this work.

What landed

One new crate, core/crates/layerfs-history, plus its service capability. Operations:
init_layerstack, fork, read_branch, stage_changes, commit_staged, commit, add_layer,
discard_stage, reserve_inodes, and typed get/list, Commit-history, Layer-history and stage
inspection. Catalog schema 1 is a separate file (application id 1279677256, seven tables); C2 stays
schema 7 and is never touched from C5. Service and bridge use opcodes 6/7 on operation profile 2 with
an exhaustive checked permission mapping and exhaustive semantic dispatch; the daemon's generic relay
is reused unchanged. Initialization is real production code, not the prepare_store fixture.

Milestones

M0 contract freeze, M1 catalog, M2 transitions, M3 service + bridge: DONE. M4: PARTIAL —
the host production-transport route passes 13 cases, the Linux Docker daemon/service route is
NOT_RUN. M5: this handoff.

Acceptance cases

H01, H02, H03, H05, H06 (partial), H07, H09, H10, H11, H12, H13 covered by external tests over
production paths: 16 service-level cases through Service::handle, 18 catalog-level cases through
the public HistoryCatalog API, 11 codec-level cases, 13 driver cases.

H04 NOT_RUN. No production observation establishes overlapping C2 save lifetimes or the
reverse-completion schedule, and no fault hook, sleep or alternate algorithm was added to fabricate
one. The existing C2 W=2 receipt is not counted as service/history evidence.
H08 PARTIAL — known/unknown outcome classes and retained stages are asserted; an actual
identical-root publication by a second writer is not exercised.
H14 NOT_RUN — no component-substitution harness was run here.

Checks (at the tip, clean tree)

cargo +1.85.1 test --offline --workspace 103 binaries / 0 failures; build --workspace --examples;
fmt --check; clippy --all-targets -D warnings 0 errors; core/tools/check_product_boundary.py
PASS (193 files); python3 -m unittest discover -s core/tools OK; layerfs-history
--no-default-features build PASS; layerfs-daemon/tests/history_route.py PASS (13 cases, host
client). No CI, no aggregate preflight, no benchmark and no performance claim.

Defects the new tests found and fixed

1. Lineage continuation was off by one — a continued page repeated a record.
2. The manifest count pre-check used a typical entry width instead of the smallest legal one.
3. The catalog's worst-case LayerStack record width was 116 bytes where the codec writes 179, so it
   would have packed a page the codec then refused (128 records instead of 90).
4. ReserveInodes with a zero count was Capacity in the bridge and InvalidInput in C5.

Production LOC (first parent → committed tree, tools/production_loc.py)

90811 → 95665, delta +4854 across five commits (018d9c366 +4850, 572d61f03 +1,
daccb403a 0, 6e310b249 +3, 4cbb8aecb 0). Core 25394 → 30248; reference 65417 → 65417.
Component variance against the specification's plan is recorded in the handoff: history+SQL 2523
(plan 2505), service +906 (plan 900), bridge +1425 (plan 650).

Not established by this work

Independent subagent review — the subagent channel was unreachable in this environment (codex exec
fails on a model/CLI version mismatch for every model; the claude CLI has no valid credential), so
the audit is self-review plus external tests. The Linux Docker route is unrun. Limitations are listed
in core/docs/architecture/16-history.md §16.9 and in the PR.

[yifanxuaaa]

Implemented and pushed the remediation in draft follow-up PR #213, targeting codex/pair2-history-implementation. Final branch head: a4a144af8af9b46c3ce3100047466f4817332808; repaired product: f2021367e11f6433d1d9e055198cac5ad280f6c9; additional disjoint-edit test head: 8a3fc0a774d07bf03e83b2d10f6a6c213030a556.

R01–R14 now have source-backed fixes and corrected external regressions. R15 remains an unchanged pre-existing fixed-width expect observation. The independent read-only reviewer found two additional issues (continuation traversal ceiling and SQL schema-prefix wildcard); both were corrected and re-reviewed, with no remaining actionable source findings. The committed product diff matches the review hash.

Fresh verification: 602 core tests passed at 8a3fc0a; examples, bins, fmt, Clippy all targets with -D warnings, no-native history build, boundary guard (194 files), and six guard self-tests all passed. Production host route: 14 cases PASS. Source-matched ARM64 Linux Docker daemon → macOS host service route: 14 cases PASS. Existing unchanged C1/C2 substitution consumer: four arms and four persisted-data cross checks PASS with matching canonical identities. All raw failures and qualification receipts are preserved in the new handoff.

Code remediation is ready for review; full qualification remains incomplete.

• H04 real history/C2-save overlap, B completion then A failure, and B-stage survival: UNVERIFIED.
• H06 independent-stack upload overlap: NOT_RUN; catalog independence and conditional publication are covered.
• H08 actual I/O/connection-loss at the four persistence boundaries and identical-root A-failed/unknown versus B-success schedules: NOT_RUN. External SQL/provider refusal tests cover narrower behavior and are not substitutes.
• H14 full unchanged history-consumer substitution across distinct supported component revisions: UNVERIFIED. The existing C1/C2 consumer passes, but the selected C1/C2 source is identical and that consumer does not exercise history.

History-only compatibility details are explicit: keyed cursor v2 requires an externally retained authority capability, and typed failure/root descriptor DTOs changed. Legacy profile-1/opcodes-1–5 bytes, C5's seven tables, C2 schema 7, W=2/two saves/Q=0 and exact successful finish before stage are preserved. No benchmarks, worker/cache/timeout retuning, third-party modifications, CI/preflight, retry, guessed rollback/discard or durability upgrade.

Production LOC was independently counted from exact first-parent/staged/committed snapshots: core 30248 → 30927 (+679); reference 65417 → 65417 (+0); combined 95665 → 96344 (+679). The test and evidence follow-ups each have delta 0, with per-commit counts/methods in their messages and handoff.

The reviewed implementation branch and original worktree remain untouched. #212 and #213 are not merged; this issue and the related parent/component issues remain open. No release, full M4/H01–H14 acceptance, writable restart or cross-store crash-atomicity claim is made.

[yifanxuaaa]

Bridge consolidation is pushed to draft PR #213 at 72d95323205e1352f1310a81a9e287414c7c9957. Product/test commit: 7a6db4c2d7f4b9182ea7b0c923eabcafe96dbb0c; the final commit adds evidence and source pinning only.

The refactor removes 76 production LOC by sharing optional-field codecs and the five history page envelopes, constructing decoded records directly, and deleting redundant Fork matching. No DTO, dependency, wire tag/version, validation, bound or C1/C2/C5 behavior change. Bridge 3734 → 3658; core 30927 → 30851; reference 65417 unchanged; combined 96344 → 96268. The evidence follow-up has delta 0. Exact first-parent/staged/committed counting is recorded in both commit messages.

Fresh results: 603 core tests PASS, including 46 pre-refactor golden wire encodings; examples/bins/fmt/Clippy -D warnings/no-native build PASS; boundary guard 194 files and six self-tests PASS. Fresh production host and matched ARM64 Linux Docker daemon routes each pass 14 cases. Independent read-only review found no actionable issue; its product diff hash matches the committed refactor.

Consolidation handoff and raw evidence. The original failed test-formatting lint is preserved alongside its corrected run. Previous remediation receipts remain untouched.

H04/H06/H08/H14's previously documented qualification gaps remain open. The C1/C2-only substitution harness was not rerun for this bridge-only refactor; no full acceptance, performance, release, merge or issue closure is claimed.