Skip to content

fix(metadata): fence checkpoint restore references#406

Merged
wchwawa merged 2 commits into
NoKV-Lab:mainfrom
wchwawa:agent/object-reference-gc-fence
Jul 14, 2026
Merged

fix(metadata): fence checkpoint restore references#406
wchwawa merged 2 commits into
NoKV-Lab:mainfrom
wchwawa:agent/object-reference-gc-fence

Conversation

@wchwawa

@wchwawa wchwawa commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

This is PR1 of the LingTai workbench_restore rollout. It hardens NoKV's checkpoint, recovery-log, object-reference, and fork-retention foundations; it intentionally does not expose the final workbench_restore MCP tool yet.

  • bind prepared uploads and object-GC work to an exact GC epoch/claim identity, restaging stale uploads without starving later reclaimable rows
  • make checkpoint images and recovery log segments immutable, digest-bound, prefix-bound, and exactly matched by the control-plane recovery identity
  • serialize controlled metadata writes through synchronous log publication before ACK, and keep reads/readiness fail-closed while local state is not exactly represented in control
  • allow Fresh resurrection only for a crashed Recovering generation that never served and published no recovery identity; require explicit failover after any generation reached Serving
  • retain snapshot-backed source objects for zero-copy forks until the borrowing fork no longer references them
  • validate startup graft reconciliation against the complete control topology and accept an existing target only when its inode identity is exact
  • preserve staged client data on uncertain ownership/publication errors and refresh/restage against a newer object-GC epoch

FUSE scope

No FUSE workflow or live gate is added. nokv-fuse changes by seven added lines only: the legacy publish journal supplies GC epoch 0, which the shared validation rejects fail-closed, and three test constructors provide the new required field.

Compatibility and known limitations

  • reader-based artifact upload now requires Read + Seek, because a stale GC epoch must rewind and fully restage; non-seekable external callers need an adapter
  • the prepared-artifact wire schema has a new required GC-claim field, so mixed old-client/new-server deployment is not supported by this PR
  • custom out-of-tree MetadataStore implementations must implement committed-request readback; the unsafe default was removed
  • controlled restore rejects legacy checkpoint identities without the new immutable proof, and legacy FUSE journal entries without a durable GC epoch fail closed instead of being guessed safe
  • rollback now rejects hardlinked snapshot/current subtrees whose identity cannot be preserved safely
  • once failover durability is enabled by backup/shared-log, object deletion remains conservatively blocked by the persisted durability marker; this prevents data loss but can accumulate objects until a future durability watermark provides a safe release condition

These are deliberate safety/compatibility tradeoffs, not claims of behavior-preserving deployment across mixed versions.

Validation

Static and package gates:

  • cargo fmt --all -- --check
  • git diff --check
  • all repository shell scripts pass bash -n
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace
  • cargo test --workspace --all-features --exclude nokv-python
  • NoKV Python checkpoint/fsspec/torch tests: 13 passed
  • LingTai MCP contract and skill tests: 30 passed
  • closed-feat(workbench): restore checkpoints to COW forks #403 live harness unit tests: 12 passed

The macOS PyO3 extension-module link mode cannot produce an all-features Rust test binary because CPython symbols are intentionally unresolved in that mode. It is covered here by all-features clippy, the normal workspace test, the all-features Rust matrix excluding only nokv-python, and the standalone Python tests.

Live gates, all without FUSE:

  • external RustFS object-reference/GC fence: 2 passed
  • real etcd Fresh-resurrection control test: passed
  • process-level controlled resurrection: blocked checkpoint PUT -> epoch-1 Recovering crash -> TTL expiry -> epoch-2 Fresh Serving; a later Fresh owner is rejected after the shard has served
  • metadata HA checkpoint + shared-log failover: passed, fsck clean
  • paused stale-owner chaos: epoch-1 owner resumed after epoch-2 takeover and rejected the stale write
  • two-shard fleet failover and client re-resolution: passed, both shards fsck clean
  • 1 GiB zero-copy fork: clone added zero objects; source deletion + GC retained all 256 blocks; fork SHA-256 matched; retirement was blocked while borrowed; retirement then reclaimed all 256 source blocks; final fsck clean
  • disaster recovery: local metadata deleted, restored from the RustFS archive, body byte-identical, new writes accepted, fsck clean
  • closed-feat(workbench): restore checkpoints to COW forks #403 full LingTai/NoKV reference harness with --allow-missing-restore: overall passed; 8 A+B scenarios passed and the 3 scenarios requiring the not-yet-present workbench_restore tool were explicitly skipped

The PR that introduces workbench_restore must rerun that reference harness with --require-all and zero skips.

Validation base:

  • NoKV: 1dd1184470d6acc5269e698f8cf8de9a990eb3ba
  • LingTai: e54c27688140da6d11dec887647dc972aadaab64

Signed-off-by: wchwawa <wch19961116@gmail.com>
@feichai0017 feichai0017 moved this to In Progress in NoKV Delivery Jul 13, 2026
Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa marked this pull request as ready for review July 14, 2026 04:55
@wchwawa
wchwawa merged commit 0826826 into NoKV-Lab:main Jul 14, 2026
4 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in NoKV Delivery Jul 14, 2026
@feichai0017 feichai0017 moved this from Done to In Progress in NoKV Delivery Jul 14, 2026
@feichai0017 feichai0017 moved this from In Progress to Done in NoKV Delivery Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants