diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c85a597..f957a23 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -11,14 +11,20 @@ Disk ──► load(path) ──► replay_log() ──► HashMap │ watcher.watch_all_from(cursor, tx) │ - CursorExpired? - / \ + CursorExpired? (at resume, or mid-watch + / \ from the floor guard) Yes No │ │ - stale-key resync (synthetic delta stream - deletes) + watch_all(tx) │ - (state-sync re-list) │ - └───────┬─────────────┘ + bucket evicts current values? delta stream + / \ │ + No Yes │ + │ │ │ + key-listing diff (synthetic artifact restore: │ + deletes) + watch_all(tx) fold := artifact, │ + (state-sync re-list) resume from its │ + │ cursor │ + └──────────┬─────────┘ │ + └──────────┬─────────────┘ │ KvUpdate → cache.apply() + snap.write_update() │ @@ -123,14 +129,17 @@ Every non-`_from` watch is a **state-sync** stream (NATS `DeliverPolicy::LastPer | `AppendLogSnapshot` | Default `SnapshotStore`: append-only log + in-RAM fold (pure-Rust) | Not for folds larger than RAM | | `FjallSnapshot` | On-disk `SnapshotStore` (fjall LSM, `feature = "fjall"`) for large folds | Not in the pure-Rust core; opt-in feature | | `RocksDbSnapshot` | On-disk `SnapshotStore` (RocksDB, `feature = "rocksdb"`) for large folds | Not pure-Rust; opt-in feature with a C++ build dep | -| `watch_applied` | Combinator: batch → apply → *then* advance cursor / fold into `SnapshotStore`; resyncs stale keys on cursor expiry | Not a raw watch; the cursor follows `apply`, not receipt | +| `watch_applied` | Combinator: batch → apply → *then* advance cursor / fold into `SnapshotStore`; repairs the fold when its cursor expires | Not a raw watch; the cursor follows `apply`, not receipt | +| `ExpiryRepair` | How `watch_applied` repairs an expired cursor: `Relist` (key-listing diff), `Restore { reader, restore }` (artifact; the reader's live listing keeps it from deleting live keys), `Auto` (by bucket retention), `None` | Not a retry policy; it decides what the fold may trust | +| `Retention` | What a watcher's log can evict: `evicts_current_values` + `first_revision`, read live from the stream (`KvWatcher::retention`) | Not config; read at the moment of expiry | +| `RestoreSource` / `ArtifactRestore` | Supplies the newest published artifact, opened as a temporary fold of the consumer's backend | Not a bootstrap tool for new nodes' paths; feeds `ExpiryRepair::Restore` | | `WatchScope` | What `watch_applied` watches: `All`, `Prefix`, or `Prefixes` (multi-filter union) | Not N consumers; `Prefixes` costs one consumer | | `ConnectionCapabilities` | Feature flags for runtime branching (CAS, streaming watch, …) | Not enforced; purely advisory | | `ExportRequest` | One-shot channel message that asks `watch_applied` to flush + export the current fold | Not a store operation; handled between flushes | | `ExportLease` | Fleet-wide at-most-one coordinator: CAS key that prevents N nodes exporting the same round | Not a correctness gate; only dedup. Pointer monotonicity is the gate | | `LeaseGuard` | RAII guard for a won export round: `complete(cursor)` stamps success, `abandon()` frees it early via CAS delete | Not held across restarts; process crash → expiry → next node steals | | `LeaseRecord` | Value stored in the lease key: holder, acquired/expires timestamps (wall-clock, embedded — no server TTL), optional completed cursor | Expiry is compared by callers, not the store; requires NTP-sane clocks | -| `ExportManifest` | Artifact metadata committed to disk and object store: backend identity, format generation, cursor, per-file BLAKE3 hashes | The manifest bytes are their own content address (blake3[..8]) | +| `ExportManifest` | Artifact metadata committed to disk and object store: backend identity, format generation, cursor, per-file BLAKE3 hashes, and (schema 2) the exporting watch's key scope | The manifest bytes are their own content address (blake3[..8]) | | `ArtifactTransport` | Trait: `upload(key, dir) → ManifestAndOutcome`, `download(key) → ManifestAndDir`, `pointer(key)`, `prune(key, grace)` | Not an object store; adapts `ObjectStore` for the pointer-swap protocol | | `ObjectStoreTransport` | Concrete `ArtifactTransport` over any `object_store` backend (S3, GCS, Azure, local) | `file://` lacks CAS and FAILS CLOSED unless `with_non_atomic_pointer_fallback()` (dev only) | | `PublishOutcome` | `Published` (pointer advanced) or `SupersededByNewer` (refused — a newer pointer exists) | Not an error; a slow exporter's normal exit under concurrent rounds | @@ -173,7 +182,7 @@ Every non-`_from` watch is a **state-sync** stream (NATS `DeliverPolicy::LastPer The cursor is the NATS stream sequence number at the last checkpoint. On restart, pass it to `watch_all_from()` to subscribe at `cursor+1` — only the delta arrives, not the full history. -When the cursor expires (NATS retention window evicted those records), `CursorExpired` is returned. The fallback `watch_all()` re-list re-delivers the current value of every live key, but it cannot cover keys **deleted during the gap whose delete markers were also evicted** — those need synthetic `Delete` events diffed from prior state. `watch_applied` does this automatically when given a reader (see below). A raw-API caller hand-rolls the same diff with `Snapshot::stale_keys()`: +When the cursor expires (NATS retention window evicted those records), `CursorExpired` is returned. The fallback `watch_all()` re-list re-delivers the current value of every live key, but it cannot cover keys **deleted during the gap whose delete markers were also evicted** — those need synthetic `Delete` events diffed from prior state. On a bucket whose retention never evicts current values, `watch_applied` does this automatically (see below), and a raw-API caller hand-rolls the same diff with `Snapshot::stale_keys()`: ```rust match watcher.watch_all_from(&snap.cursor, tx).await { @@ -189,6 +198,8 @@ match watcher.watch_all_from(&snap.cursor, tx).await { } ``` +That diff is only sound where "not in the bucket" means "deleted". On a bucket with `max_age`, per-message TTLs, or `discard: old` under a limit, retention evicts **current** values too: a key can age out with nobody deleting it, and a write made during the gap can age out before the consumer returns. NATS cannot tell "deleted, marker evicted" from "aged out", so no listing diff can be made correct there — `watch_applied` restores from an artifact instead (see [Cursor-expiry repair](#cursor-expiry-repair)). + ### Applied-Cursor Watch (`watch_applied`) The resumable watch above hands the caller raw machinery — a channel of `KvUpdate`s, a cursor, a snapshot writer — and trusts each one to hand-roll the loop that batches updates, applies them, and advances the cursor. Every hand-rolled instance got the same step wrong: it advanced the cursor on **receipt** of an update (`high_water = rev` at `rx.recv()`) and applied the batch afterward. The combinator `watch_applied` exists to encode the correct discipline once. @@ -207,13 +218,29 @@ This is the lesson of Saltzer, Reed & Clark, *End-to-End Arguments in System Des The one exception: an update carrying the **unknown** version (an unparseable ACK subject on the hand-built multi-prefix consumer path) never touches `batch_high` — it can neither mint a cursor position nor clobber the real high from earlier in the batch. The update is still applied; skipping it under-advances at worst, and re-delivery on resume is idempotent. (The pre-guard code fabricated revision 0 for these, which the loop adopted as a real position — regressing the persisted cursor to 0 and forcing a full replay on the next restart.) +**Persist the store's cursor, not `on_applied`'s, when there is a store.** Across a transient store failure the batch is re-queued, but `apply` has run and `on_applied` fires with the advanced cursor — ahead of what the store made durable. Resume from the cursor the store reports (`load`/`open`), never from one persisted separately in `on_applied`: that one can skip the re-queued batch. Without a store, `on_applied`'s cursor is the one to persist. + **Snapshot consistency.** Raw `KvUpdate`s stream to the snapshot log as they arrive, but the *checkpoint* cursor is the post-apply cursor. A crash after a raw record is written but before its `apply`/checkpoint leaves the log holding data *ahead* of its cursor — which is safe: the cursor never names a revision whose `apply` had not returned, so resume re-delivers and re-applies that tail rather than skipping it. Compaction runs off the hot path via `spawn_blocking`, as everywhere else in the snapshot subsystem. **Flush triggers.** A batch flushes when any of these fires: the `window` elapses, `batch.len()` reaches `config.max`, a shutdown is signalled, or the channel closes with a pending batch (the remainder is flushed before returning). **Transient store failures re-queue, not drop.** If `store.apply()` returns an error, the raw batch is prepended to the next flush's accumulation and the watch continues. The streak counter increments; at 16 consecutive failures the watch fail-stops with `KvError::WatchError`. Dropping the failed batch and continuing was the shipped behavior until `transient_store_failure_never_leaves_a_cursor_gap` reproduced the bug: a transient failure followed by a successful flush advanced the cursor over a hole that survived every restart, because the restart re-folds from the advanced cursor, skipping the missing range. Cursor authority requires the store's cursor and contents to advance together, always (`applied.rs:303–394`, `tests/model_applied.rs`). -**Cursor-expired resync.** On `CursorExpired` from the resume path the combinator falls back to the full-scope watch (`watch_all` / `watch_prefix` / `watch_prefixes`), whose state-sync re-list re-delivers every live key as puts. The re-list cannot cover keys deleted during the gap whose markers were evicted with the cursor, so — when the combinator is given a `KvReader` and a store — it closes that hole first: the watch task lists the bucket's live keys, hands them to the main loop, and waits for an ack; the main loop flushes, diffs the fold's in-scope keys against the listing, and runs a synthetic `KvUpdate::Delete` (unknown version — never advances the cursor) through `parse`/`apply`/store for each key that vanished; only then does the fallback watch start. That ack ordering is the invariant: a synthetic delete always precedes the re-list put for the same key, so delete-then-recreate during the gap converges. Without a reader the fallback is re-list-only and logs the possible stale keys. +**Cursor-expiry repair.** The cursor expires at resume (`CursorExpired` from the `*_from` watch) or mid-watch (the live floor guard ends an All-scope watch with `CursorExpired` when retention overruns it). Both take one path. Correct means **every write minus every real delete** — not "whatever NATS still lists"; the two agree only when retention never evicts current values. The repair (`ExpiryRepair`, `src/repair.rs`) is chosen accordingly: + +- **Key-listing diff** (`Relist`; `Auto` on a bucket that keeps current values). The watch task lists the bucket's live keys, hands them to the main loop, and waits for an ack; the main loop diffs the fold's in-scope keys against the listing and runs a synthetic `KvUpdate::Delete` (unknown version — never advances the cursor) through `parse`/`apply`/store for each key that vanished; only then does the full-scope re-list (`watch_all` / `watch_prefix` / `watch_prefixes`) start. That ack ordering is the invariant: a synthetic delete always precedes the re-list put for the same key, so delete-then-recreate during the gap converges. **Refused** — the watch fails — when the watcher reports retention that evicts current values (`KvWatcher::retention`, read live from the stream config): there it deletes valid keys. `tests/eviction.rs` reproduced exactly that against a live server before the fix. +- **Artifact restore** (`Restore`; `Auto` on a bucket that evicts current values or can't say). The watch task peeks the newest artifact's manifest and checks it before downloading: it must cover the watch scope (recorded in the manifest by the exporting `watch_applied`), and pass `protocol::restore_allowed` — strictly ahead of the local fold, cursor still inside retention. It then downloads and imports it into scratch as a temporary fold of the consumer's backend, re-checks what actually arrived, lists the bucket's live keys, and hands both to the main loop. The main loop re-checks "ahead" against its own applied cursor (a mid-watch expiry delivered past the resume cursor), diffs the in-scope fold against the artifact (keys only), and folds the difference through `parse`/`apply`/store in `max`-sized chunks, then advances the cursor to the artifact's in one final commit and replies. The diff takes an artifact entry only when it is newer than the local one, and deletes a key the artifact lacks only when the bucket does not list it live. An artifact is complete *together with the log after its cursor* — a NATS cursor C means every retained message ≤ C is applied, not "the truth at C": a key whose latest write is after C may be missing or older in an artifact exported mid catch-up — and in both skipped cases the key's latest write is after C, so the resume delivers it. The consumer therefore never sees a key move backward or a live key deleted, not even transiently. The watch resumes from that cursor (which can itself expire later and go round again, each time only for a strictly newer artifact). A failed check fails the watch and logs the reason at `error`: a stale artifact has no safe recovery. +- **None** (`None`, or no store): the re-list alone, with a warning that deleted keys may persist. + +Whichever runs, the main loop first **drains** the watch channel: a mid-watch expiry leaves delivered updates buffered there, and the repair must fold them first — one applied after the repair would resurrect pre-repair state and could move the cursor backward. The watch task is parked on the ack/reply, so the drain takes exactly that backlog. + +A cursor-less start is repaired before it watches in two cases. A fold that holds **data but no cursor** (a torn first checkpoint, or a populated store started without one) runs the planned repair like an expiry at revision 0: a re-list alone never removes what it doesn't deliver. An **empty** fold with `Restore`/`Auto` on a bucket that has already evicted current values seeds from the artifact, since a re-list would be incomplete. + +The listing counts as the truth — "not listed" means "deleted" — when retention never evicts current values, or when it can but has never evicted anything yet (first retained revision ≤ 1). Otherwise the key-listing diff is refused. + +Before diffing, both repairs flush until the store holds every delivered update: a batch re-queued by a transient store failure is invisible to a diff of the store, and a re-queued put of a key deleted during the gap would otherwise commit after the diff and resurrect it. + +**The key-listing repair's one axiom.** Between taking the listing and the full re-list delivering what the bucket holds, retention must not evict a delete marker (axiom 6 of `tests/model.rs`, narrowed to this window). On a bucket that keeps current values, the only such eviction is an admin purge of markers, and NATS's `purge_deletes` keeps markers younger than 30 minutes by default. `tests/model_repair.rs` drops the axiom and the checker reaches the purge-inside-the-window trace; the artifact restore resumes on a floor-guarded watch and holds without it. This is the layer the tunnel router (swap route table) and edge origin watcher (rebuild hashrings) both collapse onto: `parse` extracts the domain registration, `apply` swaps the live state, `on_applied` persists the cursor. @@ -221,7 +248,7 @@ This is the layer the tunnel router (swap route table) and edge origin watcher ( NATS does **not** error when a live consumer's position is overrun by retention — it silently skips evicted messages (the same silent-clamp behaviour that `resume_window_ok` closes at resume time). For an All-scope watch, a skipped delete marker permanently diverges the fold, with zero log lines. The floor guard closes this mid-stream: -- **In-band (primary):** when a delivered revision jumps the frontier by more than 1, fetch `first_sequence` and call `resume_window_ok(frontier, first_seq)`. A benign interior gap — per-subject overwrites below the floor threshold — passes. Head eviction past the frontier fails the watch into the restart → `CursorExpired` → resync repair path. The check runs *before* the entry is processed: the fold never advances past unexamined evidence of loss. +- **In-band (primary):** when a delivered revision jumps the frontier by more than 1, fetch `first_sequence` and call `resume_window_ok(frontier, first_seq)`. A benign interior gap — per-subject overwrites below the floor threshold — passes. Head eviction past the frontier ends the watch with `CursorExpired`, which `watch_applied` routes in process into the same [cursor-expiry repair](#cursor-expiry-repair) a resume takes — the artifact restore on a bucket that evicts current values, never the key-listing diff there. The check runs *before* the entry is processed: the fold never advances past unexamined evidence of loss. - **Backstop (30 s):** when no deliveries arrive, the periodic probe catches the no-traffic case where there is no in-band evidence to act on. A periodic-only design was rejected by the model checker: deliveries catching the frontier up past a gap between probes erase the evidence, leaving permanent silent divergence with the guard running. The in-band check is what makes the design correct (`tests/model_live_watch.rs`). @@ -265,7 +292,7 @@ Every NATS operation is wrapped in `timed()` (30 s). Without it, a CLOSE_WAIT co ### Machine-Checked Protocol Kernels (`protocol.rs`) -Three pure-function guards live in `protocol.rs`. Production code and the Stateright exhaustive model checker (`tests/model.rs`) call the **same functions** — not two hand-synchronized copies. A change to any guard is re-verified against the full bounded state space on the next `cargo test --test model`. Mutation tests prove each guard is load-bearing: substituting a broken variant produces a counterexample. +Four pure-function guards live in `protocol.rs`. Production code and the Stateright exhaustive model checker (`tests/model.rs`) call the **same functions** — not two hand-synchronized copies. A change to any guard is re-verified against the full bounded state space on the next `cargo test --test model`. Mutation tests prove each guard is load-bearing: substituting a broken variant produces a counterexample. **`pointer_publish_allowed(current: &PointerState, candidate_rank: u64) → bool`** @@ -304,6 +331,16 @@ first_sequence > revision + 1 → CursorExpired Machine-checked as: _"bootstrap never silently diverges."_ Empirically pinned by `tests/resync.rs::nats_silently_clamps_resume_below_first_seq`. +**`restore_allowed(artifact_revision, local_revision, first_sequence) → bool`** + +The restore guard. An expired fold may be replaced by an artifact only if the artifact is strictly ahead of it (a restore never moves the fold, or the consumer's domain state, backward) and its cursor is still inside retention (`resume_window_ok`), so the resume from it is gap-free. + +``` +artifact_revision > local_revision && resume_window_ok(artifact_revision, first_sequence) +``` + +Machine-checked as: _"bootstrap never silently diverges from every write minus every real delete"_ on an evicting bucket (`tests/model.rs`), and _"a repair never moves the fold backward"_ (`tests/model_live_watch.rs`, where removing the guard is a mutation the checker catches). + ### Export Round State Machine ``` @@ -356,10 +393,11 @@ Machine-checked as: _"bootstrap never silently diverges."_ Empirically pinned by | Flush | `store.apply()` succeeds | streak < 16 | Streak reset; `on_applied` fired; continue watching | | Flush | `store.apply()` transient error | streak < 16 | Raw batch prepended to next `raw_batch`; streak++; warn; continue | | Flush | `store.apply()` error | streak ≥ 16 | `KvError::WatchError` returned; watch fail-stops | -| Watching | Watch task returns `CursorExpired` | first_seq > cursor+1 | Resync (if reader wired): list keys → synthetic deletes → ack; then `watch_all()` fallback | -| Resync | `reader.keys()` fails | — | Fatal: `KvError::WatchError`; restart re-runs the full resume → expiry → resync path | -| Resync | Fold range fails | — | Fatal: same | -| Watching | `GuardTrip` (in-band gap or 30s backstop) | `!resume_window_ok(frontier, first_seq)` | Watch task fails; `KvError::WatchError` surfaces; caller's restart hits `CursorExpired` and runs resync | +| Watching | Watch task gets `CursorExpired` (resume, or floor-guard trip mid-watch) | bucket keeps current values (or `Relist` chosen on unknown retention) | Drain channel + flush; key-listing diff: list keys → synthetic deletes → ack; then full re-list | +| Watching | Same | bucket evicts current values, `Restore`/`Auto` | Peek + check artifact (scope, ahead, fresh) → download → re-check → drain + flush → diff → chunked fold → cursor := artifact's → reply; resume `*_from(artifact cursor)` | +| Watching | Same | bucket evicts current values, `Relist` | Fatal: `KvError::WatchError` ("evicts current values"); fold untouched | +| Repair | listing / fold scan / download / check fails | — | Fatal: `KvError::WatchError`; restart re-runs the full resume → expiry → repair path (a part-done restore committed under the old cursor, so it re-runs idempotently) | +| Watching | `GuardTrip` (in-band gap or 30s backstop) | `!resume_window_ok(frontier, first_seq)` | Floor-guarded watch ends with `CursorExpired` → the repair rows above, in process | | Watching | `ExportRequest` | — | Flush pending batch; `store.export_to()` on blocking task; reply on oneshot; continue | | Any | Shutdown signal | — | Flush pending batch; abort watch task; return applied cursor | | Any | Channel closes cleanly | — | Flush remaining batch; return applied cursor | @@ -467,7 +505,7 @@ After upload: `completed_cursor_hex` and `completed_at_unix` are filled in by `c When the watched bucket is a **bounded** log (size-capped, history evicted), a fold is no longer rebuildable from NATS alone — the folds become the only full replicas. Export/import makes them transferable, which is what lets a new node, a node with a lost/corrupt fold, or a node whose cursor aged out of the log bootstrap at all. -**Artifact anatomy.** A directory: the backend's files under `data/`, plus `MANIFEST.json` carrying the artifact schema version, the backend identity and its on-disk format generation, per-file sizes + BLAKE3 digests, and — the load-bearing field — the **watch cursor the payload is exactly consistent with**. Import resumes the watch from that cursor and replays only the log tail. The manifest is written last and the whole stage is atomically renamed, so an artifact that exists is complete; a crash mid-export leaves only a hidden temp dir. +**Artifact anatomy.** A directory: the backend's files under `data/`, plus `MANIFEST.json` carrying the artifact schema version, the backend identity and its on-disk format generation, per-file sizes + BLAKE3 digests, and — the load-bearing field — the **watch cursor the payload is exactly consistent with**. Import resumes the watch from that cursor and replays only the log tail. The manifest is written last and the whole stage is atomically renamed, so an artifact that exists is complete; a crash mid-export leaves only a hidden temp dir. Artifacts exported through `watch_applied` also record the exporting watch's key **scope** (schema 2; the manifest is rewritten atomically after the seal), which the expiry restore checks covers the restoring watcher. Schema 1 (no scope) is still read; builds older than schema 2 refuse schema-2 manifests, so upgrade importers before exporters. **The cursor-consistency invariant.** `export_to(&mut self)` cannot run concurrently with `apply` (exclusive borrow), and inside `watch_applied` exports run between flushes via the `ExportRequest` channel (pending batch flushed first) — so the embedded cursor equals the applied cursor, exactly. Every backend re-proves it at export time by **reopening the copy** and checking cursor equality: because every `apply` commits the cursor in the same atomic batch as its data, a recovered cursor that matches the live one is a complete tail-loss detector. @@ -528,6 +566,22 @@ async-nats's bare `watch`/`watch_all`/`watch_many` ride `DeliverPolicy::New` — The fallback watch's re-list already carries every live key's value, so the resync only needs to learn which keys *no longer exist* — `reader.keys()` (headers only, no value bytes) is sufficient and cheap. The fold side of the diff is equally key-only: it streams via `for_each_in_range` rather than `range()`, so an All-scope resync against an on-disk backend never materializes the whole fold (values included) on the repair path. The synthetic deletes carry an unknown version and never advance the cursor: the fold's persisted cursor stays at its (expired) position until real re-list revisions move it, so a crash mid-resync just re-runs the same idempotent diff on the next start. Ordering, not versioning, provides correctness: the resync acks before the fallback watch is established, so a synthetic delete can never land after the re-list put that resurrects the same key. +### Why can't the key-listing resync be fixed for buckets that evict current values? + +Because the information isn't there. With `max_age` or `discard: old`, a key missing from the bucket is either "deleted, and its delete marker was evicted" or "aged out, never deleted". Both leave nothing in the stream. A listing diff can only guess, and either guess loses data: delete and you drop valid keys (the bug `tests/eviction.rs` reproduced live); keep and you resurrect real deletes. Writes made during the gap that also aged out aren't in the stream either. The only place that still has every write minus every real delete is a fold that was watching at the time, so the repair restores from one: the newest published artifact. Choosing between the two repairs reads the stream's live config at the moment of expiry (`KvWatcher::retention`) rather than trusting `StoreConfig`, which only applies at bucket creation and can be edited out-of-band. + +The same reasoning removed the old definition of convergence from the model. `tests/model.rs` used to judge a fold against what the bucket lists and never evicted a current value, so it certified the key-listing resync on evicting buckets as correct. It now judges against every write minus every real delete, has an `AgeOut` step, and keeps the old resync as a mutation the checker must catch. + +### Why apply a restore as a diff instead of swapping in the artifact's files? + +The result is a wholesale replacement within the watch scope, not a merge: every in-scope key ends up exactly as the artifact has it, and keys it lacks are deleted. But the mechanism is a diff folded through the normal path, for three reasons: + +- **Crash safety without a new on-disk protocol.** Swapping a live LSM directory isn't atomic (rename can't replace a non-empty directory), and a half-swapped fold with a cursor is worse than a stale one. The diff commits each chunk under the old, expired cursor and moves the cursor to the artifact's only in the final commit, so a crash mid-restore leaves a fold whose cursor is still expired: the restart runs the restore again, idempotently. That's the same argument the key-listing resync already relies on. +- **The consumer's domain state.** `apply` built routing tables (or hashrings) from the old fold. A file swap would leave them stale with no signal; the diff delivers exactly the changes through `parse`/`apply`, the same way the synthetic deletes always have. +- **Bounded memory.** The diff holds keys only; values are read back from the artifact a chunk at a time. + +The cost is a scan of both folds over the scope. Restores are rare, and the alternative costs correctness. + ### Why KvError: Clone instead of Box? A failed connect future may be observed by multiple concurrent callers waiting on a shared result. `Clone` lets the error fan out to N waiters without `Arc`. The cost: `std::io::Error` and `async-nats` error types are not `Clone`, so their structured cause chain is flattened into a pre-rendered `String` at the boundary. The trade-off is explicit: no `#[source]` chain, but the message carries context instead. @@ -631,7 +685,7 @@ Production code and an exhaustive model checker running the same logic closes th **What the transport layer does NOT verify:** - Object store authorization (delegated to `object_store` credentials) -- Pointer freshness (callers decide whether the embedded cursor is recent enough) +- Pointer freshness for manual bootstrap (callers decide whether the embedded cursor is recent enough). The automatic expiry restore does check it: `restore_allowed` refuses an artifact outside retention - Whether the exporting node's fold was correct (garbage in, garbage out — the integrity guarantees cover transport, not source correctness) ## Failure Modes @@ -640,10 +694,15 @@ Production code and an exhaustive model checker running the same logic closes th | ------------------------------------ | --------------------------------------------------------------------------- | ------------------------------------------------- | | Transient `store.apply()` failure | Raw batch re-queued at front; next flush commits cumulatively; streak counter incremented | Automatic up to 16 consecutive failures | | 16 consecutive `store.apply()` failures | Watch fail-stops: `KvError::WatchError` returned | Restart re-folds from last good cursor; investigate store (disk, permissions) | -| Live watch retention overrun (All scope) | Floor guard detects in-band (gapped delivery) or via 30 s backstop; watch fails with `KvError::WatchError` | Caller restarts; resume hits `CursorExpired`; resync repairs stale keys | +| Live watch retention overrun (All scope) | Floor guard detects in-band (gapped delivery) or via 30 s backstop; the watch ends with `CursorExpired` | `watch_applied` repairs in process (key-listing diff, or artifact restore on an evicting bucket) | | Live watch retention overrun (Prefix scope) | **Undetected until the next restart** — prefix watches deliver sparse revisions, so eviction gaps are indistinguishable from non-matching subjects client-side; the floor guard is sound for All scope only (a periodic probe here would false-positive into spurious resyncs, a design the model checker rejected) | Next restart's resume-window check hits `CursorExpired` → resync repairs; operate prefix-scoped watches with retention window comfortably above the restart/redeploy interval | | Resync `reader.keys()` failure | Watch fail-stops (fail-stop, not degrade — degrade semantics proven to leave stale keys permanently) | Restart re-runs full resume → expiry → resync | -| `CursorExpired` | `watch_applied` lists live keys, diffs fold, applies synthetic deletes, then falls back to state-sync re-list | Automatic; raw callers use `stale_keys()` manually | +| `CursorExpired`, bucket keeps current values | `watch_applied` lists live keys, diffs fold, applies synthetic deletes, then falls back to state-sync re-list | Automatic; raw callers use `stale_keys()` manually | +| `CursorExpired`, bucket evicts current values | `watch_applied` restores the in-scope fold from the newest artifact and resumes from its cursor | Automatic with `ExpiryRepair::Restore`/`Auto` | +| Fold holds data but no cursor (torn first checkpoint) | Repaired like an expiry at revision 0 before watching | Automatic with a repair armed | +| A write ages out before ANY node folds it | Every node that needed it fail-stops (no fold anywhere holds it; `tests/model_fleet.rs` proves this is the only way the fleet fail-stops) | Data loss by retention; widen retention or export more often | +| Key-listing resync on a bucket that evicts current values | Refused: `KvError::WatchError` ("evicts current values"); fold untouched | Wire `ExpiryRepair::Restore`/`Auto` with an artifact source | +| Restore: no artifact ahead, stale artifact, or one that doesn't cover the scope | `KvError::WatchError`, reason logged at `error`; fold untouched | Fix the export pipeline (run exports well inside `max_age` / `discard: old` turnover); restart | | `WatchError` | Watch stream dropped (NATS restart, reconnect) | Re-subscribe | | `Timeout` on any NATS op | CLOSE_WAIT half-dead TCP parks the `await` without this guard | Call `shutdown()` + `connect()` | | `RevisionMismatch` on CAS | Concurrent writer won the race | Re-read with `entry()`, resolve, retry | @@ -659,7 +718,7 @@ Production code and an exhaustive model checker running the same logic closes th | Artifact backend/format mismatch | `ArtifactInvalid` before any open; engine format markers re-checked internally | Fetch another artifact | | Export under churn — fjall copy | File GC'd mid-copy; retry ×3; verify-by-reopen catches anything torn | Abandon lease on persistent failure | | Export/upload fails mid-round | Lease abandoned (CAS delete); local artifact deleted | Next trigger elects a new node | -| Artifact cursor older than NATS log | `CursorExpired` on resume → full watch fallback + stale-key resync | Checkpoint more often than log retention window | +| Artifact cursor older than NATS log | `CursorExpired` on resume → the cursor-expiry repair; on an evicting bucket a stale artifact fails the restore | Export more often than the log's retention window | | Corrupt lease value | Treated as expired: CAS-stolen by next acquirer | Non-fatal; one bad write cannot wedge the fleet | | `prune()` I/O error | Stale payloads linger; warning logged | Retried next export round; correctness unaffected | @@ -678,7 +737,8 @@ Production code and an exhaustive model checker running the same logic closes th | `src/artifact.rs` | `ExportManifest`, `ArtifactFile`, BLAKE3 integrity; stage-then-rename discipline; backend `export_to` + `import` (append-log, fjall, RocksDB) | | `src/export_lease.rs` | `ExportLease`, `LeaseGuard`, `LeaseRecord`: fleet-wide at-most-one via embedded-expiry CAS | | `src/transport.rs` | `ObjectStoreTransport`, `ArtifactTransport` trait, `run_export_round`: monotonic pointer swap, multipart upload, prune, content-addressed keys (`feature = "transport"`) | -| `src/applied.rs` | `watch_applied` cursor-after-apply combinator, generic over `SnapshotStore`: `WatchScope`, `BatchConfig`, cursor-expired stale-key resync, `ExportRequest` handling | +| `src/applied.rs` | `watch_applied` cursor-after-apply combinator, generic over `SnapshotStore`: `WatchScope`, `BatchConfig`, cursor-expiry repair (drain, key-listing diff, artifact restore), `ExportRequest` handling (stamps the scope) | +| `src/repair.rs` | `ExpiryRepair`, `RestoreSource`, `RestoredFold`; scope coverage, restore checks, the in-scope restore diff | | `src/lib.rs` | Re-exports all public types; no logic | | `benches/` | Criterion benchmarks: snapshot write/checkpoint/load throughput, batch throughput, ACK subject parsing | | `tests/integration.rs` | NATS JetStream backend integration suite: each test boots its own `nats-server` on a free port; covers bucket create, CAS, watch semantics, cursor resume, and delete reconciliation | @@ -688,10 +748,14 @@ Production code and an exhaustive model checker running the same logic closes th | `tests/bootstrap.rs` | Tier-2 bootstrap proofs on live NATS + on-disk backends: exports from a live `watch_applied` loop under churn, imports as a second node, and asserts **delta-only resume** via delivery count (not just convergence — a full replay would produce the same end state) | | `tests/multi_export.rs` | Prevention proofs on live NATS + real fjall/RocksDB: slow-exporter clobber refused, crash window keeps bootstrap available, multi-SST post-compaction fidelity | | `tests/resync.rs` | Live-NATS conformance: NATS silent-clamp pinned; full expiry → resync chain e2e; no-reader divergence pinned | -| `tests/model.rs` | Stateright exhaustive model (~250M states, fleet size 2–3, unbounded rounds): pointer swap theorems (no regression, no torn pair, no dangling pointer, no silent divergence, terminal liveness); mutation tests prove each protocol guard load-bearing | +| `tests/repair_dst.rs` | Deterministic fault injection over the REAL `watch_applied`, on every backend: a simulated NATS log and a fault store (transient failure, crash before/after/torn) at every store-apply call, across 17 scenarios (expiry repairs and the normal path; All, Prefix, multi-prefix; complete and mid-catch-up artifacts) × both delivery orders × batch sizes — ~32k schedules on the append log, every single fault on fjall and RocksDB (deep tier for the full sweep). Asserts convergence to every write minus every real delete, domain state == fold, cursor monotone, no revision regression or phantom delete seen by `apply`, and every mid-run export resumes to the truth. Reverting any of nine repair steps fails it | +| `tests/model_repair.rs` | Stateright model of the repair AS STEPS (drain, pre-repair flush retry, diff, chunk, cursor commit, ack/reply) interleaved with deliveries, writes, eviction, publishes, transient failures, and crashes; checks that a resumable cursor reaches the truth, cursors are monotone, the domain never regresses or loses a current value, and every run converges; eleven mutations, each caught | +| `tests/model_fleet.rs` | Stateright fleet model where the exporters themselves expire, restore, start cursor-less, and publish their actual folds: every published artifact is the truth at its cursor (discharges the "exporters are correct" axiom); fleet-poisoning mutations caught | +| `tests/eviction.rs` | Live NATS + local object store, on `discard: old` and `max_age` buckets: expiry restores from the artifact (aged-out key kept, aged-out gap write recovered, real delete applied, identical to the exporter); key-listing resync refused; stale artifact fails the watch; a stalled, floor-guarded watcher overrun by a 20 MB burst trips mid-stream and is restored in process, identical to the exporter | +| `tests/model.rs` | Stateright exhaustive model (fleet size 2–3, unbounded rounds): pointer swap theorems (no regression, no torn pair, no dangling pointer, terminal liveness) and convergence to every write minus every real delete, on buckets that keep and that evict current values (`AgeOut`, artifact restore); mutation tests prove each protocol guard load-bearing, and keep the key-listing resync on an evicting bucket as a mutation the checker must catch | | `tests/model_applied.rs` | Stateright cursor-authority model: delivery → flush → transient failure → crash/restart; `DropFailedBatch` and `ResumeFromMemApplied` mutations pin both pre-fix bug classes | | `tests/model_resync_order.rs` | Stateright resync ack-barrier model: proves synthetic deletes strictly precede re-list puts; `NoAckBarrier` mutation reaches lost-recreate divergence | -| `tests/model_live_watch.rs` | Stateright floor-guard model: guarded variant proves fold always converges; unguarded variant pins permanent silent divergence as the machine-checked record of the pre-guard code | +| `tests/model_live_watch.rs` | Stateright floor-guard model: guarded variant proves the fold always converges to the truth, repaired by the key-listing diff or (evicting bucket) the artifact restore; unguarded variant pins permanent silent divergence; mutations: relist on an evicting bucket, restore without `restore_allowed` | | `tests/common/mod.rs` | Shared test helpers: ephemeral NATS server, MinIO harness, `ManifestPutCrash` transport injection | ## Configuration @@ -705,7 +769,7 @@ Config applies only at creation. If the bucket already exists, the existing one | `max_bytes` | 10 MiB | Required by Synadia Cloud; omit only for self-hosted NATS | | `max_history` | 1 | Config stores rarely need change history | | `num_replicas` | 1 | Set to 3 for production HA clusters | -| `max_age` | None | Set to gate retention window (also determines when cursors expire) | +| `max_age` | None | Set to gate retention window (also determines when cursors expire). Evicts current values: expired consumers must restore from artifacts, exported well inside `max_age` | ### NatsConnectionConfig diff --git a/README.md b/README.md index 08ff794..b0035b8 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,8 @@ match watcher.watch_all_from(&cursor, tx.clone()).await { } ``` +The full replay only re-delivers what NATS still holds. On a bucket with `max_age` or `discard: old`, values that aged out are gone from it; [`watch_applied`](#cursor-expiry) restores those from an artifact. + `watch_prefix_from()` works the same way for prefix-filtered streams, and `watch_prefixes_from()` resumes the union of several prefixes on one multi-filter consumer. @@ -298,7 +300,7 @@ let (resume, store) = AppendLogSnapshot::load(Path::new("/var/lib/svc/state.snap let final_cursor = watch_applied( watcher, WatchScope::All, Some(resume), - Some(reader), // arms the cursor-expired stale-key resync; None to skip + repair, // cursor-expiry repair (see "Cursor expiry"); None to skip Some(store), None, // store; export-request channel BatchConfig::default(), parse, apply, on_applied, shutdown, @@ -318,8 +320,8 @@ let final_cursor = watch_applied( watcher, WatchScope::All, // or Prefix("node.".into()) / Prefixes(vec![...]) Some(resume), // Option — resume here, or None - Some(reader), // Option> — arms the - // cursor-expired stale-key resync, or None + repair, // ExpiryRepair — how an expired cursor is + // repaired (see below), or None Some(store), // any SnapshotStore (e.g. AppendLogSnapshot), or None None, // Option> — live exports BatchConfig::default(), // 10ms window, 100 updates per batch @@ -332,14 +334,46 @@ let final_cursor = watch_applied( A batch closes when `window` elapses or it hits `max` updates, whichever comes first. Then, in order: `apply(batch)` runs to completion, the cursor advances to the batch's highest revision, the batch + cursor are folded into the `store` atomically (on a blocking task), and `on_applied` fires. +With a `store`, resume from the cursor the store reports (`load`/`open`), not one you persisted in `on_applied`: across a transient store failure, `on_applied` reports a cursor ahead of what the store has made durable. + Persist the cursor on receipt instead and a crash between receive and apply loses data: the cursor reads "caught up to rev N" while rev N sits in an unapplied buffer, and the next resume starts past it. `watch_applied` checkpoints at the applied cursor, so a persisted cursor always means every update up to it has been applied. - `parse` returning `None` (corrupt bytes, irrelevant key) still advances the cursor — nothing to apply means nothing to skip. -- On `CursorExpired`, it falls back to a full watch automatically. With a `reader` wired, it first diffs the fold against the bucket's live keys and applies synthetic deletes for keys that vanished during the gap (their delete markers were evicted with the cursor) — the one case the fallback re-list can't cover. +- On `CursorExpired` it repairs the fold, then keeps watching. See [Cursor expiry](#cursor-expiry). - It returns the final applied cursor on shutdown or stream close. `apply` runs inline. If it panics, the panic aborts the watch. +### Cursor expiry + +A cursor expires when NATS has evicted the revisions after it: at resume, or mid-watch when retention overruns a live All-scope watch (the floor guard). Both take the same repair, and the right repair depends on what the bucket's retention evicts. + +**Buckets that keep current values** (`discard: new`, no `max_age`). Only old history and delete markers get evicted, so a key missing from the bucket was deleted. `ExpiryRepair::Relist(reader)` lists the bucket's live keys, deletes the fold's in-scope keys that are gone, then re-lists every live value. + +**Buckets that evict current values** (`max_age`, or `discard: old` under `max_bytes`). A key missing from the bucket may just have aged out, and a write made while the node was offline may have aged out too. NATS can't tell those apart from a delete, so the key list can't be trusted. `ExpiryRepair::Restore { reader, restore }` replaces the in-scope fold with the newest published artifact (which holds every write minus every real delete) and resumes from its cursor. It never moves a key to an older revision and never deletes a key the bucket lists as live, so even an artifact exported while its exporter was catching up produces no transient phantom deletes or regressions. `Relist` is refused on these buckets: the watch fails rather than delete valid keys. + +`ExpiryRepair::Auto { reader, restore }` reads the bucket's retention at expiry and picks for you: + +```rust +use slipstream::{ArtifactRestore, ExpiryRepair}; + +let repair = ExpiryRepair::Auto { + reader: bucket.reader(), + restore: Arc::new(ArtifactRestore::new( + transport, // Arc, same key the fleet exports to + "routes/latest", + "/var/lib/svc/scratch", // same filesystem as the fold + |artifact, dest| FjallSnapshot::import(artifact, dest, config.clone()), + )), +}; +``` + +A restore only accepts an artifact that is ahead of the local fold, covers the watch's scope (recorded in the artifact by the exporting `watch_applied`), and whose cursor NATS still retains. Otherwise there is no safe recovery: the watch fails and logs the reason at `error`. **Run exports well inside `max_age`** (or the `discard: old` turnover), or an expired node has nothing fresh to restore from. + +A node with no cursor on a bucket that has already evicted current values seeds from the artifact too, since a re-list alone would miss whatever aged out. + +`Some(reader)` from older code still compiles and means `Relist`. `None` (or no `store`) falls back to the re-list alone and warns that deleted keys may persist. + ## NATS mapping | Concept | NATS primitive | @@ -371,7 +405,7 @@ if caps.global_ordering { /* VersionToken is globally ordered across keys */ } | `NotConnected` | Operation before `connect()` | Call `connect()` | | `AlreadyExists` | `create()` on a live key | Read current state, decide | | `RevisionMismatch` | CAS conflict on `update()` / `delete_with_version()` | Re-read, retry | -| `CursorExpired` | `watch_*_from()` cursor compacted by NATS | Fall back to `watch_all()` | +| `CursorExpired` | `watch_*_from()` cursor compacted by NATS (at resume, or mid-stream when retention overruns an All-scope watch) | `watch_applied` repairs it; raw callers fall back to `watch_all()` on buckets that keep current values | | `WatchError` | NATS stream dropped | Re-subscribe | ## Credentials diff --git a/mise.toml b/mise.toml index 8e183f9..d0ba7d7 100644 --- a/mise.toml +++ b/mise.toml @@ -11,10 +11,11 @@ yamlfmt = "0.12.1" dprint = "0.50.2" "ubi:nats-io/nats-server" = "2.14.1" # Real-S3 transport tests (tests/transport_s3.rs): single static binaries, no -# Docker — same pattern as nats-server. mc creates the test bucket. (ubi can't -# install minio: its releases ship extension-less raw binaries that ubi's -# asset filter rejects, hence the asdf-backend plugin.) -"asdf:mise-plugins/mise-minio" = "2025-09-07T16-13-09Z" +# Docker — same pattern as nats-server. mc creates the test bucket. MinIO no +# longer publishes community server binaries (dl.min.io answers 410 Gone), so +# the same pinned release is built from source with `go install` (~45 s). +go = "1.25" +"go:github.com/minio/minio" = "RELEASE.2025-09-07T16-13-09Z" "aqua:minio/mc" = "RELEASE.2025-08-13T08-35-41Z" [tasks.format] diff --git a/src/applied.rs b/src/applied.rs index 1429020..1c0c8c7 100644 --- a/src/applied.rs +++ b/src/applied.rs @@ -42,7 +42,11 @@ //! logic; for the tunnel router it swaps the route table, for the edge origin //! watcher it rebuilds the hashrings. //! - `on_applied`: fires once per flush, *after* `apply` returns, with the new -//! applied cursor. Callers use it to persist the cursor for the next restart. +//! applied cursor. Without a store, callers use it to persist the cursor for +//! the next restart. WITH a store, resume from the store's own cursor +//! instead: across a transient store failure the batch is re-queued, but +//! `on_applied` has already reported the advanced cursor — ahead of what the +//! store made durable — so resuming from it could skip that batch. //! //! ## Panics //! @@ -56,10 +60,15 @@ use std::time::Duration; use tokio::sync::mpsc; use tokio::sync::{oneshot, watch}; -use tracing::warn; +use tracing::{error, info, warn}; use crate::artifact::ExportManifest; -use crate::kv::{KvError, KvReader, KvUpdate, KvWatcher, WatchCursor}; +use crate::kv::{KvError, KvReader, KvUpdate, KvWatcher, Retention, WatchCursor}; +use crate::protocol::resume_window_ok; +use crate::repair::{ + ExpiryRepair, RestoreSource, RestoredFold, check_restore, cursor_rank, materialize, + restore_diff, +}; use crate::snapshot::{SnapshotError, SnapshotStore}; /// A request, sent into a running [`watch_applied`] loop, to export the fold it @@ -112,20 +121,50 @@ impl WatchScope { } } -/// Internal: a cursor-expired resync handoff from the watch task to the main -/// loop. Carries the bucket's live key listing for the watch scope; the main -/// loop diffs it against the fold and applies synthetic deletes for keys that -/// vanished during the gap, then acks so the watch task can start the fallback -/// watch — the ack ordering guarantees every synthetic delete is applied before -/// the first re-list put arrives. -struct ResyncRequest { - live_keys: Vec, - ack: oneshot::Sender<()>, +/// Internal: a cursor-expired repair handoff from the watch task to the main +/// loop, which owns the fold. The watch task stays parked on the ack/reply +/// until the main loop has folded the repair, so the repair is strictly +/// ordered between everything delivered before the expiry and everything the +/// next watch delivers. +enum RepairRequest { + /// The key-listing diff: the bucket's live keys for the watch scope. The + /// main loop applies synthetic deletes for in-scope keys missing from it, + /// then acks so the watch task can start the fallback re-list. + Relist { + live_keys: Vec, + ack: oneshot::Sender<()>, + }, + /// The artifact restore: a verified artifact fold, and the bucket's live + /// keys for the scope (listed after the fetch). The main loop folds the + /// in-scope difference, advances to the artifact's cursor, and replies + /// with it so the watch task resumes from there. + Restore { + restored: RestoredFold, + live: std::collections::HashSet, + reply: oneshot::Sender>, + }, } -/// Internal: what the watch task needs to initiate a resync — the reader that -/// lists live keys and the channel into the main loop that owns the fold. -type ResyncHandle = (Arc, mpsc::Sender); +/// Internal: what the watch task needs to repair an expiry — the caller's +/// chosen repair, and the channel into the main loop that owns the fold. +struct RepairHandle { + mode: ExpiryRepair, + tx: mpsc::Sender>, +} + +/// Internal: the repair the watch task settled on for one expiry. +enum Plan<'a, S> { + /// Fall back to the re-list alone (nothing armed). + ReListOnly, + /// The key-listing diff, then the re-list. + Relist(&'a Arc, &'a mpsc::Sender>), + /// The artifact restore, then a resume from the artifact's cursor. + Restore( + &'a Arc, + &'a Arc>, + &'a mpsc::Sender>, + ), +} /// Batching policy for [`watch_applied`]. /// @@ -180,31 +219,44 @@ impl Default for BatchConfig { /// whose `apply` had not returned. The store fold is atomic (data + cursor), so a /// crash leaves the store consistent and resume re-folds only the tail. /// -/// # Cursor expiry and stale-key resync +/// # Cursor expiry /// -/// On [`KvError::CursorExpired`] from the `*_from` resume path, this logs and -/// falls back to a full-scope watch (`watch_all` / `watch_prefix` / -/// `watch_prefixes`), whose state-sync re-list re-delivers the current value of -/// every in-scope key as puts. The re-list alone cannot cover keys that were -/// **deleted during the gap** and whose delete markers the backend has since -/// evicted — they simply don't appear, leaving the fold (and the caller's -/// domain state) holding them forever. +/// The cursor expires when the log no longer holds the revision after it: at +/// resume ([`KvError::CursorExpired`] from the `*_from` watch), or mid-watch, +/// when the live floor guard finds retention overran the consumer (All scope). +/// Both take the same path. Everything delivered before the expiry is folded +/// first; then `repair` (an [`ExpiryRepair`], or the pre-0.8 +/// `Option>`, which converts to it) decides what the fold +/// can trust: /// -/// When both `reader` and `store` are provided, the expiry path closes that -/// hole: before the fallback watch starts, the bucket's live keys are listed -/// via `reader`, diffed against the fold's in-scope keys, and a synthetic -/// [`KvUpdate::Delete`] (with an unknown [`VersionToken`](crate::VersionToken)) is run through -/// `parse`/`apply`/`store` for each key that vanished. The synthetic deletes -/// are strictly ordered before the first re-list put, so a key deleted and -/// re-created during the gap converges correctly. Without a `reader` (or -/// without a `store` to diff against) the fallback is re-list-only and a -/// warning marks the possible stale keys. +/// - [`ExpiryRepair::Relist`]: the key-listing diff. The bucket's live keys +/// are listed via the reader and diffed against the fold's in-scope keys; a +/// synthetic [`KvUpdate::Delete`] (unknown +/// [`VersionToken`](crate::VersionToken), never moves the cursor) runs +/// through `parse`/`apply`/store for each key that vanished, strictly before +/// the full-scope re-list (`watch_all` / `watch_prefix` / `watch_prefixes`) +/// starts. Sound only when "not in the bucket" means "deleted", so it is +/// REFUSED — the watch fails — when the watcher reports that retention +/// evicts current values (`max_age`, `discard: old`). +/// - [`ExpiryRepair::Restore`]: the artifact restore. The newest published +/// artifact is checked (ahead of the fold, covers this scope, cursor still +/// inside retention), downloaded, and the in-scope fold replaced with its +/// contents: changed keys as puts carrying their real revisions, keys it +/// lacks as synthetic deletes, all through `parse`/`apply`/store, then the +/// cursor advanced to the artifact's. The watch resumes from there. Each +/// chunk commits under the old (expired) cursor until the last, so a crash +/// mid-restore re-runs it on the next start. A failed check fails the watch +/// with the reason logged at `error`: there is no safe recovery from a +/// stale artifact. +/// - [`ExpiryRepair::Auto`]: Relist when the bucket never evicts current +/// values, Restore otherwise. +/// - [`ExpiryRepair::None`] (or no `store`): the re-list alone, with a warning +/// that keys deleted during the gap may persist. /// -/// A resync that was armed but FAILS (live-key listing or fold diff error) is -/// fatal to the watch — degrading to re-list-only would silently leave -/// deleted keys in the fold (`tests/model.rs` proves that divergence -/// reachable), so the error surfaces and the caller's restart retries the -/// resume → expiry → resync path from scratch. +/// A repair that was armed but FAILS (listing, fold scan, download, checks) is +/// fatal to the watch — degrading would silently leave the fold wrong +/// (`tests/model.rs` proves that divergence reachable) — and the caller's +/// restart retries from scratch. /// /// See `ARCHITECTURE.md` ("Applied-Cursor Watch") for the invariant and its /// rationale. @@ -223,20 +275,22 @@ impl Default for BatchConfig { // must sit on the function: a statement-scoped `#[allow]` inside the macro body // trips the experimental attributes-on-expressions gate (E0658) on stable. #[allow(unused_assignments)] -pub async fn watch_applied( +pub async fn watch_applied( watcher: Arc, scope: WatchScope, resume: Option, - // `Some(reader)` arms the cursor-expired stale-key resync (see the function - // docs); `None` keeps the re-list-only fallback. Only consulted on expiry — - // the hot path never touches it. - reader: Option>, + // How an expired cursor is repaired (see the function docs). Only + // consulted on expiry (and, for a restore, at a cursor-less start on a + // bucket that has already evicted current values) — the hot path never + // touches it. + repair: R, mut store: Option, // `Some(rx)` arms an export-request arm in the select loop: each // [`ExportRequest`] is handled between flushes (pending batch flushed // first), so the exported artifact's cursor is the applied cursor (or, // across a transiently failed store flush, the store's own lagging but - // self-consistent cursor — never a cursor past unfolded data). + // self-consistent cursor — never a cursor past unfolded data). The + // artifact's manifest records this watch's scope, which a restore checks. // `None` (or dropping the paired sender) leaves the loop's behavior // unchanged. mut exports: Option>, @@ -255,6 +309,7 @@ where P: FnMut(&KvUpdate) -> Option + Send, A: FnMut(Vec) + Send, O: FnMut(WatchCursor) + Send, + R: Into>, { // The cursor we'll return. Initialized from the resume position so that a // watch which receives nothing new still reports the position it resumed @@ -265,31 +320,50 @@ where None => WatchCursor::none(), }; - // The scope's prefixes, for the resync diff against the fold. Cloned out - // before `scope` moves into the watch task. + // The scope's prefixes, for the repair diffs against the fold and the + // scope recorded in exported artifacts. Cloned out before `scope` moves + // into the watch task. let scope_prefixes = scope.prefixes(); - // Cursor-expired resync channel, armed only when there is a reader to list - // live keys AND a store to diff them against. The watch task sends the live - // listing here and waits for the ack before starting the fallback watch, so - // synthetic deletes always precede the re-list. - let (resync_pair, mut resyncs): (Option, Option>) = - match reader { - Some(reader) if store.is_some() => { - let (rs_tx, rs_rx) = mpsc::channel(1); - (Some((reader, rs_tx)), Some(rs_rx)) - } - _ => (None, None), - }; + // Repair channel, armed only when there is a repair to run AND a store to + // run it against (both repairs edit the fold). The watch task sends the + // repair here and waits for the ack/reply before starting its next watch. + let (repair_handle, mut repairs): ( + Option>, + Option>>, + ) = match repair.into() { + ExpiryRepair::None => (None, None), + mode if store.is_some() => { + let (tx, rx) = mpsc::channel(1); + (Some(RepairHandle { mode, tx }), Some(rx)) + } + _ => (None, None), + }; - // Spawn the watch task. It owns the cursor-expired fallback so the main loop - // only ever sees a clean ordered stream of updates on `rx`. + // A fold with data but no cursor can't be re-listed blind (see + // `run_watch`). Only the first entry is read. + let unanchored = match &store { + Some(st) if resume.as_ref().is_none_or(WatchCursor::is_none) => fold_has_entries(st) + .map_err(|e| KvError::WatchError(format!("reading the fold at start failed: {e}")))?, + _ => false, + }; + + // Spawn the watch task. It owns the cursor-expiry handling so the main loop + // only ever sees a clean ordered stream of updates on `rx`, plus repairs. let (tx, mut rx) = mpsc::channel::(config.channel_capacity.max(1)); let handle = { let watcher = Arc::clone(&watcher); - tokio::spawn( - async move { run_watch(watcher.as_ref(), &scope, resume, resync_pair, tx).await }, - ) + tokio::spawn(async move { + run_watch( + watcher.as_ref(), + &scope, + resume, + unanchored, + repair_handle, + tx, + ) + .await + }) }; // Batch state. @@ -321,6 +395,16 @@ where // the absolute instant lives in the pinned `sleep` future below. let mut batch_deadline: Option = None; + // A single timer future, reset in place each time a batch opens. The old + // `tokio::time::sleep(timeout)` lived inside the select arm, so it was + // re-created on every loop iteration — one Arc-backed timer-wheel entry + // allocated, registered, and immediately dropped per received update. + // Pinning one future and `reset`-ing it reuses that single allocation; the + // `if batch_deadline.is_some()` guard keeps it from firing while idle, so + // its initial already-elapsed deadline is never observed. + let sleep = tokio::time::sleep(Duration::ZERO); + tokio::pin!(sleep); + // Flush the current batch, in order: run the domain `apply` (if non-empty) to // completion, advance the cursor, fold the raw batch + cursor durably into // `store`, then fire `on_applied`. The store fold runs on a blocking task @@ -334,8 +418,8 @@ where macro_rules! flush { () => {{ // Nothing received since the last flush → nothing to do at all. - // (`raw_batch` can be non-empty with no cursor advance only via the - // resync path's synthetic deletes, which carry no revision.) + // (`raw_batch` can be non-empty with no cursor advance only via a + // repair's synthetic updates, which carry no stream position.) if !batch.is_empty() || !raw_batch.is_empty() || !batch_high.is_none() { if !batch.is_empty() { // INVARIANT: apply() runs and RETURNS before any cursor @@ -351,15 +435,19 @@ where if advanced { applied = batch_high.clone(); } - if !raw_batch.is_empty() + // A cursor-only advance (nothing raw to fold) still commits + // to the store: a restore's final step moves the cursor to the + // artifact's with no updates left. In the steady state every + // advance carries raw updates, so this changes nothing there. + if (!raw_batch.is_empty() || advanced) && let Some(mut st) = store.take() { let raw = std::mem::take(&mut raw_batch); - // Fold at the post-advance cursor. A synthetic-deletes-only - // batch leaves the cursor where it was (the deletes are a - // state correction, not log entries), which is safe: an - // unchanged — possibly expired — cursor only ever re-runs - // this same resync on the next restart. + // Fold at the post-advance cursor. A repair-only batch + // (synthetic updates) leaves the cursor where it was (they + // are a state correction, not log entries), which is safe: + // an unchanged — possibly expired — cursor only ever re-runs + // the same repair on the next restart. let cur = applied.clone(); // Hand the store AND the raw batch back on a clean return: // a *failed* apply (Ok(Err)) RE-QUEUES the batch so the @@ -422,15 +510,63 @@ where }}; } - // A single timer future, reset in place each time a batch opens. The old - // `tokio::time::sleep(timeout)` lived inside the select arm, so it was - // re-created on every loop iteration — one Arc-backed timer-wheel entry - // allocated, registered, and immediately dropped per received update. - // Pinning one future and `reset`-ing it reuses that single allocation; the - // `if batch_deadline.is_some()` guard keeps it from firing while idle, so - // its initial already-elapsed deadline is never observed. - let sleep = tokio::time::sleep(Duration::ZERO); - tokio::pin!(sleep); + // Take one update off the watch channel into the pending batch. + macro_rules! ingest { + ($u:expr) => {{ + let u: KvUpdate = $u; + // Cursor authority: every received update bumps the pending + // high-water, regardless of whether `parse` keeps it — but only + // when it carries a real position. An unknown version (e.g. an + // unparseable ACK subject on the hand-built multi-prefix consumer + // path) must neither mint a fake cursor nor clobber the real high + // from earlier in the batch; skipping it under-advances at worst, + // and re-delivery on resume is idempotent. + if !u.version().is_unknown() { + batch_high = WatchCursor::from_version(u.version().clone()); + } + + // Buffer the raw update for the durable store fold (which commits + // the whole batch + cursor atomically on flush). Done before + // `parse` consumes `u` by reference, and only when a store is + // present so the no-persistence path keeps its zero-copy cost. + if store.is_some() { + raw_batch.push(u.clone()); + } + + if let Some(parsed) = parse(&u) { + batch.push(parsed); + } + + // Arm the window on the first received update of a batch — even a + // parse-rejected one, so the cursor advances within `window` even + // through a run of irrelevant keys. Reset the pinned timer to the + // new deadline rather than allocating a fresh `Sleep`. + if batch_deadline.is_none() { + let deadline = tokio::time::Instant::now() + config.window; + sleep.as_mut().reset(deadline); + batch_deadline = Some(deadline); + } + + // Flush on a full parsed batch, or — when persisting — a full raw + // batch, so a window packed with parse-rejected updates can't grow + // `raw_batch` without bound before the window elapses. + if batch.len() >= config.max || raw_batch.len() >= config.max { + flush!(); + } + }}; + } + + // Fail the watch from inside a repair: the fold is either unchanged or + // part-way through a repair that commits under its old (expired) cursor, + // so the caller's restart re-runs the whole expiry path. + macro_rules! repair_fatal { + ($($msg:tt)*) => {{ + let msg = format!($($msg)*); + error!(%msg, "cursor-expiry repair failed; aborting watch"); + handle.abort(); + return Err(KvError::WatchError(msg)); + }}; + } loop { tokio::select! { @@ -461,90 +597,224 @@ where flush!(); } - // Cursor-expired resync. Placed before `rx.recv()` (biased) so the - // synthetic deletes are folded before any update the fallback watch - // delivers — though the ack protocol already guarantees the fallback - // hasn't started while this arm runs. Diff the fold's in-scope keys - // against the bucket's live listing; anything the fold holds that - // the bucket no longer does vanished during the gap (its delete - // marker evicted with the cursor), so synthesize the delete the - // re-list can't deliver. - req = async { resyncs.as_mut().expect("arm guarded by is_some").recv().await }, - if resyncs.is_some() => { + // Cursor-expiry repair. Placed before `rx.recv()` (biased), but + // ordering against the update stream comes from the drain below + // and the ack/reply protocol: the watch task is parked until this + // arm finishes, so nothing from its next watch can interleave. + req = async { repairs.as_mut().expect("arm guarded by is_some").recv().await }, + if repairs.is_some() => { match req { - Some(ResyncRequest { live_keys, ack }) => { - // Flush first so the diff runs against a fold that - // reflects everything received so far. + Some(req) => { + // Fold everything the watch task delivered before it + // asked for this repair. A floor-guard trip ends a + // live watch mid-stream with updates still buffered in + // the channel: the repair must see them (it diffs the + // fold) and supersede them (one applied after the + // repair would resurrect pre-repair state, and could + // even move the cursor backward). The watch task is + // parked on the ack/reply, so this drains exactly that + // backlog. + while let Ok(u) = rx.try_recv() { + ingest!(u); + } flush!(); - let live: std::collections::HashSet<&str> = - live_keys.iter().map(String::as_str).collect(); - let mut stale: Vec = Vec::new(); - if let Some(st) = &store { - for prefix in &scope_prefixes { - // Stream the fold's keys rather than `range()`, - // which buffers every in-scope entry — values - // included — into one Vec. On an on-disk backend - // holding a fold larger than RAM (the case those - // backends exist for), an All-scope resync would - // materialize the entire fold on the repair - // path. Only the keys matter for the diff. - if let Err(e) = st.for_each_in_range(prefix, |entry| { - if !live.contains(entry.key.as_str()) { - stale.push(entry.key); + // Both repairs diff the STORE, so it must hold + // everything delivered. A transient store failure + // re-queues the batch in `raw_batch`, which the diff + // can't see: it would miss a re-queued put that a + // deletion during the gap has since removed, and the + // put would then commit and resurrect the key. Retry + // until the store has it all; a persistent failure + // hits the flush's own fail-stop bound. + while !raw_batch.is_empty() { + flush!(); + } + match req { + RepairRequest::Relist { live_keys, ack } => { + // Diff the fold's in-scope keys against the + // bucket's live listing; anything the fold + // holds that the bucket no longer does vanished + // during the gap (its delete marker evicted with + // the cursor), so synthesize the delete the + // re-list can't deliver. + let live: std::collections::HashSet<&str> = + live_keys.iter().map(String::as_str).collect(); + let mut stale: Vec = Vec::new(); + if let Some(st) = &store { + for prefix in &scope_prefixes { + // Stream the fold's keys rather than + // `range()`, which buffers every in-scope + // entry — values included — into one Vec. + // On an on-disk backend holding a fold + // larger than RAM (the case those backends + // exist for), an All-scope resync would + // materialize the entire fold on the + // repair path. Only the keys matter. + if let Err(e) = st.for_each_in_range(prefix, |entry| { + if !live.contains(entry.key.as_str()) { + stale.push(entry.key); + } + Ok(()) + }) { + // FATAL, not a degrade: an incomplete + // diff silently leaves deleted keys in + // the fold forever (tests/model.rs + // proves the divergence reachable + // under degrade semantics). Fail the + // watch; the restart re-runs the + // resume → expiry → resync from + // scratch. + warn!(error = %e, prefix = %prefix, + "resync fold scan failed; aborting watch rather than diverging"); + handle.abort(); + return Err(KvError::WatchError(format!( + "cursor-expired resync failed listing fold prefix {prefix:?}: {e}" + ))); + } } - Ok(()) - }) { - // FATAL, not a degrade: an incomplete - // diff silently leaves deleted keys in - // the fold forever (tests/model.rs - // proves the divergence reachable - // under degrade semantics). Fail the - // watch; the restart re-runs the - // resume → expiry → resync from - // scratch. - warn!(error = %e, prefix = %prefix, - "resync fold scan failed; aborting watch rather than diverging"); - handle.abort(); - return Err(KvError::WatchError(format!( - "cursor-expired resync failed listing fold prefix {prefix:?}: {e}" - ))); } + // Overlapping prefixes can list a key twice. + stale.sort_unstable(); + stale.dedup(); + if !stale.is_empty() { + warn!(stale = stale.len(), "cursor-expired resync: deleting keys that vanished during the gap"); + } + for key in stale { + // Synthetic: carries no revision (unknown + // version) and so never advances the cursor. + let u = KvUpdate::Delete { + key, + version: crate::kv::VersionToken::unknown(), + }; + if store.is_some() { + raw_batch.push(u.clone()); + } + if let Some(parsed) = parse(&u) { + batch.push(parsed); + } + } + flush!(); + // Ack AFTER the deletes are applied: the watch + // task is holding the fallback watch until it + // hears back, which is what orders deletes + // before the re-list + // (tests/model_resync_order.rs proves the + // barrier load-bearing). If the flush's STORE + // apply failed transiently, the deletes sit + // re-queued at the FRONT of the raw batch — + // still strictly before any re-list put in the + // eventual cumulative commit, and the domain + // apply saw them before this ack either way. + let _ = ack.send(()); } - } - // Overlapping prefixes can list a key twice. - stale.sort_unstable(); - stale.dedup(); - if !stale.is_empty() { - warn!(stale = stale.len(), "cursor-expired resync: deleting keys that vanished during the gap"); - } - for key in stale { - // Synthetic: carries no revision (unknown version) - // and so never advances the cursor. - let u = KvUpdate::Delete { - key, - version: crate::kv::VersionToken::unknown(), - }; - if store.is_some() { - raw_batch.push(u.clone()); - } - if let Some(parsed) = parse(&u) { - batch.push(parsed); + RepairRequest::Restore { + restored, + live, + reply, + } => { + let target = restored.manifest().cursor.clone(); + // The authoritative ahead check (the watch + // task's ran against its resume cursor; a + // mid-watch expiry delivered past that). A + // restore never moves the fold backward. + if cursor_rank(&target) <= cursor_rank(&applied) { + let msg = format!( + "cursor-expired restore: the latest artifact (cursor \ + {target:?}) is not ahead of the fold (cursor {applied:?}); \ + nothing newer to restore from" + ); + error!(%msg, "cursor-expiry repair refused"); + // The watch task fails with this; its + // channel closing ends the loop. + let _ = reply.send(Err(KvError::WatchError(msg))); + } else { + let Some(st) = store.take() else { + unreachable!("repairs are armed only with a store") + }; + let prefixes = scope_prefixes.clone(); + let (st, mut restored, diff) = + match tokio::task::spawn_blocking(move || { + let diff = + restore_diff(&st, restored.fold(), &prefixes, &live); + (st, restored, diff) + }) + .await + { + Ok(v) => v, + Err(e) => repair_fatal!( + "cursor-expired restore: diff task panicked: {e}" + ), + }; + store = Some(st); + let diff = match diff { + Ok(d) => d, + Err(e) => repair_fatal!( + "cursor-expired restore: diffing the fold against the \ + artifact failed: {e}" + ), + }; + info!( + from = ?applied, + to = ?target, + changed = diff.len(), + "cursor-expired restore: replacing the in-scope fold with the artifact's" + ); + // Fold the difference through parse/apply/ + // store in `max`-sized chunks, values read + // back per chunk. These carry no stream + // position: every chunk commits under the + // old cursor, so a crash part-way re-runs + // the (idempotent) restore on restart. + let mut ops = diff.into_iter(); + loop { + let chunk: Vec<_> = ops.by_ref().take(batch_cap).collect(); + if chunk.is_empty() { + break; + } + let (r, updates) = match tokio::task::spawn_blocking(move || { + let updates = materialize(restored.fold(), chunk); + (restored, updates) + }) + .await + { + Ok(v) => v, + Err(e) => repair_fatal!( + "cursor-expired restore: read task panicked: {e}" + ), + }; + restored = r; + let updates = match updates { + Ok(u) => u, + Err(e) => repair_fatal!( + "cursor-expired restore: reading the artifact failed: {e}" + ), + }; + for u in updates { + if store.is_some() { + raw_batch.push(u.clone()); + } + if let Some(parsed) = parse(&u) { + batch.push(parsed); + } + } + flush!(); + } + // Now the fold IS the artifact's (in + // scope): take its cursor, committed to + // the store with the final chunk's + // re-queued remainder, if any. + batch_high = target.clone(); + flush!(); + // Closing an LSM can block; drop the + // temporary fold and its scratch dir off + // the async thread. + let _ = tokio::task::spawn_blocking(move || drop(restored)).await; + let _ = reply.send(Ok(target)); + } } } - flush!(); - // Ack AFTER the deletes are applied: the watch task is - // holding the fallback watch until it hears back, which - // is what orders deletes before the re-list - // (tests/model_resync_order.rs proves the barrier - // load-bearing). If the flush's STORE apply failed - // transiently, the deletes sit re-queued at the FRONT - // of the raw batch — still strictly before any - // re-list put in the eventual cumulative commit, and - // the domain apply saw them before this ack either - // way. - let _ = ack.send(()); } - None => resyncs = None, + None => repairs = None, } } @@ -558,7 +828,9 @@ where // tests/model_applied.rs); the artifact never includes unfolded // data, and a bootstrap from it simply replays the short gap. // The export itself runs on a blocking task with the store moved - // in and taken back — the same offload the flush path uses. + // in and taken back — the same offload the flush path uses. The + // artifact's manifest records this watch's scope, so a restore can + // tell whether it covers the restoring watcher. req = async { exports.as_mut().expect("arm guarded by is_some").recv().await }, if exports.is_some() => { match req { @@ -566,8 +838,11 @@ where flush!(); match store.take() { Some(mut st) => { + let scope = scope_prefixes.clone(); match tokio::task::spawn_blocking(move || { - let res = st.export_to(&dest_dir); + let res = st.export_to(&dest_dir).and_then(|_| { + crate::artifact::stamp_scope(&dest_dir, &scope) + }); (st, res) }) .await @@ -606,51 +881,7 @@ where update = rx.recv() => { match update { - Some(u) => { - // Cursor authority: every received update bumps the - // pending high-water, regardless of whether `parse` - // keeps it — but only when it carries a real position. - // An unknown version (e.g. an unparseable ACK subject - // on the hand-built multi-prefix consumer path) must - // neither mint a fake cursor nor clobber the real high - // from earlier in the batch; skipping it under-advances - // at worst, and re-delivery on resume is idempotent. - if !u.version().is_unknown() { - batch_high = WatchCursor::from_version(u.version().clone()); - } - - // Buffer the raw update for the durable store fold (which - // commits the whole batch + cursor atomically on flush). - // Done before `parse` consumes `u` by reference, and only - // when a store is present so the no-persistence path keeps - // its zero-copy cost. - if store.is_some() { - raw_batch.push(u.clone()); - } - - if let Some(parsed) = parse(&u) { - batch.push(parsed); - } - - // Arm the window on the first received update of a batch - // — even a parse-rejected one, so the cursor advances - // within `window` even through a run of irrelevant keys. - // Reset the pinned timer to the new deadline rather than - // allocating a fresh `Sleep`. - if batch_deadline.is_none() { - let deadline = tokio::time::Instant::now() + config.window; - sleep.as_mut().reset(deadline); - batch_deadline = Some(deadline); - } - - // Flush on a full parsed batch, or — when persisting — a - // full raw batch, so a window packed with parse-rejected - // updates can't grow `raw_batch` without bound before the - // window elapses. - if batch.len() >= config.max || raw_batch.len() >= config.max { - flush!(); - } - } + Some(u) => ingest!(u), None => { // Stream closed. Flush the remainder, then surface the // watch task's terminal result: a clean end returns the @@ -670,107 +901,314 @@ where } } +/// Does the fold hold any entry? Stops at the first. +fn fold_has_entries(store: &S) -> Result { + let mut found = false; + match store.for_each_in_range("", |_| { + found = true; + // Stop the scan; `found` carries the answer. + Err(SnapshotError::Backend(String::new())) + }) { + Err(e) if !found => Err(e), + _ => Ok(found), + } +} + /// Run the underlying watch for `scope`, resuming from `resume` when it carries -/// a position, with the [`KvError::CursorExpired`] → resync + full-watch -/// fallback. -async fn run_watch( +/// a position, with cursor-expiry repair: an expiry (at resume, or mid-watch +/// from the floor guard) runs the planned repair and then either falls back to +/// the full-scope re-list (Relist / nothing armed) or resumes from the restored +/// artifact's cursor (Restore) — which can itself expire later and go round +/// again, each time only for a strictly newer artifact. +/// +/// `unanchored`: the fold holds data but no cursor (a torn first checkpoint, +/// or a populated store started without one). A re-list alone never removes +/// what it doesn't deliver, so that start is repaired like an expiry at +/// revision 0 before watching. +async fn run_watch( watcher: &dyn KvWatcher, scope: &WatchScope, resume: Option, - resync: Option, + unanchored: bool, + repair: Option>, tx: mpsc::Sender, ) -> Result<(), KvError> { // Resume only when the cursor carries a real position; an absent or `none()` // cursor falls through to a full watch. Binding `cursor` here makes "we have a // resume position" structural — there is no separate bool whose truth a later // edit could let drift from the `Some`. - let resume_cursor = resume.filter(|c| !c.is_none()); - - match scope { - WatchScope::All => { - if let Some(cursor) = resume_cursor { - match watcher.watch_all_from(&cursor, tx.clone()).await { - Err(KvError::CursorExpired) => { - warn!( - "watch cursor expired; resyncing, then falling back to full watch_all" - ); - resync_stale_keys(scope, &resync).await?; - watcher.watch_all(tx).await - } - other => other, - } - } else { - watcher.watch_all(tx).await + let mut resume = resume.filter(|c| !c.is_none()); + + // A cursor-less start that must be repaired before it watches: a fold + // with data and no cursor (any repair), or an empty fold on a bucket that + // already evicted current values with a restore armed (its re-list would + // seed an incomplete fold). + let mut repair_first = resume.is_none() + && match &repair { + // (A handle exists only for a real repair mode.) + Some(_) if unanchored => { + warn!( + "the fold holds data but no cursor; repairing it like an expired cursor \ + before watching (a re-list alone never removes what it doesn't deliver)" + ); + true } - } - WatchScope::Prefix(prefix) => { - if let Some(cursor) = resume_cursor { - match watcher.watch_prefix_from(prefix, &cursor, tx.clone()).await { - Err(KvError::CursorExpired) => { - warn!( - "watch cursor expired; resyncing, then falling back to full watch_prefix" - ); - resync_stale_keys(scope, &resync).await?; - watcher.watch_prefix(prefix, tx).await - } - other => other, + Some(h) => { + let seed = fresh_start_needs_restore(watcher, h).await?; + if seed { + warn!( + "no resume cursor, and the bucket has already evicted current values: a \ + re-list would seed an incomplete fold; restoring from the latest artifact" + ); } - } else { - watcher.watch_prefix(prefix, tx).await + seed + } + None => false, + }; + + loop { + let cursor = if std::mem::take(&mut repair_first) { + WatchCursor::none() + } else { + let Some(cursor) = resume.take() else { + return watch_scope(watcher, scope, tx).await; + }; + match watch_scope_from(watcher, scope, &cursor, tx.clone()).await { + Err(KvError::CursorExpired) => cursor, + other => return other, + } + }; + match plan_repair(watcher, repair.as_ref()).await? { + Plan::ReListOnly => { + warn!( + "watch cursor expired with no repair armed (needs a store and a reader or \ + restore source); falling back to the re-list alone — keys deleted during \ + the gap may persist in the fold" + ); + return watch_scope(watcher, scope, tx).await; + } + Plan::Relist(reader, repairs) => { + warn!( + "watch cursor expired; resyncing stale keys, then falling back to the full re-list" + ); + resync_stale_keys(scope, reader, repairs).await?; + return watch_scope(watcher, scope, tx).await; + } + Plan::Restore(reader, source, repairs) => { + warn!("watch cursor expired; restoring the fold from the latest artifact"); + let restored = + restore_from_artifact(watcher, scope, reader, source, repairs, &cursor) + .await + .map_err(|e| match e { + KvError::WatchError(msg) if cursor.is_none() => { + KvError::WatchError(format!( + "{msg}. This node has no cursor, so it can only seed from an \ + artifact. If none exists yet (first deploy onto a bucket that \ + already evicted values), start one node with ExpiryRepair::None \ + — accepting that values which already aged out are gone — and \ + publish an export from it" + )) + } + other => other, + })?; + resume = Some(restored); } } + } +} + +/// The full-scope (state-sync re-list) watch for `scope`. +async fn watch_scope( + watcher: &dyn KvWatcher, + scope: &WatchScope, + tx: mpsc::Sender, +) -> Result<(), KvError> { + match scope { + WatchScope::All => watcher.watch_all(tx).await, + WatchScope::Prefix(prefix) => watcher.watch_prefix(prefix, tx).await, WatchScope::Prefixes(prefixes) => { let refs: Vec<&str> = prefixes.iter().map(String::as_str).collect(); - if let Some(cursor) = resume_cursor { - match watcher - .watch_prefixes_from(&refs, &cursor, tx.clone()) - .await - { - Err(KvError::CursorExpired) => { - warn!( - "watch cursor expired; resyncing, then falling back to full watch_prefixes" - ); - resync_stale_keys(scope, &resync).await?; - watcher.watch_prefixes(&refs, tx).await - } - other => other, - } - } else { - watcher.watch_prefixes(&refs, tx).await + watcher.watch_prefixes(&refs, tx).await + } + } +} + +/// The delta watch for `scope`, resuming after `cursor`. +async fn watch_scope_from( + watcher: &dyn KvWatcher, + scope: &WatchScope, + cursor: &WatchCursor, + tx: mpsc::Sender, +) -> Result<(), KvError> { + match scope { + WatchScope::All => watcher.watch_all_from(cursor, tx).await, + WatchScope::Prefix(prefix) => watcher.watch_prefix_from(prefix, cursor, tx).await, + WatchScope::Prefixes(prefixes) => { + let refs: Vec<&str> = prefixes.iter().map(String::as_str).collect(); + watcher.watch_prefixes_from(&refs, cursor, tx).await + } + } +} + +/// Is the bucket's key listing the truth — does "not listed" mean "deleted"? +/// Yes when retention never evicts current values, and also when it can but +/// has never evicted anything (the first retained revision is still 1). Read +/// live; `None` when the backend can't say. +fn listing_is_truth(r: &Retention) -> bool { + !r.evicts_current_values || resume_window_ok(0, r.first_revision) +} + +/// Decide how to repair one expiry, reading the bucket's retention live when +/// the choice depends on it. +async fn plan_repair<'a, S>( + watcher: &dyn KvWatcher, + repair: Option<&'a RepairHandle>, +) -> Result, KvError> { + let Some(h) = repair else { + return Ok(Plan::ReListOnly); + }; + Ok(match &h.mode { + ExpiryRepair::None => Plan::ReListOnly, + ExpiryRepair::Relist(reader) => { + if watcher + .retention() + .await? + .is_some_and(|r| !listing_is_truth(&r)) + { + let msg = "watch cursor expired on a bucket whose retention evicts current \ + values (max_age, per-message TTL, or discard:old): its key listing \ + can't tell a deleted key from an aged-out one, so the key-listing \ + resync would delete valid keys. Refusing. Repair from artifacts with \ + ExpiryRepair::Restore or ExpiryRepair::Auto (or accept a re-list-only \ + fallback with ExpiryRepair::None)"; + error!(msg); + return Err(KvError::WatchError(msg.into())); } + Plan::Relist(reader, &h.tx) } + ExpiryRepair::Restore { reader, restore } => Plan::Restore(reader, restore, &h.tx), + ExpiryRepair::Auto { reader, restore } => match watcher.retention().await? { + Some(r) if listing_is_truth(&r) => Plan::Relist(reader, &h.tx), + _ => Plan::Restore(reader, restore, &h.tx), + }, + }) +} + +/// Must an EMPTY cursor-less start seed from an artifact? Only with a restore +/// armed, on a bucket whose re-list is no longer complete (it evicted current +/// values). A bucket whose listing is the truth re-lists completely. +async fn fresh_start_needs_restore( + watcher: &dyn KvWatcher, + h: &RepairHandle, +) -> Result { + if !matches!( + h.mode, + ExpiryRepair::Restore { .. } | ExpiryRepair::Auto { .. } + ) { + return Ok(false); + } + Ok(watcher + .retention() + .await? + .is_some_and(|r| !listing_is_truth(&r))) +} + +/// The artifact restore, watch-task half: check the newest artifact before +/// downloading it (ahead of `local`, covers the scope, still inside +/// retention), download it, check what actually arrived, then hand it to the +/// main loop and wait for the cursor it restored to. +async fn restore_from_artifact( + watcher: &dyn KvWatcher, + scope: &WatchScope, + reader: &Arc, + source: &Arc>, + repairs: &mpsc::Sender>, + local: &WatchCursor, +) -> Result { + let prefixes = scope.prefixes(); + let fail = |msg: String| { + error!(%msg, "cursor-expiry restore refused; no safe recovery"); + KvError::WatchError(format!("cursor-expired restore: {msg}")) + }; + + let first = watcher.retention().await?.map(|r| r.first_revision); + let latest = source + .latest() + .await + .map_err(|e| fail(format!("reading the latest artifact failed: {e}")))?; + check_restore(&latest, local, first, &prefixes).map_err(fail)?; + + let restored = source + .fetch() + .await + .map_err(|e| fail(format!("fetching the latest artifact failed: {e}")))?; + // The pointer may have advanced between the peek and the download, and + // retention may have moved: check what actually arrived, against now. + let first = watcher.retention().await?.map(|r| r.first_revision); + check_restore(restored.manifest(), local, first, &prefixes).map_err(fail)?; + let age_secs = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|now| { + now.as_secs() + .saturating_sub(restored.manifest().created_at_unix) + }) + .unwrap_or(0); + info!( + cursor = ?restored.manifest().cursor, + age_secs, + first_retained = ?first, + "restoring the fold from the latest artifact" + ); + + // The bucket's live keys, listed after the artifact's cursor is fixed: + // the restore never deletes one of these (see `restore_diff`). A failed + // listing fails the watch, like the key-listing repair's. + let mut live = std::collections::HashSet::new(); + for prefix in &prefixes { + let keys = reader + .keys(prefix) + .await + .map_err(|e| fail(format!("listing live keys under {prefix:?} failed: {e}")))?; + live.extend(keys); } + + let (reply_tx, reply_rx) = oneshot::channel(); + repairs + .send(RepairRequest::Restore { + restored, + live, + reply: reply_tx, + }) + .await + .map_err(|_| KvError::WatchError("watch loop ended during a restore".into()))?; + reply_rx + .await + .map_err(|_| KvError::WatchError("watch loop dropped a restore reply".into()))? } -/// Cursor-expired stale-key resync, run BEFORE the fallback watch is -/// established: list the scope's live keys, hand them to the main loop (which -/// diffs them against the fold and applies synthetic deletes), and wait for the -/// ack. That ordering — deletes applied, then fallback watch armed — is what -/// makes a delete-then-recreate during the gap converge: the synthetic delete -/// always lands before the re-list put. +/// Cursor-expired stale-key resync (the key-listing diff), run BEFORE the +/// fallback watch is established: list the scope's live keys, hand them to the +/// main loop (which diffs them against the fold and applies synthetic +/// deletes), and wait for the ack. That ordering — deletes applied, then +/// fallback watch armed — is what makes a delete-then-recreate during the gap +/// converge: the synthetic delete always lands before the re-list put. /// -/// With no reader/store wired (`resync` is `None`) the caller explicitly opted -/// out: warn and fall back re-list-only (keys deleted during the gap stay in -/// the fold — `tests/model.rs` pins this divergence as reachable). +/// Sound only on a bucket that never evicts current values; `plan_repair` +/// refuses it otherwise. /// -/// A FAILED listing, by contrast, is **fatal** — it fails the watch rather -/// than degrading. The resync is load-bearing for the "stale, never corrupt" -/// convergence guarantee: a silently degraded resync leaves the fold holding -/// keys the bucket deleted, with one warn line as the only witness -/// (`tests/model.rs` proves this divergence reachable under degrade -/// semantics). Failing the watch turns the violated guarantee into a visible -/// error; the caller's restart re-resumes, hits `CursorExpired` again, and -/// retries the resync from scratch. -async fn resync_stale_keys( +/// A FAILED listing is **fatal** — it fails the watch rather than degrading. +/// The resync is load-bearing for the "stale, never corrupt" convergence +/// guarantee: a silently degraded resync leaves the fold holding keys the +/// bucket deleted, with one warn line as the only witness (`tests/model.rs` +/// proves this divergence reachable under degrade semantics). Failing the +/// watch turns the violated guarantee into a visible error; the caller's +/// restart re-resumes, hits `CursorExpired` again, and retries the resync from +/// scratch. +async fn resync_stale_keys( scope: &WatchScope, - resync: &Option, + reader: &Arc, + repairs: &mpsc::Sender>, ) -> Result<(), KvError> { - let Some((reader, resync_tx)) = resync else { - warn!( - "no reader wired for cursor-expired resync; keys deleted during the gap may persist in the fold" - ); - return Ok(()); - }; let mut live_keys = Vec::new(); for prefix in scope.prefixes() { match reader.keys(&prefix).await { @@ -784,8 +1222,8 @@ async fn resync_stale_keys( } } let (ack_tx, ack_rx) = oneshot::channel(); - if resync_tx - .send(ResyncRequest { + if repairs + .send(RepairRequest::Relist { live_keys, ack: ack_tx, }) @@ -2375,4 +2813,774 @@ mod tests { "{err}" ); } + + // --- Cursor-expiry repair: evicting buckets, artifact restore ----------- + + use crate::kv::Retention; + use crate::repair::{ExpiryRepair, RestoreSource, RestoredFold}; + use std::collections::VecDeque; + + /// A watcher scripted per `*_from` call, reporting configurable retention. + /// Each step delivers its updates and then either ends in `CursorExpired` + /// (a resume-time expiry when the step is empty, a mid-watch floor-guard + /// trip when it isn't) or returns cleanly, closing the stream. Records the + /// cursor every `*_from` call resumed from. + struct ScriptedWatcher { + from_steps: Mutex, bool)>>, + full: Mutex>>, + retention: Option, + resumed_from: Mutex>, + } + + impl ScriptedWatcher { + fn new(retention: Option, steps: Vec<(Vec, bool)>) -> Self { + Self { + from_steps: Mutex::new(steps.into()), + full: Mutex::new(Some(vec![])), + retention, + resumed_from: Mutex::new(vec![]), + } + } + + async fn step(&self, cursor: &WatchCursor, tx: Sender) -> Result<(), KvError> { + self.resumed_from + .lock() + .unwrap() + .push(cursor.as_u64().unwrap_or(0)); + let step = self.from_steps.lock().unwrap().pop_front(); + let Some((updates, expire)) = step else { + return Ok(()); + }; + for u in updates { + if tx.send(u).await.is_err() { + return Ok(()); + } + } + if expire { + Err(KvError::CursorExpired) + } else { + Ok(()) + } + } + + async fn full(&self, tx: Sender) -> Result<(), KvError> { + let updates = self.full.lock().unwrap().take().unwrap_or_default(); + for u in updates { + let _ = tx.send(u).await; + } + Ok(()) + } + } + + #[async_trait] + impl KvWatcher for ScriptedWatcher { + async fn watch_all(&self, tx: Sender) -> Result<(), KvError> { + self.full(tx).await + } + async fn watch_prefix(&self, _p: &str, tx: Sender) -> Result<(), KvError> { + self.full(tx).await + } + async fn watch_prefixes(&self, _p: &[&str], tx: Sender) -> Result<(), KvError> { + self.full(tx).await + } + async fn watch_all_from( + &self, + cursor: &WatchCursor, + tx: Sender, + ) -> Result<(), KvError> { + self.step(cursor, tx).await + } + async fn watch_prefix_from( + &self, + _p: &str, + cursor: &WatchCursor, + tx: Sender, + ) -> Result<(), KvError> { + self.step(cursor, tx).await + } + async fn watch_prefixes_from( + &self, + _p: &[&str], + cursor: &WatchCursor, + tx: Sender, + ) -> Result<(), KvError> { + self.step(cursor, tx).await + } + async fn retention(&self) -> Result, KvError> { + Ok(self.retention) + } + } + + /// `ExpiryRepair::Restore` with a reader that lists no live keys (every + /// key the artifact lacks is really gone). + fn restore_mode(artifact: Arc>) -> ExpiryRepair { + ExpiryRepair::Restore { + reader: Arc::new(MockReader { live: vec![] }), + restore: artifact, + } + } + + fn evicting(first_revision: u64) -> Option { + Some(Retention { + evicts_current_values: true, + first_revision, + }) + } + + /// A restore source serving one artifact: a fold built from `updates` at + /// `cursor`, recorded under `scope`. Counts downloads. + struct MockRestore { + dir: tempfile::TempDir, + updates: Vec, + cursor: u64, + scope: Option>, + fetches: AtomicU64, + } + + impl MockRestore { + fn new(updates: Vec, cursor: u64, scope: Option<&[&str]>) -> Arc { + Arc::new(Self { + dir: tempfile::TempDir::new().unwrap(), + updates, + cursor, + scope: scope.map(|s| s.iter().map(|p| p.to_string()).collect()), + fetches: AtomicU64::new(0), + }) + } + + fn manifest(&self) -> ExportManifest { + ExportManifest { + schema_version: crate::ARTIFACT_SCHEMA_VERSION, + backend: "append-log".into(), + backend_version: "2".into(), + cursor: WatchCursor::from_u64(self.cursor), + created_at_unix: 0, + files: vec![], + scope: self.scope.clone(), + } + } + } + + #[async_trait] + impl RestoreSource for MockRestore { + async fn latest(&self) -> Result { + Ok(self.manifest()) + } + async fn fetch(&self) -> Result, SnapshotError> { + let n = self.fetches.fetch_add(1, Ordering::SeqCst); + let path = self.dir.path().join(format!("artifact-{n}.snap")); + let (_c, mut fold) = AppendLogSnapshot::open(&path, u64::MAX)?; + fold.apply(&self.updates, &WatchCursor::from_u64(self.cursor))?; + Ok(RestoredFold::new(self.manifest(), fold)) + } + } + + /// The local fold every restore test starts from: three keys at cursor 4. + fn expired_fold(dir: &tempfile::TempDir) -> (std::path::PathBuf, AppendLogSnapshot) { + let path = dir.path().join("fold.snap"); + let (_r, mut store) = AppendLogSnapshot::open(&path, u64::MAX).unwrap(); + store + .apply( + &[ + put("node.keep", b"v1", 1), + put("node.changed", b"old", 2), + put("node.gone", b"x", 3), + put("node.a", b"a", 4), + ], + &WatchCursor::from_u64(4), + ) + .unwrap(); + (path, store) + } + + /// What one watch run did: every update `parse` saw, as (key, is_delete), + /// every cursor `on_applied` reported, and the watch's result. + struct Run { + seen: Vec<(String, bool)>, + cursors: Vec, + result: Result, + } + + async fn run_repair( + watcher: Arc, + scope: WatchScope, + resume: Option, + repair: ExpiryRepair, + store: AppendLogSnapshot, + max: usize, + ) -> Run { + let seen = Arc::new(Mutex::new(Vec::new())); + let cursors = Arc::new(Mutex::new(Vec::new())); + let (s, c) = (Arc::clone(&seen), Arc::clone(&cursors)); + let (_sd_tx, sd_rx) = watch::channel(false); + let result = watch_applied( + watcher, + scope, + resume.map(WatchCursor::from_u64), + repair, + Some(store), + None, + BatchConfig { + max, + ..BatchConfig::default() + }, + move |u: &KvUpdate| { + s.lock() + .unwrap() + .push((u.key().to_string(), matches!(u, KvUpdate::Delete { .. }))); + Some(()) + }, + |_batch: Vec<()>| {}, + move |cur: WatchCursor| c.lock().unwrap().push(cur.as_u64().unwrap()), + sd_rx, + ) + .await; + let seen = seen.lock().unwrap().clone(); + let cursors = cursors.lock().unwrap().clone(); + Run { + seen, + cursors, + result, + } + } + + /// THE FIX, end to end against the store: on a bucket whose retention + /// evicts current values, an expired cursor is repaired from the artifact, + /// not from the bucket's key listing. `Auto` picks the restore; the reader + /// it also carries lists NOTHING (every key aged out), so a relist would + /// have deleted the whole fold. The in-scope fold becomes the artifact's + /// (changed value with its real revision, the gap write, the real delete), + /// identical entries are not re-applied, chunked across `max`, the cursor + /// moves to the artifact's only after the last chunk, and the watch + /// resumes from there. + #[tokio::test] + async fn auto_restores_from_artifact_on_evicting_bucket() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = expired_fold(&dir); + let artifact = MockRestore::new( + vec![ + put("node.keep", b"v1", 1), + put("node.changed", b"new", 7), + put("node.a", b"a", 4), + put("node.late", b"gap-write", 8), + ], + 10, + Some(&[""]), + ); + let watcher = Arc::new(ScriptedWatcher::new( + evicting(9), + vec![(vec![], true), (vec![put("node.tail", b"t", 11)], false)], + )); + let run = run_repair( + Arc::clone(&watcher), + WatchScope::All, + Some(4), + ExpiryRepair::Auto { + reader: Arc::new(MockReader { live: vec![] }), + restore: artifact.clone(), + }, + store, + 2, + ) + .await; + + assert_eq!(run.result.unwrap().as_u64(), Some(11)); + assert_eq!( + run.seen, + vec![ + ("node.changed".into(), false), + ("node.gone".into(), true), + ("node.late".into(), false), + ("node.tail".into(), false), + ], + "the diff (key order), then the delta from the artifact's cursor" + ); + assert_eq!( + run.cursors, + vec![10, 11], + "cursor reaches the artifact's after the diff" + ); + assert_eq!(*watcher.resumed_from.lock().unwrap(), vec![4, 10]); + assert_eq!(artifact.fetches.load(Ordering::SeqCst), 1); + + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert_eq!(snap.cursor.as_u64(), Some(11)); + let keys: std::collections::BTreeSet<&str> = + snap.entries.keys().map(String::as_str).collect(); + assert_eq!( + keys, + [ + "node.a", + "node.changed", + "node.keep", + "node.late", + "node.tail" + ] + .into(), + "aged-out keys kept, gap write recovered, real delete applied" + ); + let changed = &snap.entries["node.changed"]; + assert_eq!(changed.value, b"new"); + assert_eq!( + changed.version.as_u64(), + Some(7), + "restored entries keep their revision" + ); + } + + /// REGRESSION PIN for the reported bug: the key-listing resync on a + /// bucket that evicts current values. The listing is empty because every + /// key aged out; pre-fix, the resync deleted the whole fold. Now the + /// legacy `Some(reader)` argument (which means Relist) is refused and the + /// fold is left exactly as it was. + #[tokio::test] + async fn relist_refused_on_evicting_bucket() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = expired_fold(&dir); + let watcher = Arc::new(ScriptedWatcher::new(evicting(9), vec![(vec![], true)])); + let legacy: Option> = Some(Arc::new(MockReader { live: vec![] })); + let run = run_repair(watcher, WatchScope::All, Some(4), legacy.into(), store, 100).await; + + let err = run + .result + .expect_err("relist on an evicting bucket must be refused"); + assert!(err.to_string().contains("evicts current values"), "{err}"); + assert!(run.seen.is_empty(), "no synthetic deletes reached apply"); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert_eq!(snap.entries.len(), 4, "the fold is untouched"); + assert_eq!(snap.cursor.as_u64(), Some(4)); + } + + /// `Auto` on a bucket that never evicts current values keeps the + /// key-listing diff (absence there does mean deleted) and never touches + /// the artifact store. + #[tokio::test] + async fn auto_relists_when_bucket_never_evicts() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = expired_fold(&dir); + let artifact = MockRestore::new(vec![], 10, Some(&[""])); + let watcher = Arc::new(ScriptedWatcher::new( + Some(Retention { + evicts_current_values: false, + first_revision: 9, + }), + vec![(vec![], true)], + )); + let live = ["node.keep", "node.changed", "node.a"] + .map(String::from) + .to_vec(); + let run = run_repair( + watcher, + WatchScope::All, + Some(4), + ExpiryRepair::Auto { + reader: Arc::new(MockReader { live }), + restore: artifact.clone(), + }, + store, + 100, + ) + .await; + + run.result.unwrap(); + assert_eq!(run.seen, vec![("node.gone".into(), true)]); + assert_eq!( + artifact.fetches.load(Ordering::SeqCst), + 0, + "no artifact needed" + ); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert!(!snap.entries.contains_key("node.gone")); + assert_eq!(snap.entries.len(), 3); + } + + /// No safe recovery from a stale artifact: its cursor is outside the + /// log's retention window, so the evictions between them exist nowhere. + /// Refused from the manifest alone (never downloaded); the fold is + /// untouched and the watch fails loudly. + #[tokio::test] + async fn restore_refuses_stale_artifact() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = expired_fold(&dir); + let artifact = MockRestore::new(vec![put("node.keep", b"v1", 1)], 6, Some(&[""])); + let watcher = Arc::new(ScriptedWatcher::new(evicting(9), vec![(vec![], true)])); + let run = run_repair( + watcher, + WatchScope::All, + Some(4), + restore_mode(artifact.clone()), + store, + 100, + ) + .await; + + let err = run + .result + .expect_err("a stale artifact must fail the watch"); + assert!(err.to_string().contains("retention window"), "{err}"); + assert_eq!( + artifact.fetches.load(Ordering::SeqCst), + 0, + "refused before download" + ); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert_eq!((snap.entries.len(), snap.cursor.as_u64()), (4, Some(4))); + } + + /// An artifact must cover the restoring watcher's scope — and an + /// unscoped (pre-scope / direct `export_to`) artifact can't be shown to. + #[tokio::test] + async fn restore_refuses_artifact_not_covering_scope() { + for (scope, why) in [ + (Some(&["edge."][..]), "does not cover"), + (None, "no key scope"), + ] { + let dir = tempfile::TempDir::new().unwrap(); + let (_path, store) = expired_fold(&dir); + let artifact = MockRestore::new(vec![], 10, scope); + let watcher = Arc::new(ScriptedWatcher::new(evicting(9), vec![(vec![], true)])); + let run = run_repair( + watcher, + WatchScope::Prefix("node.".into()), + Some(4), + restore_mode(artifact), + store, + 100, + ) + .await; + let err = run + .result + .expect_err("an uncovering artifact must be refused"); + assert!(err.to_string().contains(why), "{err}"); + } + } + + /// A prefix-scoped restore replaces only the in-scope fold: an + /// out-of-scope key in the local fold survives, and an out-of-scope key in + /// a wider (All-scope) artifact is not imported. + #[tokio::test] + async fn restore_respects_watch_scope() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, mut store) = expired_fold(&dir); + store + .apply(&[put("other.local", b"mine", 5)], &WatchCursor::from_u64(5)) + .unwrap(); + let artifact = MockRestore::new( + vec![ + put("node.keep", b"v1", 1), + put("other.remote", b"theirs", 6), + ], + 10, + Some(&[""]), + ); + let watcher = Arc::new(ScriptedWatcher::new(evicting(9), vec![(vec![], true)])); + let run = run_repair( + watcher, + WatchScope::Prefix("node.".into()), + Some(5), + restore_mode(artifact), + store, + 100, + ) + .await; + run.result.unwrap(); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + let keys: std::collections::BTreeSet<&str> = + snap.entries.keys().map(String::as_str).collect(); + assert_eq!(keys, ["node.keep", "other.local"].into()); + } + + /// THE FLOOR-GUARD ROUTE: a live watch delivers past its resume cursor, + /// then retention overruns it (the NATS floor guard ends the stream with + /// `CursorExpired`). The delivered backlog is folded BEFORE the restore — + /// never after it, where a stale put would resurrect old state and drag + /// the cursor backward — and the restore diffs against that backlog. + #[tokio::test] + async fn floor_guard_trip_folds_backlog_then_restores() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = expired_fold(&dir); + let artifact = MockRestore::new( + vec![ + put("node.keep", b"v1", 1), + put("node.changed", b"old", 2), + put("node.a", b"a2", 5), + put("node.b", b"b-newer", 9), + ], + 10, + Some(&[""]), + ); + let watcher = Arc::new(ScriptedWatcher::new( + evicting(9), + vec![ + ( + vec![put("node.a", b"a2", 5), put("node.b", b"b-older", 6)], + true, + ), + (vec![put("node.c", b"c", 12)], false), + ], + )); + let run = run_repair( + Arc::clone(&watcher), + WatchScope::All, + Some(4), + restore_mode(artifact), + store, + 100, + ) + .await; + + assert_eq!(run.result.unwrap().as_u64(), Some(12)); + assert_eq!( + run.seen, + vec![ + ("node.a".into(), false), + ("node.b".into(), false), + ("node.b".into(), false), + ("node.gone".into(), true), + ("node.c".into(), false), + ], + "backlog (a, b-older), then the diff (b-newer, gone), then the delta" + ); + assert!( + run.cursors.windows(2).all(|w| w[0] <= w[1]), + "the cursor never moves backward: {:?}", + run.cursors + ); + assert_eq!(*watcher.resumed_from.lock().unwrap(), vec![4, 10]); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert_eq!(snap.entries["node.b"].value, b"b-newer"); + assert_eq!(snap.cursor.as_u64(), Some(12)); + } + + /// The main loop's ahead check is the authoritative one: a mid-watch + /// expiry can deliver past the artifact the watch task accepted against + /// its (older) resume cursor. Restoring it would move the fold backward, + /// so it's refused and the delivered fold kept. + #[tokio::test] + async fn restore_behind_the_delivered_frontier_is_refused() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = expired_fold(&dir); + let artifact = MockRestore::new(vec![put("node.keep", b"v1", 1)], 10, Some(&[""])); + let watcher = Arc::new(ScriptedWatcher::new( + evicting(9), + vec![(vec![put("node.z", b"z", 12)], true)], + )); + let run = run_repair( + watcher, + WatchScope::All, + Some(4), + restore_mode(artifact), + store, + 100, + ) + .await; + let err = run + .result + .expect_err("an artifact behind the fold must be refused"); + assert!(err.to_string().contains("not ahead"), "{err}"); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert_eq!(snap.cursor.as_u64(), Some(12)); + assert!( + snap.entries.contains_key("node.gone"), + "fold not rolled back" + ); + } + + /// A cursor-less start on a bucket that has already evicted current + /// values can't seed a complete fold from the re-list; with a restore + /// armed it seeds from the artifact and resumes from its cursor instead + /// of re-listing. + #[tokio::test] + async fn fresh_start_seeds_from_artifact_when_bucket_already_evicted() { + let dir = tempfile::TempDir::new().unwrap(); + let (_r, store) = + AppendLogSnapshot::open(&dir.path().join("fresh.snap"), u64::MAX).unwrap(); + let artifact = MockRestore::new(vec![put("node.old", b"aged-out", 2)], 7, Some(&[""])); + let watcher = + ScriptedWatcher::new(evicting(5), vec![(vec![put("node.new", b"n", 8)], false)]); + *watcher.full.lock().unwrap() = Some(vec![put("RELIST", b"must not run", 99)]); + let watcher = Arc::new(watcher); + let run = run_repair( + Arc::clone(&watcher), + WatchScope::All, + None, + restore_mode(artifact), + store, + 100, + ) + .await; + assert_eq!(run.result.unwrap().as_u64(), Some(8)); + assert_eq!( + run.seen, + vec![("node.old".into(), false), ("node.new".into(), false)] + ); + assert_eq!(*watcher.resumed_from.lock().unwrap(), vec![7]); + } + + /// Every artifact a watch exports records its scope (schema 2), so a + /// restore can check coverage. + #[tokio::test(start_paused = true)] + async fn export_records_watch_scope() { + let dir = tempfile::TempDir::new().unwrap(); + let (_r, store) = AppendLogSnapshot::open(&dir.path().join("fold.snap"), u64::MAX).unwrap(); + let watcher = Arc::new(MockWatcher::new(vec![put("node.a", b"1", 1)], true)); + let (sd_tx, sd_rx) = watch::channel(false); + let (ex_tx, ex_rx) = mpsc::channel(1); + let task = tokio::spawn(watch_applied( + watcher, + WatchScope::Prefixes(vec!["node.".into(), "edge.".into()]), + None, + None, + Some(store), + Some(ex_rx), + BatchConfig::default(), + parse_put, + |_b: Vec>| {}, + |_| {}, + sd_rx, + )); + tokio::time::sleep(Duration::from_millis(50)).await; + let artifact = dir.path().join("artifact"); + let (reply_tx, reply_rx) = tokio::sync::oneshot::channel(); + ex_tx + .send(ExportRequest { + dest_dir: artifact.clone(), + reply: reply_tx, + }) + .await + .unwrap(); + let manifest = reply_rx.await.unwrap().unwrap(); + let want = Some(vec!["node.".to_string(), "edge.".to_string()]); + assert_eq!(manifest.scope, want); + assert_eq!(manifest.schema_version, crate::ARTIFACT_SCHEMA_VERSION); + assert_eq!( + crate::artifact::read_manifest(&artifact).unwrap().scope, + want + ); + sd_tx.send(true).unwrap(); + task.await.unwrap().unwrap(); + } + + #[async_trait] + impl RestoreSource for MockRestore { + async fn latest(&self) -> Result { + Ok(self.manifest()) + } + async fn fetch(&self) -> Result, SnapshotError> { + let restored = RestoreSource::::fetch(self).await?; + let manifest = restored.manifest().clone(); + let path = self.dir.path().join(format!( + "wrapped-{}.snap", + self.fetches.load(Ordering::SeqCst) + )); + drop(restored); + let (_c, mut inner) = AppendLogSnapshot::open(&path, u64::MAX)?; + inner.apply(&self.updates, &WatchCursor::from_u64(self.cursor))?; + Ok(RestoredFold::new( + manifest, + FailOnceStore { + inner, + failed: std::sync::atomic::AtomicBool::new(true), + }, + )) + } + } + + /// A fold at cursor 4 whose FIRST store apply will fail transiently. + fn fail_once_fold(dir: &tempfile::TempDir) -> (std::path::PathBuf, FailOnceStore) { + let (path, inner) = expired_fold(dir); + ( + path, + FailOnceStore { + inner, + failed: std::sync::atomic::AtomicBool::new(false), + }, + ) + } + + async fn run_fail_once( + watcher: Arc, + repair: ExpiryRepair, + store: FailOnceStore, + ) -> Result { + let (_sd_tx, sd_rx) = watch::channel(false); + watch_applied( + watcher, + WatchScope::All, + Some(WatchCursor::from_u64(4)), + repair, + Some(store), + None, + BatchConfig { + window: Duration::from_secs(3600), + ..BatchConfig::default() + }, + |u: &KvUpdate| Some(u.key().to_string()), + |_b: Vec| {}, + |_| {}, + sd_rx, + ) + .await + } + + /// A repair diffs the STORE, so the store must hold everything delivered + /// before it. Here the pre-repair flush's store apply fails transiently + /// and re-queues `node.new@5`; a diff against the store would not see it, + /// no synthetic delete would be made, and the re-queued put would then + /// resurrect a key that was deleted during the gap (marker evicted, so the + /// listing lacks it). + #[tokio::test] + async fn relist_diff_sees_updates_requeued_by_a_failed_flush() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = fail_once_fold(&dir); + let watcher = Arc::new(ScriptedWatcher::new( + Some(Retention { + evicts_current_values: false, + first_revision: 9, + }), + vec![(vec![put("node.new", b"recreated-then-deleted", 5)], true)], + )); + let live = ["node.keep", "node.changed", "node.gone", "node.a"] + .map(String::from) + .to_vec(); + let repair: Option> = Some(Arc::new(MockReader { live })); + run_fail_once(watcher, repair.into(), store).await.unwrap(); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert!( + !snap.entries.contains_key("node.new"), + "a key gone from the bucket must not be resurrected by a re-queued put" + ); + } + + /// The restore twin: the artifact (cursor 10) lacks `node.new` (deleted at + /// 8), so the restore must delete the `node.new@5` the failed flush + /// re-queued — which it can only do if the diff sees it. + #[tokio::test] + async fn restore_diff_sees_updates_requeued_by_a_failed_flush() { + let dir = tempfile::TempDir::new().unwrap(); + let (path, store) = fail_once_fold(&dir); + let artifact = MockRestore::new( + vec![ + put("node.keep", b"v1", 1), + put("node.changed", b"old", 2), + put("node.gone", b"x", 3), + put("node.a", b"a", 4), + ], + 10, + Some(&[""]), + ); + let watcher = Arc::new(ScriptedWatcher::new( + evicting(9), + vec![(vec![put("node.new", b"deleted-at-8", 5)], true)], + )); + run_fail_once(watcher, restore_mode(artifact), store) + .await + .unwrap(); + let snap = crate::snapshot::load(&path).unwrap().unwrap(); + assert_eq!(snap.cursor.as_u64(), Some(10)); + assert!( + !snap.entries.contains_key("node.new"), + "the restored fold must equal the artifact's" + ); + } } diff --git a/src/artifact.rs b/src/artifact.rs index 06ec4ac..f5e549b 100644 --- a/src/artifact.rs +++ b/src/artifact.rs @@ -45,7 +45,17 @@ use crate::snapshot::SnapshotError; /// Version of the artifact layout itself (`MANIFEST.json` schema + `data/` /// payload convention). Bumped only when the artifact shape changes; the /// *payload* format is governed separately by [`ExportManifest::backend_version`]. -pub const ARTIFACT_SCHEMA_VERSION: u32 = 1; +/// +/// This is the newest layout this build writes and reads: schema 2 adds the +/// exporter's key [`scope`](ExportManifest::scope). Schema 1 (no scope) is +/// still read, and still written for artifacts exported outside +/// [`watch_applied`](crate::watch_applied), which has no scope to record. +/// Builds older than schema 2 refuse a schema-2 manifest, so upgrade the +/// importing nodes before the exporting ones. +pub const ARTIFACT_SCHEMA_VERSION: u32 = 2; + +/// The pre-scope artifact layout: identical to schema 2 minus `scope`. +pub(crate) const SCHEMA_UNSCOPED: u32 = 1; /// Manifest file name at the artifact root. Written last and fsynced, so its /// presence (after the atomic rename) means the artifact is complete. @@ -78,6 +88,13 @@ pub struct ExportManifest { /// Every payload file, with size and BLAKE3 digest. Import verifies all of /// them and rejects undeclared extras. pub files: Vec, + /// The key scope the exporting fold covered, as key prefixes (`[""]` is + /// every key). [`watch_applied`](crate::watch_applied) records its + /// [`WatchScope`](crate::WatchScope) on every export it serves. `None` for + /// an artifact exported directly through `export_to` or by a pre-scope + /// build (schema 1): its coverage is unknown, so the automatic expiry + /// restore refuses it. + pub scope: Option>, } /// One payload file in an [`ExportManifest`]. @@ -116,6 +133,9 @@ struct ManifestWire { cursor_hex: String, created_at_unix: u64, files: Vec, + /// Schema 2 only (required there, forbidden in schema 1). + #[serde(default, skip_serializing_if = "Option::is_none")] + scope: Option>, } #[derive(Serialize, Deserialize)] @@ -194,6 +214,7 @@ pub(crate) fn write_manifest( blake3: f.blake3.clone(), }) .collect(), + scope: manifest.scope.clone(), }; let json = serde_json::to_vec_pretty(&wire) .map_err(|e| SnapshotError::Backend(format!("manifest serialization failed: {e}")))?; @@ -228,11 +249,24 @@ pub(crate) fn manifest_from_slice(data: &[u8]) -> Result {} + (SCHEMA_UNSCOPED, Some(_)) => { + return Err(invalid(format!( + "schema_version {SCHEMA_UNSCOPED} manifest carries a scope (introduced in {ARTIFACT_SCHEMA_VERSION})" + ))); + } + (ARTIFACT_SCHEMA_VERSION, None) => { + return Err(invalid(format!( + "schema_version {ARTIFACT_SCHEMA_VERSION} manifest is missing its scope" + ))); + } + (v, _) => { + return Err(invalid(format!( + "unsupported artifact schema_version {v} (this build supports \ + {SCHEMA_UNSCOPED}..={ARTIFACT_SCHEMA_VERSION})" + ))); + } } for f in &wire.files { validate_payload_path(&f.path)?; @@ -253,9 +287,30 @@ pub(crate) fn manifest_from_slice(data: &[u8]) -> Result Result { + let mut manifest = read_manifest(artifact_dir)?; + manifest.schema_version = ARTIFACT_SCHEMA_VERSION; + manifest.scope = Some(scope.to_vec()); + write_manifest(artifact_dir, &manifest)?; + fsync_dir(artifact_dir)?; + Ok(manifest) +} + /// Reject any manifest path that could escape the artifact when joined: it must /// be relative, `/`-separated, contain no `..`/`.` components, and live under /// `data/`. @@ -494,7 +549,9 @@ impl ExportStage { let root = self.dir.path(); let files = hash_payload(root)?; let manifest = ExportManifest { - schema_version: ARTIFACT_SCHEMA_VERSION, + // Unscoped: the backend doesn't know the watch scope. The + // `watch_applied` export path stamps it (`stamp_scope`). + schema_version: SCHEMA_UNSCOPED, backend: backend.to_string(), backend_version: backend_version.to_string(), cursor: cursor.clone(), @@ -503,6 +560,7 @@ impl ExportStage { .map(|d| d.as_secs()) .unwrap_or(0), files, + scope: None, }; write_manifest(root, &manifest)?; fsync_dir(root)?; @@ -660,12 +718,13 @@ mod tests { fn manifest_with(files: Vec, cursor: WatchCursor) -> ExportManifest { ExportManifest { - schema_version: ARTIFACT_SCHEMA_VERSION, + schema_version: SCHEMA_UNSCOPED, backend: "append-log".into(), backend_version: "2".into(), cursor, created_at_unix: 1_765_400_000, files, + scope: None, } } @@ -692,6 +751,56 @@ mod tests { assert_eq!(got.files[0].size, 42); } + /// `stamp_scope` upgrades a sealed (schema 1) manifest to schema 2 with + /// the scope, and the result reads back; schema/scope disagreement in + /// either direction is refused. + #[test] + fn scope_stamp_round_trips_and_schema_must_agree() { + let dir = TempDir::new().unwrap(); + write_manifest(dir.path(), &manifest_with(vec![], WatchCursor::from_u64(9))).unwrap(); + let scope = vec!["node.".to_string(), "edge.".to_string()]; + let stamped = stamp_scope(dir.path(), &scope).unwrap(); + assert_eq!(stamped.schema_version, ARTIFACT_SCHEMA_VERSION); + let got = read_manifest(dir.path()).unwrap(); + assert_eq!(got.scope.as_deref(), Some(&scope[..])); + assert_eq!(got.cursor, WatchCursor::from_u64(9)); + + // Schema 1 with a scope, schema 2 without one: both refused. + write_raw_manifest( + dir.path(), + &wire_json("", "[]", SCHEMA_UNSCOPED).replace('}', r#","scope":[""]}"#), + ); + assert!(matches!( + read_manifest(dir.path()), + Err(SnapshotError::ArtifactInvalid(_)) + )); + write_raw_manifest(dir.path(), &wire_json("", "[]", ARTIFACT_SCHEMA_VERSION)); + assert!(matches!( + read_manifest(dir.path()), + Err(SnapshotError::ArtifactInvalid(_)) + )); + } + + /// The whole schema/scope matrix: a manifest is accepted iff it is + /// schema 1 without a scope or schema 2 with one. + #[test] + fn schema_scope_matrix() { + let dir = TempDir::new().unwrap(); + for schema in 0..=3u32 { + for scoped in [false, true] { + let mut json = wire_json("", "[]", schema); + if scoped { + json = json.replace('}', r#","scope":["node."]}"#); + } + write_raw_manifest(dir.path(), &json); + let accepted = read_manifest(dir.path()).is_ok(); + let want = (schema == SCHEMA_UNSCOPED && !scoped) + || (schema == ARTIFACT_SCHEMA_VERSION && scoped); + assert_eq!(accepted, want, "schema {schema}, scoped {scoped}"); + } + } + } + #[test] fn manifest_round_trips_none_cursor() { let dir = TempDir::new().unwrap(); @@ -728,7 +837,7 @@ mod tests { let dir = TempDir::new().unwrap(); for bad in ["zz", "abc", "0102030405060708090a0b"] { // non-hex, odd length, 11 bytes (> token capacity) - write_raw_manifest(dir.path(), &wire_json(bad, "[]", ARTIFACT_SCHEMA_VERSION)); + write_raw_manifest(dir.path(), &wire_json(bad, "[]", SCHEMA_UNSCOPED)); match read_manifest(dir.path()) { Err(SnapshotError::ArtifactInvalid(_)) => {} other => panic!("cursor_hex {bad:?}: expected ArtifactInvalid, got {other:?}"), @@ -767,7 +876,7 @@ mod tests { r#"[{{"path":"{}","size":0,"blake3":""}}]"#, bad.replace('\\', "\\\\") ); - write_raw_manifest(dir.path(), &wire_json("", &files, ARTIFACT_SCHEMA_VERSION)); + write_raw_manifest(dir.path(), &wire_json("", &files, SCHEMA_UNSCOPED)); match read_manifest(dir.path()) { Err(SnapshotError::ArtifactInvalid(_)) => {} other => panic!("path {bad:?}: expected ArtifactInvalid, got {other:?}"), diff --git a/src/kv.rs b/src/kv.rs index a0a39df..1ee88bd 100644 --- a/src/kv.rs +++ b/src/kv.rs @@ -79,10 +79,32 @@ pub enum KvError { Timeout, /// The watch cursor/revision is too old — the backend has compacted past it. /// Callers should fall back to a full scan + watch. + /// + /// A `*_from` resume can also return this MID-STREAM, after delivering + /// updates: the NATS All-scope resume watch does so when retention overruns + /// the live consumer (the floor guard). Everything delivered before the + /// error is valid; the position after it is expired. #[error("watch cursor expired (compacted)")] CursorExpired, } +/// What a backend's retention can do to the log a watch resumes from. These +/// are the facts [`watch_applied`](crate::watch_applied) needs to repair an +/// expired cursor without guessing. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Retention { + /// Retention can evict a key's CURRENT value, not just superseded history + /// (NATS: `max_age`, per-message TTLs, or `discard: old` under a byte or + /// message limit). When true, a key missing from the bucket may never have + /// been deleted, so the bucket's key listing is not evidence of deletion. + pub evicts_current_values: bool, + /// The oldest revision still in the log (NATS: the stream's + /// `first_sequence`). Resuming after cursor `C` is gap-free iff + /// `first_revision <= C + 1` + /// ([`resume_window_ok`](crate::protocol::resume_window_ok)). + pub first_revision: u64, +} + /// Opaque version token that abstracts store-specific versioning. /// /// Different stores use different versioning schemes: @@ -332,6 +354,16 @@ pub trait KvWatcher: Send + Sync { let _ = cursor; self.watch_prefixes(prefixes, tx).await } + + /// The watched log's retention, read live: whether it can evict current + /// values, and the oldest revision it still holds. + /// + /// `Ok(None)` (the default) means the backend can't say. Callers must then + /// assume current values CAN be evicted, so the cursor-expired key-listing + /// diff is not trusted unless the caller explicitly chose it. + async fn retention(&self) -> Result, KvError> { + Ok(None) + } } /// Write operations - optional, edge proxy is primarily read-only. diff --git a/src/lib.rs b/src/lib.rs index 171a67b..179cfe4 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -30,6 +30,7 @@ mod export_lease; mod kv; mod nats; pub mod protocol; +mod repair; pub mod snapshot; #[cfg(feature = "fjall")] mod snapshot_fjall; @@ -45,10 +46,11 @@ pub use applied::{BatchConfig, ExportRequest, WatchScope, watch_applied}; pub use artifact::{ARTIFACT_SCHEMA_VERSION, ArtifactFile, ExportManifest, MANIFEST_FILE}; pub use export_lease::{ExportLease, LeaseGuard, LeaseRecord}; pub use kv::{ - KvEntry, KvError, KvPurge, KvReader, KvTtl, KvUpdate, KvWatcher, KvWriter, VersionToken, - WatchCursor, + KvEntry, KvError, KvPurge, KvReader, KvTtl, KvUpdate, KvWatcher, KvWriter, Retention, + VersionToken, WatchCursor, }; pub use nats::{NatsConnection, NatsConnectionConfig, nats_connect}; +pub use repair::{ExpiryRepair, RestoreSource, RestoredFold}; pub use snapshot::{AppendLogSnapshot, SnapshotStore}; #[cfg(feature = "fjall")] pub use snapshot_fjall::{FjallConfig, FjallReader, FjallSnapshot}; @@ -58,4 +60,6 @@ pub use stores::{ Connection, ConnectionCapabilities, DiscardPolicy, KvStore, StorageType, StoreConfig, }; #[cfg(feature = "transport")] -pub use transport::{ArtifactTransport, ObjectStoreTransport, PublishOutcome, run_export_round}; +pub use transport::{ + ArtifactRestore, ArtifactTransport, ObjectStoreTransport, PublishOutcome, run_export_round, +}; diff --git a/src/nats.rs b/src/nats.rs index 01e4b33..d8376b9 100644 --- a/src/nats.rs +++ b/src/nats.rs @@ -10,7 +10,8 @@ use tokio::sync::{RwLock, mpsc::Sender}; use tracing::{debug, error, info, warn}; use crate::kv::{ - KvEntry, KvError, KvPurge, KvReader, KvUpdate, KvWatcher, KvWriter, VersionToken, WatchCursor, + KvEntry, KvError, KvPurge, KvReader, KvUpdate, KvWatcher, KvWriter, Retention, VersionToken, + WatchCursor, }; use crate::stores::{Connection, ConnectionCapabilities, DiscardPolicy, KvStore, StoreConfig}; @@ -1053,22 +1054,24 @@ const FLOOR_GUARD_INTERVAL: Duration = Duration::from_secs(30); /// before the entry is processed: fetch `first_sequence` and apply the /// shared kernel (`protocol::resume_window_ok`) to the frontier. A benign /// gap (interior per-subject eviction with the floor still at or below the -/// frontier) passes; head eviction past the frontier fails the watch, and -/// the caller's restart routes into the verified resume → `CursorExpired` → -/// resync repair path. The periodic probe backstops the no-traffic case. +/// frontier) passes; head eviction past the frontier ends the watch with +/// [`KvError::CursorExpired`], after everything up to the frontier has been +/// delivered. `watch_applied` routes that into the same expiry repair a +/// resume-time expiry takes (artifact restore on a bucket that evicts current +/// values, the key-listing diff otherwise), in process. The periodic probe +/// backstops the no-traffic case. /// /// Scope: sound only where density holds — the unfiltered resume watch. /// Prefix-scoped watches deliver sparse revisions by design and cannot /// distinguish benign from hazardous eviction client-side; they retain the /// (narrowed) retention-outlives-lag operating axiom plus the resume-time -/// check on every restart (model axiom 5). +/// check on every restart (model axiom 6). /// /// The guarantee split, precisely: the SAFETY half — never folding past /// unexamined evidence of loss — is unconditional in this loop (the gap /// check precedes processing, and a stalled downstream stalls folding too). -/// The REPAIR half is conditional on the caller restarting the failed watch -/// (standard supervision; same posture as the resync fail-stop): a trip -/// with no restart is a loudly dead watch, never a silently wrong one. +/// The REPAIR half belongs to the caller of the `_from` watch: a raw caller +/// that ignores the error has a loudly dead watch, never a silently wrong one. async fn stream_watch_floor_guarded( mut watcher: async_nats::jetstream::kv::Watch, tx: &Sender, @@ -1083,18 +1086,19 @@ async fn stream_watch_floor_guarded( .map_err(|e| KvError::OperationFailed(format!("floor guard stream lookup: {e}")))?; Ok::(stream.cached_info().state.first_sequence) }; + // The frontier's position has expired exactly as a resume cursor would + // have: surface it as `CursorExpired` (mid-stream) so `watch_applied` + // routes it into the same expiry repair as a resume-time expiry, in + // process, instead of failing the watch and depending on a restart. fn trip(frontier: u64, first: u64, bucket: &str) -> KvError { warn!( frontier, first_sequence = first, bucket, - "stream retention overran this live watch; failing so the restart can resync \ - (messages in the gap were evicted unseen)" + "stream retention overran this live watch (messages in the gap were evicted \ + unseen); reporting the frontier as an expired cursor so it is repaired" ); - KvError::WatchError(format!( - "stream retention overran live watch (first_sequence {first} > delivered \ - frontier {frontier} + 1); restart will resync" - )) + KvError::CursorExpired } let mut frontier = resume_revision; @@ -1302,6 +1306,24 @@ impl NatsKvWatcher { } } +/// Whether a KV stream's retention can evict a key's CURRENT value — not just +/// history its own later revisions superseded. Any of these can: +/// - `max_age`: every message ages out, the newest revision of a key included. +/// - per-message TTLs (`allow_msg_ttl`): a key's value can carry its own expiry. +/// - `discard: old` under a byte or message limit: the oldest messages go when +/// the stream fills, whether or not they are still current. +/// +/// `discard: new` rejects writes at the limit instead of evicting, and the +/// per-subject history limit only drops superseded revisions, so neither +/// counts. When this returns false, a key missing from the bucket was deleted; +/// when it returns true, it may simply have aged out. +fn evicts_current_values(config: &async_nats::jetstream::stream::Config) -> bool { + use async_nats::jetstream::stream::DiscardPolicy as NatsDiscard; + !config.max_age.is_zero() + || config.allow_message_ttl + || (config.discard == NatsDiscard::Old && (config.max_bytes > 0 || config.max_messages > 0)) +} + #[async_trait] impl KvWatcher for NatsKvWatcher { async fn watch_all(&self, tx: Sender) -> Result<(), KvError> { @@ -1537,6 +1559,21 @@ impl KvWatcher for NatsKvWatcher { } Ok(()) } + + async fn retention(&self) -> Result, KvError> { + // Fresh stream info, not a cached handle: the expiry repair decides + // from this whether the key listing can be trusted, so it must reflect + // the bucket's config now (an operator can edit a live stream), and + // `first_revision` gates an artifact's freshness. + let stream = timed(self.js.get_stream(format!("KV_{}", self.bucket))) + .await? + .map_err(|e| KvError::OperationFailed(format!("get KV stream for retention: {e}")))?; + let info = stream.cached_info(); + Ok(Some(Retention { + evicts_current_values: evicts_current_values(&info.config), + first_revision: info.state.first_sequence, + })) + } } struct NatsKvWriterImpl { @@ -1649,6 +1686,53 @@ impl std::fmt::Debug for NatsConnection { mod tests { use super::*; + #[test] + fn evicts_current_values_classifies_retention() { + use async_nats::jetstream::stream::{Config, DiscardPolicy as NatsDiscard}; + let kv = |f: fn(&mut Config)| { + // KV defaults: discard:new, unlimited age, per-subject history. + let mut c = Config { + max_messages_per_subject: 1, + discard: NatsDiscard::New, + ..Default::default() + }; + f(&mut c); + evicts_current_values(&c) + }; + assert!( + !kv(|_| {}), + "history-only eviction drops superseded revisions only" + ); + assert!( + !kv(|c| c.max_bytes = 1 << 20), + "discard:new rejects at the limit" + ); + assert!(kv(|c| c.max_age = Duration::from_secs(60)), "max_age"); + assert!(kv(|c| c.allow_message_ttl = true), "per-message TTLs"); + assert!( + kv(|c| { + c.discard = NatsDiscard::Old; + c.max_bytes = 1 << 20; + }), + "discard:old under a byte limit" + ); + assert!( + kv(|c| { + c.discard = NatsDiscard::Old; + c.max_messages = 100; + }), + "discard:old under a message limit" + ); + assert!( + !kv(|c| { + c.discard = NatsDiscard::Old; + c.max_bytes = -1; + c.max_messages = -1; + }), + "discard:old with no limit has nothing to discard" + ); + } + #[test] fn raw_create_success_has_no_error() { // A successful STREAM.CREATE echoes back the stream config, no "error". @@ -2070,10 +2154,9 @@ mod floor_guard_tests { .await .expect("the trip must be IN-BAND (immediate), not backstop-paced") .expect_err("a gapped delivery over an advanced floor must trip"); - assert!( - err.to_string().contains("retention overran live watch"), - "{err}" - ); + // Reported as an expired cursor so `watch_applied` repairs it in + // process through the same path as a resume-time expiry. + assert!(matches!(err, KvError::CursorExpired), "{err:?}"); drop(tx); let _ = drain.await; } @@ -2115,4 +2198,133 @@ mod floor_guard_tests { assert_eq!(got, vec![4, 5, 6, 7], "interior gaps jumped, tail dense"); guard.abort(); // endless live watch; the assertion above is the test } + + /// `retention()` reads the live stream: `discard: old` under a byte limit + /// and `max_age` both evict current values; the default `discard: new` + /// bucket does not. `first_revision` tracks the stream's head. + #[tokio::test(flavor = "multi_thread")] + async fn retention_reports_current_value_eviction() { + let server = start_server().await; + let conn = crate::NatsConnection::new(crate::NatsConnectionConfig { + url: server.url.clone(), + creds: None, + creds_file: None, + }); + conn.connect().await.unwrap(); + let open = |name: &str, max_age, discard| StoreConfig { + name: name.to_string(), + max_age, + max_bytes: Some(1 << 20), + discard, + ..Default::default() + }; + + let config = conn + .store_with_config(open("config", None, DiscardPolicy::New)) + .await + .unwrap(); + let log = conn + .store_with_config(open("log", None, DiscardPolicy::Old)) + .await + .unwrap(); + let aged = conn + .store_with_config(open( + "aged", + Some(Duration::from_secs(3600)), + DiscardPolicy::New, + )) + .await + .unwrap(); + + let r = |s: &Arc| { + let w = s.watcher().unwrap(); + async move { + w.retention() + .await + .unwrap() + .expect("NATS reports retention") + } + }; + assert!( + !r(&config).await.evicts_current_values, + "discard:new never evicts" + ); + assert!( + r(&log).await.evicts_current_values, + "discard:old under max_bytes" + ); + assert!(r(&aged).await.evicts_current_values, "max_age"); + + // The two inputs no StoreConfig expresses, set on live streams: + // per-message TTLs, and discard:old under a message-count limit. + let raw = async_nats::connect(&server.url).await.unwrap(); + let js = async_nats::jetstream::new(raw); + let ttl = conn + .store_with_config(open("ttl", None, DiscardPolicy::New)) + .await + .unwrap(); + let mut cfg = js + .get_stream("KV_ttl") + .await + .unwrap() + .cached_info() + .config + .clone(); + cfg.allow_message_ttl = true; + js.update_stream(cfg) + .await + .expect("enable per-message TTLs"); + assert!(r(&ttl).await.evicts_current_values, "per-message TTLs"); + + let counted = conn + .store_with_config(open("counted", None, DiscardPolicy::New)) + .await + .unwrap(); + let mut cfg = js + .get_stream("KV_counted") + .await + .unwrap() + .cached_info() + .config + .clone(); + cfg.max_bytes = -1; + cfg.max_messages = 100; + cfg.discard = async_nats::jetstream::stream::DiscardPolicy::Old; + js.update_stream(cfg) + .await + .expect("discard:old under max_msgs"); + assert!( + r(&counted).await.evicts_current_values, + "discard:old under max_msgs" + ); + + // And an edit to a live stream is seen: retention is read live, not + // remembered from creation. + let mut cfg = js + .get_stream("KV_config") + .await + .unwrap() + .cached_info() + .config + .clone(); + cfg.max_age = Duration::from_secs(3600); + js.update_stream(cfg) + .await + .expect("add max_age to a live bucket"); + assert!( + r(&config).await.evicts_current_values, + "max_age added out-of-band" + ); + + // first_revision follows head eviction. + let w = config.writer().unwrap(); + for _ in 0..3 { + w.put("k", b"v").await.unwrap(); + } + assert_eq!( + r(&config).await.first_revision, + 3, + "history 1 superseded revs 1-2" + ); + } } diff --git a/src/protocol.rs b/src/protocol.rs index 826820b..e225632 100644 --- a/src/protocol.rs +++ b/src/protocol.rs @@ -11,6 +11,7 @@ //! | [`pointer_publish_allowed`]| `transport::swap_pointer` | `Act::Publish` | //! | [`payload_prunable`] | `transport::ObjectStoreTransport::prune` | `Act::Prune` | //! | [`resume_window_ok`] | `nats` resume paths (`check_resume_window`) | `Act::Resume` | +//! | [`restore_allowed`] | `applied` artifact restore (`check_restore`) | `Act::RestoreRead`, live `GuardRepair` | //! //! Because the model transitions call these very functions, a change to any //! guard is re-verified against the full bounded state space on the next @@ -98,7 +99,11 @@ pub fn payload_prunable( /// the gap is safe for a last-write-wins fold (an overwrite-evicted revision /// implies a later revision of the same subject exists and will be /// delivered); lost DELETES come from head eviction, which is exactly what -/// advances `first_sequence`. +/// advances `first_sequence`. On a bucket whose retention also evicts +/// *current* values (`max_age`, `discard: old`), head eviction loses live puts +/// too. The same check detects that, but the repair then has to come from an +/// artifact ([`restore_allowed`]), because the bucket's key listing no longer +/// separates "deleted" from "aged out". /// /// This check must be performed by US: NATS does not error on a below-head /// start sequence — it silently clamps to the first retained message @@ -111,6 +116,28 @@ pub fn resume_window_ok(revision: u64, first_sequence: u64) -> bool { first_sequence <= revision.saturating_add(1) } +/// THE restore guard: may a fold whose cursor (`local_revision`) fell out of +/// the log be replaced by an artifact exported at `artifact_revision`? +/// +/// Both must hold: +/// - **Ahead**: the artifact is strictly newer than the local fold. A restore +/// never moves a fold (or the consumer's domain state built from it) +/// backward. +/// - **Fresh**: the artifact's cursor is still inside the log's retention +/// window ([`resume_window_ok`]), so the watch can resume from it and replay +/// the tail without a gap. A stale artifact has no safe recovery: whatever +/// was evicted between its cursor and `first_sequence` is gone from both the +/// artifact and the log. +/// +/// "Fresh" implies "ahead" whenever the local cursor is genuinely expired; the +/// ahead half is what still protects a restore when retention can't be read +/// (or was read before a stream was recreated). Machine-checked in +/// `tests/model.rs` (`RestoreRead`) and `tests/model_live_watch.rs` +/// (`GuardRepair`). +pub fn restore_allowed(artifact_revision: u64, local_revision: u64, first_sequence: u64) -> bool { + artifact_revision > local_revision && resume_window_ok(artifact_revision, first_sequence) +} + /// Sequence the watch should start at after applying `revision`. /// /// `None` at `u64::MAX`: wrapping to 0 would silently replay the bucket @@ -159,6 +186,25 @@ mod tests { assert_eq!(resume_start_sequence(u64::MAX), None); } + #[test] + fn restore_guard_boundaries() { + // Local cursor 3 expired (first retained 6): an artifact at 5 resumes + // at 6 — ahead and fresh. + assert!(restore_allowed(5, 3, 6)); + // Artifact at 4 would resume at 5, already evicted: stale. + assert!(!restore_allowed(4, 3, 6), "stale artifact refused"); + // Equal to local is not ahead, even inside the window. + assert!(!restore_allowed(3, 3, 1), "not strictly ahead"); + assert!( + !restore_allowed(2, 3, 1), + "older artifact never regresses the fold" + ); + assert!( + restore_allowed(u64::MAX, 0, u64::MAX), + "saturating boundary" + ); + } + #[test] fn resume_start_does_not_wrap_in_nats() { let src = include_str!("nats.rs"); diff --git a/src/repair.rs b/src/repair.rs new file mode 100644 index 0000000..4e58b6a --- /dev/null +++ b/src/repair.rs @@ -0,0 +1,728 @@ +//! Cursor-expiry repair: what [`watch_applied`](crate::watch_applied) does +//! when its resume cursor, or a live watch's frontier, has fallen out of the +//! log. +//! +//! The fallback full watch re-lists every key the bucket still holds. That +//! re-list can't deliver what retention already dropped, and what retention +//! drops depends on the bucket: +//! +//! - **Never evicts current values** (`discard: new`, no `max_age`): only +//! superseded history and delete markers go. A key missing from the bucket +//! was deleted, so the listing is the truth. The repair is the +//! **key-listing diff** ([`ExpiryRepair::Relist`]): synthetic deletes for +//! in-scope keys the bucket no longer lists, then the re-list. +//! - **Evicts current values** (`max_age`, per-message TTLs, `discard: old` +//! under a limit): a key missing from the bucket may just have aged out, and +//! writes made during the gap may have aged out too. NATS can't tell +//! "deleted, marker evicted" from "aged out", so no listing diff can be +//! repaired into a correct one. The repair is an **artifact restore** +//! ([`ExpiryRepair::Restore`]): replace the in-scope fold with the newest +//! published artifact — which carries every real delete, because its +//! exporter folded them — and resume from its cursor. +//! +//! Correct means "every write minus every real delete", not "whatever NATS +//! still lists". On a bucket that evicts current values those differ, and only +//! a fold (here or in an artifact) holds the former. + +use std::any::Any; +use std::sync::Arc; + +use async_trait::async_trait; + +use crate::artifact::ExportManifest; +use crate::kv::{KvEntry, KvReader, KvUpdate, VersionToken, WatchCursor}; +use crate::protocol::{restore_allowed, resume_window_ok}; +use crate::snapshot::{SnapshotError, SnapshotStore}; + +/// How [`watch_applied`](crate::watch_applied) repairs the fold when its +/// cursor expires — at resume, or mid-watch when the live floor guard finds +/// retention overran the consumer. Both cases take the same path. +/// +/// `Option>` converts into this, so the pre-0.8 `reader` +/// argument keeps compiling: `None` is [`None`](Self::None), `Some(reader)` is +/// [`Relist`](Self::Relist). +pub enum ExpiryRepair { + /// No repair: fall back to the re-list alone. Keys deleted during the gap + /// whose delete markers were evicted stay in the fold (a warning names the + /// hazard), and on a bucket that evicts current values, gap writes that + /// aged out are missed. + None, + /// The key-listing diff: synthetic deletes for in-scope keys the bucket + /// no longer lists, then the re-list. Sound only when absence from the + /// bucket means deleted, so it is REFUSED (the watch fails) when the + /// watcher reports retention that evicts current values. On a backend that + /// can't report retention, choosing this vouches that it doesn't. + Relist(Arc), + /// Replace the in-scope fold with the newest published artifact and resume + /// from its cursor. Sound for any bucket, as long as the artifact is ahead + /// of the fold, covers the watch scope, and its cursor is still inside the + /// log's retention window. When one of those fails there is no safe + /// recovery, and the watch fails with an error (logged at `error`) rather + /// than guess. Keep the export interval well under the bucket's `max_age` + /// (or its `discard: old` turnover). + /// + /// Also used at a cursor-less start when the watcher reports that + /// retention has already evicted current values: the bucket alone can no + /// longer seed a complete fold. + /// + /// The `reader` lists the bucket's live keys at restore time. A restore + /// never deletes a key that is listed live, nor moves a key to an older + /// revision: an artifact exported while its exporter was catching up + /// lacks (or holds older values for) keys whose latest write is after + /// its cursor, and the resume delivers those — so the consumer never + /// sees a phantom delete or a regression, even transiently. + Restore { + /// Lists live keys, so the restore never deletes a live one. + reader: Arc, + /// Supplies the artifact. + restore: Arc>, + }, + /// Decide by the bucket's live retention, read at the moment of expiry: + /// [`Relist`](Self::Relist) when it never evicts current values, + /// [`Restore`](Self::Restore) when it does or can't say. + Auto { + /// Lists live keys (for either repair). + reader: Arc, + /// Supplies the artifact for a restore. + restore: Arc>, + }, +} + +impl From>> for ExpiryRepair { + fn from(reader: Option>) -> Self { + match reader { + Some(reader) => ExpiryRepair::Relist(reader), + None => ExpiryRepair::None, + } + } +} + +/// Where an [`ExpiryRepair::Restore`] gets its artifact: the newest one +/// published for this fleet, opened as a temporary fold of the consumer's own +/// backend. [`ArtifactRestore`](crate::ArtifactRestore) (feature `transport`) +/// implements it over an [`ArtifactTransport`](crate::ArtifactTransport). +#[async_trait] +pub trait RestoreSource: Send + Sync { + /// The newest published artifact's manifest, without downloading the + /// payload, so a stale, out-of-scope, or not-newer artifact is refused + /// before its download. + async fn latest(&self) -> Result; + + /// Download, verify, and open the newest published artifact as a + /// temporary fold. It may be newer than what [`latest`](Self::latest) + /// returned; the caller re-checks it. + async fn fetch(&self) -> Result, SnapshotError>; +} + +/// A verified artifact opened as a temporary, read-only fold, plus anything +/// that must outlive it (its scratch directory). Dropping it drops the fold +/// first, then the guard. +pub struct RestoredFold { + manifest: ExportManifest, + // Declaration order is drop order, and it is load-bearing: the fold (an + // open LSM, possibly) must close before its guard deletes its directory. + fold: S, + guard: Option>, +} + +impl RestoredFold { + /// Wrap a fold opened from the artifact described by `manifest`. + pub fn new(manifest: ExportManifest, fold: S) -> Self { + Self { + manifest, + fold, + guard: None, + } + } + + /// Keep `guard` (e.g. the `TempDir` holding the fold's files) alive until + /// after the fold is dropped. + pub fn holding(mut self, guard: impl Any + Send) -> Self { + self.guard = Some(Box::new(guard)); + self + } + + /// The artifact's manifest: its cursor, backend, and scope. + pub fn manifest(&self) -> &ExportManifest { + &self.manifest + } + + /// The artifact's fold. + pub fn fold(&self) -> &S { + &self.fold + } +} + +/// Rank for the restore guard: the cursor's revision, revisionless cursors 0. +pub(crate) fn cursor_rank(c: &WatchCursor) -> u64 { + c.as_u64().unwrap_or(0) +} + +/// Does an artifact exported under the `exporter` scope cover every key in +/// the `watcher` scope? Both are key-prefix lists (`[""]` is every key). A +/// watcher prefix is covered when some exporter prefix is a prefix of it. +/// Sound but conservative: a watcher prefix covered only by a union of +/// narrower exporter prefixes counts as uncovered. +pub(crate) fn scope_covers(exporter: &[String], watcher: &[String]) -> bool { + watcher + .iter() + .all(|w| exporter.iter().any(|e| w.starts_with(e.as_str()))) +} + +/// Can `artifact` repair a fold at `local` whose cursor expired? Checks scope +/// coverage, then [`restore_allowed`] (ahead + fresh) when the log's first +/// retained revision is known, the ahead half alone when it isn't (the resume +/// that follows re-checks the window). `Err` is the operator-facing reason. +pub(crate) fn check_restore( + artifact: &ExportManifest, + local: &WatchCursor, + first_revision: Option, + watch_scope: &[String], +) -> Result<(), String> { + match &artifact.scope { + None => { + return Err(format!( + "the latest artifact (cursor {:?}) records no key scope — it was exported \ + outside watch_applied or by a pre-scope build — so it can't be shown to cover \ + watch scope {watch_scope:?}; publish a new export round", + artifact.cursor + )); + } + Some(exporter) if !scope_covers(exporter, watch_scope) => { + return Err(format!( + "the latest artifact covers scope {exporter:?}, which does not cover watch \ + scope {watch_scope:?}" + )); + } + Some(_) => {} + } + let (a, l) = (cursor_rank(&artifact.cursor), cursor_rank(local)); + if a <= l { + return Err(format!( + "the latest artifact (cursor {a}) is not ahead of the local fold (cursor {l}): \ + nothing newer to restore from; the export pipeline is behind this node" + )); + } + if let Some(first) = first_revision + && !restore_allowed(a, l, first) + { + debug_assert!(!resume_window_ok(a, first)); + return Err(format!( + "the latest artifact (cursor {a}) is outside the log's retention window (first \ + retained revision {first}): what was evicted between them exists nowhere, so there \ + is no safe recovery. Exports must run well inside the bucket's max_age / discard \ + turnover" + )); + } + Ok(()) +} + +/// One key of a restore: take the artifact's entry, or delete the key. +pub(crate) enum RestoreOp { + Put(String), + Delete(String), +} + +impl RestoreOp { + fn key(&self) -> &str { + match self { + RestoreOp::Put(k) | RestoreOp::Delete(k) => k, + } + } +} + +/// Does the local entry already hold the artifact's value or a newer one? +/// Revisions decide when both have one; otherwise only identical entries +/// count. +fn local_is_current(local: &KvEntry, artifact: &KvEntry) -> bool { + match (local.version.as_u64(), artifact.version.as_u64()) { + (Some(l), Some(a)) if l != a => l > a, + _ => local.version == artifact.version && local.value == artifact.value, + } +} + +/// The in-scope ops that bring the `local` fold to the `artifact` fold's +/// state at its cursor `C`, without ever moving a key backward or deleting a +/// live one. Keys only, in key order — values are read back in bounded +/// chunks ([`materialize`]), so a restore never holds the whole changed set's +/// values at once (the same discipline as the key-listing diff). +/// +/// An artifact is complete together with the log after its cursor: every +/// RETAINED message at or below `C` is in it, but a key whose latest write is +/// after `C` may be missing or older (its exporter was mid catch-up). The +/// resume from `C` delivers those. So, per in-scope key: +/// +/// - **artifact has it**: `Put`, unless the local entry is already the same +/// or NEWER — then the key's latest write must be after `C` (the artifact +/// would hold it otherwise), and the resume delivers it; taking the older +/// artifact value would move the key backward. +/// - **artifact lacks it, local has it**: `Delete`, unless the bucket lists +/// it as `live` — then its current value is a write after `C` (again, the +/// artifact would hold it otherwise), which the resume delivers; deleting +/// it would drop a live key. A key not listed was really deleted (or its +/// delete marker evicted), which is exactly what the exporter folded. +/// +/// Out-of-scope keys are untouched on both sides. +pub(crate) fn restore_diff( + local: &S, + artifact: &S, + prefixes: &[String], + live: &std::collections::HashSet, +) -> Result, SnapshotError> { + let mut ops = Vec::new(); + for prefix in prefixes { + artifact.for_each_in_range(prefix, |entry| { + let current = local + .get(&entry.key)? + .is_some_and(|l| local_is_current(&l, &entry)); + if !current { + ops.push(RestoreOp::Put(entry.key)); + } + Ok(()) + })?; + local.for_each_in_range(prefix, |entry| { + if !live.contains(&entry.key) && artifact.get(&entry.key)?.is_none() { + ops.push(RestoreOp::Delete(entry.key)); + } + Ok(()) + })?; + } + // Overlapping prefixes visit a key more than once; a key is a Put or a + // Delete, never both (Put iff the artifact has it). + ops.sort_unstable_by(|a, b| a.key().cmp(b.key())); + ops.dedup_by(|a, b| a.key() == b.key()); + Ok(ops) +} + +/// Read one chunk of a restore's values back from the artifact fold. Deletes +/// carry the unknown version: like the key-listing diff's synthetic deletes, +/// they are a state correction, not a log entry, and never move a cursor. +pub(crate) fn materialize( + artifact: &S, + ops: Vec, +) -> Result, SnapshotError> { + ops.into_iter() + .map(|op| match op { + RestoreOp::Put(key) => match artifact.get(&key)? { + Some(entry @ KvEntry { .. }) => Ok(KvUpdate::Put(entry)), + None => Err(SnapshotError::Backend(format!( + "restored artifact fold lost key {key:?} mid-restore" + ))), + }, + RestoreOp::Delete(key) => Ok(KvUpdate::Delete { + key, + version: VersionToken::unknown(), + }), + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::snapshot::AppendLogSnapshot; + + fn put(key: &str, value: &[u8], rev: u64) -> KvUpdate { + KvUpdate::Put(KvEntry { + key: key.to_string(), + value: value.to_vec(), + version: VersionToken::from_u64(rev), + }) + } + + fn fold( + dir: &tempfile::TempDir, + name: &str, + updates: &[KvUpdate], + at: u64, + ) -> AppendLogSnapshot { + let (_c, mut s) = AppendLogSnapshot::open(&dir.path().join(name), u64::MAX).unwrap(); + s.apply(updates, &WatchCursor::from_u64(at)).unwrap(); + s + } + + fn manifest(cursor: u64, scope: Option<&[&str]>) -> ExportManifest { + ExportManifest { + schema_version: crate::artifact::ARTIFACT_SCHEMA_VERSION, + backend: "append-log".into(), + backend_version: "2".into(), + cursor: WatchCursor::from_u64(cursor), + created_at_unix: 0, + files: vec![], + scope: scope.map(|s| s.iter().map(|p| p.to_string()).collect()), + } + } + + #[test] + fn scope_coverage() { + let s = |v: &[&str]| v.iter().map(|p| p.to_string()).collect::>(); + assert!( + scope_covers(&s(&[""]), &s(&["node.", "edge."])), + "All covers anything" + ); + assert!( + scope_covers(&s(&["node."]), &s(&["node.us."])), + "wider prefix covers narrower" + ); + assert!( + !scope_covers(&s(&["node.us."]), &s(&["node."])), + "narrower never covers wider" + ); + assert!( + !scope_covers(&s(&["node."]), &s(&[""])), + "a prefix never covers All" + ); + assert!( + !scope_covers(&s(&["node."]), &s(&["node.", "edge."])), + "every watcher prefix must be covered" + ); + assert!( + scope_covers(&s(&["a."]), &s(&[])), + "an empty watch scope is vacuously covered" + ); + } + + #[test] + fn check_restore_refusals() { + let all = vec![String::new()]; + let local = WatchCursor::from_u64(3); + // Ahead + fresh + covering: accepted. + check_restore(&manifest(9, Some(&[""])), &local, Some(8), &all).unwrap(); + // Unknown retention: ahead alone suffices here (the resume re-checks). + check_restore(&manifest(9, Some(&[""])), &local, None, &all).unwrap(); + + let err = check_restore(&manifest(9, None), &local, Some(8), &all).unwrap_err(); + assert!(err.contains("no key scope"), "{err}"); + let err = check_restore(&manifest(9, Some(&["node."])), &local, Some(8), &all).unwrap_err(); + assert!(err.contains("does not cover"), "{err}"); + let err = check_restore(&manifest(3, Some(&[""])), &local, None, &all).unwrap_err(); + assert!(err.contains("not ahead"), "{err}"); + let err = check_restore(&manifest(6, Some(&[""])), &local, Some(8), &all).unwrap_err(); + assert!(err.contains("outside the log's retention window"), "{err}"); + } + + /// The diff turns the local in-scope fold into the artifact's: changed and + /// new keys are Puts, keys the artifact lacks are Deletes, identical + /// entries and out-of-scope keys are left alone. + #[test] + fn diff_is_wholesale_replacement_within_scope() { + let dir = tempfile::TempDir::new().unwrap(); + let local = fold( + &dir, + "local", + &[ + put("node.same", b"1", 1), + put("node.changed", b"old", 2), + put("node.gone", b"x", 3), + put("other.local", b"keep", 4), + ], + 4, + ); + let artifact = fold( + &dir, + "artifact", + &[ + put("node.same", b"1", 1), + put("node.changed", b"new", 7), + put("node.new", b"gap-write", 8), + put("other.remote", b"ignored", 9), + ], + 9, + ); + let ops = restore_diff( + &local, + &artifact, + &["node.".to_string()], + &Default::default(), + ) + .unwrap(); + let got: Vec<(String, bool)> = ops + .iter() + .map(|o| (o.key().to_string(), matches!(o, RestoreOp::Put(_)))) + .collect(); + assert_eq!( + got, + vec![ + ("node.changed".into(), true), + ("node.gone".into(), false), + ("node.new".into(), true), + ] + ); + + let updates = materialize(&artifact, ops).unwrap(); + match &updates[0] { + KvUpdate::Put(e) => { + assert_eq!(e.value, b"new"); + assert_eq!( + e.version.as_u64(), + Some(7), + "restored entries keep their revision" + ); + } + other => panic!("expected put, got {other:?}"), + } + assert!( + matches!(&updates[1], KvUpdate::Delete { version, .. } if version.is_unknown()), + "restore deletes never carry a revision" + ); + } + + #[test] + fn diff_dedups_overlapping_prefixes() { + let dir = tempfile::TempDir::new().unwrap(); + let local = fold(&dir, "local", &[put("node.a", b"1", 1)], 1); + let artifact = fold(&dir, "artifact", &[put("node.b", b"2", 2)], 2); + let ops = restore_diff( + &local, + &artifact, + &["node.".into(), "node".into()], + &Default::default(), + ) + .unwrap(); + assert_eq!(ops.len(), 2, "each key once despite two covering prefixes"); + } + + // --- Exhaustive small-domain checks of the real functions --------------- + + /// Every string over `alphabet` of length 0..=max_len. + fn strings(alphabet: &[char], max_len: usize) -> Vec { + let mut out = vec![String::new()]; + let mut frontier = vec![String::new()]; + for _ in 0..max_len { + let mut next = Vec::new(); + for s in &frontier { + for c in alphabet { + let mut t = s.clone(); + t.push(*c); + next.push(t); + } + } + out.extend(next.iter().cloned()); + frontier = next; + } + out + } + + /// Every subset of `items` with at most `max` elements. + fn subsets(items: &[String], max: usize) -> Vec> { + let mut out: Vec> = vec![vec![]]; + for item in items { + let grown: Vec> = out + .iter() + .filter(|s| s.len() < max) + .map(|s| { + let mut t = s.clone(); + t.push(item.clone()); + t + }) + .collect(); + out.extend(grown); + } + out + } + + /// SOUNDNESS of `scope_covers`, exhaustively over every pair of prefix + /// sets (≤ 2 prefixes of length ≤ 2 over {a, b, .}) and every key of + /// length ≤ 3: whenever it says the exporter covers the watcher, every + /// key the watcher can see is one the exporter folded. (It is + /// deliberately not complete — a union of narrower exporter prefixes is + /// refused — which only ever refuses a restore, never accepts a bad one.) + #[test] + fn exhaustive_scope_covers_is_sound() { + let alphabet = ['a', 'b', '.']; + let prefixes = strings(&alphabet, 2); + let keys = strings(&alphabet, 3); + let sets = subsets(&prefixes, 2); + let matches = + |scope: &[String], key: &str| scope.iter().any(|p| key.starts_with(p.as_str())); + let mut checked = 0usize; + for exporter in &sets { + for watcher in &sets { + if !scope_covers(exporter, watcher) { + continue; + } + for key in &keys { + assert!( + !matches(watcher, key) || matches(exporter, key), + "covers({exporter:?}, {watcher:?}) but {key:?} is visible to the \ + watcher and not in the export" + ); + checked += 1; + } + } + } + assert!(checked > 10_000, "the domain is not vacuous ({checked})"); + } + + /// `check_restore` against its specification, exhaustively over cursors + /// 0..=6, first retained revisions (unknown, 0..=8), and three scope + /// shapes: accept iff the artifact has a scope covering the watcher's, is + /// strictly ahead, and (when retention is known) resumes gap-free. + #[test] + fn exhaustive_check_restore_matches_spec() { + let all = vec![String::new()]; + let node = vec!["node.".to_string()]; + let scopes: [Option<&[&str]>; 3] = [None, Some(&[""]), Some(&["edge."])]; + for watcher in [&all, &node] { + for scope in scopes { + for a in 0..=6u64 { + for l in 0..=6u64 { + for first in std::iter::once(None).chain((0..=8u64).map(Some)) { + let got = check_restore( + &manifest(a, scope), + &WatchCursor::from_u64(l), + first, + watcher, + ) + .is_ok(); + let covered = scope.is_some_and(|s| { + let s: Vec = s.iter().map(|p| p.to_string()).collect(); + watcher + .iter() + .all(|w| s.iter().any(|e| w.starts_with(e.as_str()))) + }); + let want = covered && a > l && first.is_none_or(|f| f <= a + 1); + assert_eq!( + got, want, + "artifact {a} scope {scope:?}, local {l}, first {first:?}, \ + watcher {watcher:?}" + ); + } + } + } + } + } + } + + /// One key's state in the exhaustive restore check: absent, or a value + /// at a revision. + type KeyState = Option<(&'static [u8], u64)>; + const STATES: [KeyState; 3] = [None, Some((b"v1", 1)), Some((b"v2", 2))]; + const KEYS: [&str; 3] = ["a.1", "a.2", "b.1"]; + + fn fold_of( + dir: &tempfile::TempDir, + name: &str, + states: &[KeyState], + cursor: u64, + ) -> AppendLogSnapshot { + let updates: Vec = KEYS + .iter() + .zip(states) + .filter_map(|(k, st)| st.map(|(v, rev)| put(k, v, rev))) + .collect(); + fold(dir, name, &updates, cursor) + } + + /// THE RESTORE, exhaustively over every pair of 3-key folds (each key + /// absent or at one of two revisions: 729 pairs), four scopes including + /// overlapping prefixes, and every live-key listing (8), using the real + /// `restore_diff` and `materialize` and a real store. Per in-scope key, + /// the restored fold holds: + /// + /// - the artifact's entry, unless the local one is NEWER (or the same) — + /// never a move backward; + /// - nothing where the artifact has nothing, unless the key is listed + /// live — never a phantom delete; + /// + /// and out-of-scope keys are untouched. A crash after any prefix of the + /// ops (committed under the old cursor, as `watch_applied` does) followed + /// by a fresh diff converges to the same result — the restore is + /// idempotent, which is what makes re-running it on restart safe — and a + /// converged fold diffs empty. + #[test] + fn exhaustive_restore_diff_never_regresses_never_drops_live_and_survives_partial_application() { + let scopes: [&[&str]; 4] = [&[""], &["a."], &["b."], &["a.", "a.1"]]; + let in_scope = |scope: &[&str], key: &str| scope.iter().any(|p| key.starts_with(p)); + let lives: Vec> = (0..8u8) + .map(|mask| { + KEYS.iter() + .enumerate() + .filter(|(i, _)| mask & (1 << i) != 0) + .map(|(_, k)| k.to_string()) + .collect() + }) + .collect(); + let mut cases = 0usize; + for local_states in itertools_product(&STATES) { + for artifact_states in itertools_product(&STATES) { + let dir = tempfile::TempDir::new().unwrap(); + let artifact = fold_of(&dir, "artifact", &artifact_states, 9); + for scope in scopes { + let prefixes: Vec = scope.iter().map(|p| p.to_string()).collect(); + for (li, live) in lives.iter().enumerate() { + let full = restore_diff( + &fold_of( + &dir, + &format!("probe-{li}-{}", scope.join("|")), + &local_states, + 3, + ), + &artifact, + &prefixes, + live, + ) + .unwrap(); + // Crash after `cut` ops: apply them, then re-diff. + for cut in 0..=full.len() { + let name = format!("local-{li}-{cut}-{}", scope.join("|")); + let mut local = fold_of(&dir, &name, &local_states, 3); + let first = restore_diff(&local, &artifact, &prefixes, live).unwrap(); + let partial: Vec = first.into_iter().take(cut).collect(); + let updates = materialize(&artifact, partial).unwrap(); + local.apply(&updates, &WatchCursor::from_u64(3)).unwrap(); + let rest = restore_diff(&local, &artifact, &prefixes, live).unwrap(); + let updates = materialize(&artifact, rest).unwrap(); + local.apply(&updates, &WatchCursor::from_u64(9)).unwrap(); + + for (i, key) in KEYS.iter().enumerate() { + let (l, a) = (local_states[i], artifact_states[i]); + let want = if !in_scope(scope, key) { + l + } else { + match (l, a) { + (Some(l), Some(a)) if l.1 >= a.1 => Some(l), + (_, Some(a)) => Some(a), + (Some(l), None) if live.contains(*key) => Some(l), + (_, None) => None, + } + }; + let got = local + .get(key) + .unwrap() + .map(|e| (e.value, e.version.as_u64())); + assert_eq!( + got, + want.map(|(v, r)| (v.to_vec(), Some(r))), + "key {key} scope {scope:?} live {live:?} local {local_states:?} \ + artifact {artifact_states:?} cut {cut}" + ); + } + assert!( + restore_diff(&local, &artifact, &prefixes, live) + .unwrap() + .is_empty(), + "a converged fold must diff empty" + ); + cases += 1; + } + } + } + } + } + assert!(cases > 20_000, "the domain is not vacuous ({cases})"); + } + + /// Every assignment of `states` to the three keys. + fn itertools_product(states: &[KeyState; 3]) -> Vec<[KeyState; 3]> { + let mut out = Vec::new(); + for a in states { + for b in states { + for c in states { + out.push([*a, *b, *c]); + } + } + } + out + } +} diff --git a/src/snapshot.rs b/src/snapshot.rs index 9907a6b..21c789e 100644 --- a/src/snapshot.rs +++ b/src/snapshot.rs @@ -107,6 +107,12 @@ impl Snapshot { /// After a cursor-expired fallback to full `watch_all()`, callers should /// compare the snapshot against the live key set and emit synthetic /// `Delete` events for stale keys to ensure convergence. + /// + /// Only on a bucket that never evicts current values (`discard: new`, no + /// `max_age`). Where retention evicts current values, a key absent from + /// the scan may simply have aged out, and deleting it loses data; repair + /// from an exported artifact instead + /// ([`ExpiryRepair`](crate::ExpiryRepair)). pub fn stale_keys<'a, I>(&'a self, current_keys: I) -> Vec<&'a str> where I: IntoIterator, diff --git a/src/stores.rs b/src/stores.rs index 72e99db..528e9de 100644 --- a/src/stores.rs +++ b/src/stores.rs @@ -30,9 +30,15 @@ pub enum DiscardPolicy { /// **Evict the oldest** messages when full (NATS `discard:old`): a hard size /// ceiling that never rejects. Correct for high-churn **log** buckets whose /// consumers hold the durable fold (e.g. routing origins): the bucket is a - /// bounded change-feed, not the source of truth, so an evicted entry is - /// recovered from the consumer's fold (and the `CursorExpired` resync path), - /// while writers never freeze. + /// bounded change-feed, not the source of truth, so an evicted entry lives + /// on in the consumers' folds, while writers never freeze. + /// + /// Eviction here drops CURRENT values, not just old history, so a key + /// missing from the bucket may never have been deleted. A consumer whose + /// cursor expires must repair from an exported artifact + /// ([`ExpiryRepair::Restore`](crate::ExpiryRepair::Restore) or + /// [`Auto`](crate::ExpiryRepair::Auto)), never from the bucket's key listing, + /// and the export interval must stay well inside the bucket's turnover. Old, } @@ -57,6 +63,12 @@ pub struct StoreConfig { /// Maximum age for entries in the bucket (bucket-level TTL). /// Entries older than this are automatically removed. /// NATS: maps to `max_age` on bucket config. + /// + /// This evicts CURRENT values: a key nobody deleted disappears from the + /// bucket once its last write is older than `max_age`. A consumer whose + /// cursor expires must then repair from an exported artifact (see + /// [`ExpiryRepair`](crate::ExpiryRepair)), with exports running well + /// inside `max_age`. pub max_age: Option, /// Maximum bytes for the bucket (required by Synadia Cloud). /// NATS: maps to `max_bytes` on bucket config. diff --git a/src/transport.rs b/src/transport.rs index c42c008..96ef56d 100644 --- a/src/transport.rs +++ b/src/transport.rs @@ -69,6 +69,7 @@ use crate::artifact::{ use crate::export_lease::ExportLease; use crate::kv::WatchCursor; use crate::protocol::{PointerState, payload_prunable, pointer_publish_allowed}; +use crate::repair::{RestoreSource, RestoredFold}; use crate::snapshot::SnapshotError; /// Buffered chunk size for uploads/downloads (also the multipart part size). @@ -785,18 +786,86 @@ pub async fn run_export_round( // --------------------------------------------------------------------------- /// Download `key` into a throwaway dir under `scratch_dir`, returning the -/// artifact path and the guard keeping it alive. +/// guard keeping it alive, the artifact path, and its manifest. async fn download_to_scratch( transport: &dyn ArtifactTransport, key: &str, scratch_dir: &Path, -) -> Result<(tempfile::TempDir, PathBuf), SnapshotError> { +) -> Result<(tempfile::TempDir, PathBuf, ExportManifest), SnapshotError> { let tmp = tempfile::Builder::new() .prefix(".slipstream-bootstrap-") .tempdir_in(scratch_dir)?; let artifact = tmp.path().join("artifact"); - transport.download(key, &artifact).await?; - Ok((tmp, artifact)) + let manifest = transport.download(key, &artifact).await?; + Ok((tmp, artifact, manifest)) +} + +/// The backend import an [`ArtifactRestore`] opens artifacts with: +/// `(artifact_dir, dest) -> (cursor, fold)`, e.g. +/// `|a, d| FjallSnapshot::import(a, d, config)`. +type ImportFn = dyn Fn(&Path, &Path) -> Result<(WatchCursor, S), SnapshotError> + Send + Sync; + +/// [`RestoreSource`] over an [`ArtifactTransport`]: the newest artifact +/// published under `key`, downloaded and imported into a scratch directory +/// with the consumer's own backend. Hand it to +/// [`watch_applied`](crate::watch_applied) as [`ExpiryRepair::Restore`] (or +/// [`ExpiryRepair::Auto`]) to repair an expired cursor from artifacts — the +/// only sound repair on a bucket whose retention evicts current values. +/// +/// [`ExpiryRepair::Restore`]: crate::ExpiryRepair::Restore +/// [`ExpiryRepair::Auto`]: crate::ExpiryRepair::Auto +pub struct ArtifactRestore { + transport: Arc, + key: String, + scratch_dir: PathBuf, + import: Arc>, +} + +impl ArtifactRestore { + /// Restore from the artifacts published under `key` on `transport`. + /// `scratch_dir` must exist and should sit on the fold's filesystem (the + /// artifact is downloaded and imported there, then deleted). `import` opens + /// a verified artifact directory as a fold at a path that doesn't exist + /// yet — the backend's `import`, e.g. + /// `|artifact, dest| AppendLogSnapshot::import(artifact, dest, u64::MAX)`. + pub fn new( + transport: Arc, + key: impl Into, + scratch_dir: impl Into, + import: impl Fn(&Path, &Path) -> Result<(WatchCursor, S), SnapshotError> + Send + Sync + 'static, + ) -> Self { + Self { + transport, + key: key.into(), + scratch_dir: scratch_dir.into(), + import: Arc::new(import), + } + } +} + +#[async_trait] +impl RestoreSource for ArtifactRestore { + async fn latest(&self) -> Result { + self.transport.manifest(&self.key).await + } + + async fn fetch(&self) -> Result, SnapshotError> { + let (tmp, artifact, manifest) = + download_to_scratch(&*self.transport, &self.key, &self.scratch_dir).await?; + let import = Arc::clone(&self.import); + let dest = tmp.path().join("fold"); + let fold = tokio::task::spawn_blocking(move || -> Result { + let (_cursor, fold) = import(&artifact, &dest)?; + // Import copied (and re-verified) the payload: the downloaded + // artifact is dead weight, possibly GBs — drop it now rather than + // for the life of the restore. + std::fs::remove_dir_all(&artifact)?; + Ok(fold) + }) + .await + .map_err(|e| SnapshotError::Backend(format!("restore import task panicked: {e}")))??; + Ok(RestoredFold::new(manifest, fold).holding(tmp)) + } } impl crate::AppendLogSnapshot { @@ -810,7 +879,8 @@ impl crate::AppendLogSnapshot { dest_path: &Path, compact_threshold: u64, ) -> Result<(WatchCursor, Self), SnapshotError> { - let (_guard, artifact) = download_to_scratch(transport, key, scratch_dir).await?; + let (_guard, artifact, _manifest) = + download_to_scratch(transport, key, scratch_dir).await?; let dest = dest_path.to_path_buf(); tokio::task::spawn_blocking(move || Self::import(&artifact, &dest, compact_threshold)) .await @@ -830,7 +900,8 @@ impl crate::FjallSnapshot { dest_dir: &Path, config: crate::FjallConfig, ) -> Result<(WatchCursor, Self), SnapshotError> { - let (_guard, artifact) = download_to_scratch(transport, key, scratch_dir).await?; + let (_guard, artifact, _manifest) = + download_to_scratch(transport, key, scratch_dir).await?; let dest = dest_dir.to_path_buf(); tokio::task::spawn_blocking(move || Self::import(&artifact, &dest, config)) .await @@ -850,7 +921,8 @@ impl crate::RocksDbSnapshot { dest_dir: &Path, config: crate::RocksDbConfig, ) -> Result<(WatchCursor, Self), SnapshotError> { - let (_guard, artifact) = download_to_scratch(transport, key, scratch_dir).await?; + let (_guard, artifact, _manifest) = + download_to_scratch(transport, key, scratch_dir).await?; let dest = dest_dir.to_path_buf(); tokio::task::spawn_blocking(move || Self::import(&artifact, &dest, config)) .await diff --git a/tests/eviction.rs b/tests/eviction.rs new file mode 100644 index 0000000..d4c431b --- /dev/null +++ b/tests/eviction.rs @@ -0,0 +1,617 @@ +//! Cursor-expiry repair on buckets whose retention evicts CURRENT values +//! (`discard: old` under `max_bytes`, `max_age`), against a live nats-server +//! and a real artifact transport (local filesystem object store). +//! +//! On such a bucket "not in NATS" no longer means "deleted": a key can age out +//! without anyone deleting it, and a write made while a watcher was offline +//! can age out before the watcher returns. The pre-fix key-listing resync +//! treated every key NATS no longer listed as deleted — reproduced here +//! against a live server for both eviction kinds: a restarted watcher deleted +//! a valid key it had already folded and never saw the gap write. These tests +//! pin the fix: +//! +//! - **Restore**: the expired watcher repairs from the newest artifact (the +//! live exporter folded every write and every real delete), keeps the +//! aged-out key, recovers the gap write, applies the real delete, and ends +//! identical to the exporter. +//! - **Relist refused**: the legacy `Some(reader)` repair fails the watch on +//! such a bucket instead of deleting valid keys. +//! - **Stale artifact**: an artifact whose cursor is outside retention fails +//! the watch — there is no safe recovery — and leaves the fold untouched. +//! - **Live floor-guard trip → restore**: a resumed, floor-guarded watcher is +//! stalled while a burst overruns it on a real `discard: old` bucket (the +//! server stops pushing to a consumer that doesn't answer flow control, +//! and retention evicts what it hadn't sent). Released, it drains, the +//! floor guard trips mid-stream, and the in-process restore brings it to +//! exactly the exporter's fold. +#![cfg(feature = "transport")] + +mod common; + +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use common::TestNats; +use slipstream::snapshot::{SnapshotError, SnapshotStore}; +use slipstream::{ + AppendLogSnapshot, ArtifactRestore, ArtifactTransport, BatchConfig, Connection, DiscardPolicy, + ExpiryRepair, ExportRequest, KvError, KvReader, KvStore, KvUpdate, NatsConnection, + NatsConnectionConfig, ObjectStoreTransport, StoreConfig, WatchCursor, WatchScope, + watch_applied, +}; +use tempfile::TempDir; +use tokio::sync::{mpsc, oneshot, watch}; +use tokio::time::timeout; + +const KEY: &str = "routes/latest"; + +struct Node { + exports: mpsc::Sender, + gate: Arc, + applied: Arc, + shutdown: watch::Sender, + task: tokio::task::JoinHandle>, +} + +impl Node { + fn spawn( + bucket: &Arc, + fold: AppendLogSnapshot, + resume: Option, + repair: ExpiryRepair, + ) -> Node { + Self::spawn_with(bucket, fold, resume, repair, BatchConfig::default()) + } + + /// `apply` blocks while the node's gate is closed (see `stall`). + fn spawn_with( + bucket: &Arc, + fold: AppendLogSnapshot, + resume: Option, + repair: ExpiryRepair, + config: BatchConfig, + ) -> Node { + let gate = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let g = Arc::clone(&gate); + let (ex_tx, ex_rx) = mpsc::channel(1); + let (sd_tx, sd_rx) = watch::channel(false); + let applied = Arc::new(AtomicU64::new( + resume.as_ref().and_then(WatchCursor::as_u64).unwrap_or(0), + )); + let a = Arc::clone(&applied); + let task = tokio::spawn(watch_applied( + bucket.watcher().expect("watcher"), + WatchScope::All, + resume, + repair, + Some(fold), + Some(ex_rx), + config, + |u: &KvUpdate| Some(u.key().to_string()), + move |_batch: Vec| { + while g.load(Ordering::SeqCst) { + std::thread::sleep(Duration::from_millis(5)); + } + }, + move |c: WatchCursor| a.store(c.as_u64().unwrap_or(0), Ordering::SeqCst), + sd_rx, + )); + Node { + exports: ex_tx, + gate, + applied, + shutdown: sd_tx, + task, + } + } + + async fn wait_applied(&self, at_least: u64) { + timeout(Duration::from_secs(15), async { + while self.applied.load(Ordering::SeqCst) < at_least { + assert!( + !self.task.is_finished(), + "watch ended before applying {at_least}" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "never applied {at_least} (at {})", + self.applied.load(Ordering::SeqCst) + ) + }); + } + + async fn stop(self) -> Result { + let _ = self.shutdown.send(true); + self.task.await.expect("watch task") + } + + /// Export through the live loop and publish it. + async fn publish(&self, transport: &dyn ArtifactTransport, scratch: &Path) -> u64 { + let dest = tempfile::Builder::new() + .prefix("export-") + .tempdir_in(scratch) + .unwrap(); + let artifact = dest.path().join("artifact"); + let (reply_tx, reply_rx) = oneshot::channel(); + self.exports + .send(ExportRequest { + dest_dir: artifact.clone(), + reply: reply_tx, + }) + .await + .expect("export request"); + let manifest = reply_rx.await.expect("reply").expect("export"); + transport.upload(KEY, &artifact).await.expect("upload"); + manifest.cursor.as_u64().expect("cursor") + } +} + +struct Harness { + _nats: TestNats, + _conn: NatsConnection, + bucket: Arc, + dir: TempDir, + transport: Arc, +} + +impl Harness { + async fn new(config: StoreConfig) -> Harness { + let nats = TestNats::start().await; + let conn = NatsConnection::new(NatsConnectionConfig { + url: nats.url.clone(), + creds: None, + creds_file: None, + }); + conn.connect().await.expect("connect"); + let bucket = conn.store_with_config(config).await.expect("bucket"); + let dir = TempDir::new().unwrap(); + std::fs::create_dir(dir.path().join("objects")).unwrap(); + let store = + object_store::local::LocalFileSystem::new_with_prefix(dir.path().join("objects")) + .expect("local object store"); + // file:// has no conditional puts; the explicit dev opt-in is fine for + // a single-exporter test (the pointer protocol is proved elsewhere). + let transport: Arc = Arc::new( + ObjectStoreTransport::new(Arc::new(store), "artifacts") + .with_non_atomic_pointer_fallback(), + ); + Harness { + _nats: nats, + _conn: conn, + bucket, + dir, + transport, + } + } + + fn path(&self, name: &str) -> PathBuf { + self.dir.path().join(name) + } + + fn open(&self, name: &str) -> (WatchCursor, AppendLogSnapshot) { + AppendLogSnapshot::open(&self.path(name), u64::MAX).expect("open fold") + } + + fn auto(&self) -> ExpiryRepair { + ExpiryRepair::Auto { + reader: self.bucket.reader(), + restore: Arc::new(ArtifactRestore::new( + Arc::clone(&self.transport), + KEY, + self.dir.path(), + |artifact: &Path, dest: &Path| -> Result<_, SnapshotError> { + AppendLogSnapshot::import(artifact, dest, u64::MAX) + }, + )), + } + } + + async fn first_sequence(&self) -> u64 { + self.bucket + .watcher() + .unwrap() + .retention() + .await + .unwrap() + .expect("NATS reports retention") + .first_revision + } +} + +/// How the test makes the bucket evict current values while the watcher is +/// offline. +#[derive(Clone, Copy)] +enum Eviction { + /// `discard: old` under a small `max_bytes`: filler writes push the oldest + /// messages out. Deterministic. + DiscardOld, + /// `max_age`: everything older than one second ages out. + MaxAge, +} + +impl Eviction { + fn config(self) -> StoreConfig { + match self { + Eviction::DiscardOld => StoreConfig { + name: "routes".into(), + max_bytes: Some(4096), + discard: DiscardPolicy::Old, + ..Default::default() + }, + Eviction::MaxAge => StoreConfig { + name: "routes".into(), + max_age: Some(Duration::from_secs(1)), + ..Default::default() + }, + } + } + + /// Evict everything written so far; returns the bucket's final revision. + async fn evict(self, bucket: &Arc) -> u64 { + let w = bucket.writer().unwrap(); + let mut last = 0; + match self { + Eviction::DiscardOld => { + for i in 0..200 { + last = w + .put(&format!("filler.{i}"), &[b'x'; 64]) + .await + .unwrap() + .as_u64() + .unwrap(); + } + } + Eviction::MaxAge => { + tokio::time::sleep(Duration::from_millis(2500)).await; + last = w.put("filler.tail", b"x").await.unwrap().as_u64().unwrap(); + } + } + last + } +} + +/// The shared setup: an exporter that stays live throughout, and a watcher +/// that folds `route.keep` and `route.gone`, goes offline, and comes back to +/// a bucket that — meanwhile — got a gap write (`route.late`), a real delete +/// (`route.gone`), and evicted the current values of `route.keep` and +/// `route.late`. The watcher's resume cursor is expired. +struct Gap { + h: Harness, + exporter: Node, + resume: WatchCursor, + final_rev: u64, +} + +async fn build_gap(eviction: Eviction, export_before_eviction: bool) -> (Gap, Option) { + let h = Harness::new(eviction.config()).await; + let w = h.bucket.writer().unwrap(); + + let (_c, fold_x) = h.open("exporter.snap"); + let exporter = Node::spawn(&h.bucket, fold_x, None, h.auto()); + let (_c, fold_w) = h.open("watcher.snap"); + let watcher = Node::spawn(&h.bucket, fold_w, None, h.auto()); + + // KV watches attach asynchronously; write until both have folded a seed. + timeout(Duration::from_secs(10), async { + loop { + w.put("route.seed", b"seed").await.unwrap(); + tokio::time::sleep(Duration::from_millis(50)).await; + if exporter.applied.load(Ordering::SeqCst) > 0 + && watcher.applied.load(Ordering::SeqCst) > 0 + { + return; + } + } + }) + .await + .expect("watches never attached"); + + w.put("route.keep", b"v1").await.unwrap(); + let rev = w.put("route.gone", b"x").await.unwrap().as_u64().unwrap(); + watcher.wait_applied(rev).await; + let resume = watcher.stop().await.expect("watcher ran clean"); + + // --- Offline ------------------------------------------------------------ + w.put("route.late", b"gap-write").await.unwrap(); + w.delete("route.gone").await.unwrap(); + let stale_export = if export_before_eviction { + let head = w.put("route.marker", b"m").await.unwrap().as_u64().unwrap(); + exporter.wait_applied(head).await; + Some(exporter.publish(&*h.transport, h.dir.path()).await) + } else { + None + }; + let final_rev = eviction.evict(&h.bucket).await; + + // The premise, asserted: NATS lost the current values (not deleted — + // aged out), and the watcher's cursor is outside retention. + let reader = h.bucket.reader(); + assert!( + reader.get("route.keep").await.unwrap().is_none(), + "route.keep must have aged out" + ); + assert!( + reader.get("route.late").await.unwrap().is_none(), + "route.late must have aged out" + ); + let first = h.first_sequence().await; + let resume_rev = resume.as_u64().unwrap(); + assert!( + first > resume_rev + 1, + "first_seq {first} must pass the resume cursor {resume_rev}" + ); + + exporter.wait_applied(final_rev).await; + ( + Gap { + h, + exporter, + resume, + final_rev, + }, + stale_export, + ) +} + +fn assert_converged(fold: &AppendLogSnapshot) { + assert_eq!( + fold.get("route.keep").unwrap().map(|e| e.value), + Some(b"v1".to_vec()), + "a key that aged out of NATS was never deleted: it stays" + ); + assert_eq!( + fold.get("route.late").unwrap().map(|e| e.value), + Some(b"gap-write".to_vec()), + "the write made while the watcher was offline is recovered" + ); + assert!( + fold.get("route.gone").unwrap().is_none(), + "the real delete is applied" + ); +} + +async fn restore_recovers(eviction: Eviction) { + let (gap, _) = build_gap(eviction, false).await; + let h = &gap.h; + let artifact_rev = gap.exporter.publish(&*h.transport, h.dir.path()).await; + assert!(artifact_rev >= gap.final_rev); + + // The live exporter never expired: it folded everything, the aged-out + // keys included (NATS sends no marker when a value ages out). + let exporter_fold = gap.exporter.stop().await.expect("exporter ran clean"); + assert_eq!(exporter_fold.as_u64(), Some(gap.final_rev)); + let (_c, exporter) = h.open("exporter.snap"); + assert_converged(&exporter); + + // The watcher comes back: expiry → (bucket evicts) → restore. + let (cursor, fold) = h.open("watcher.snap"); + assert_eq!(cursor, gap.resume); + let watcher = Node::spawn(&h.bucket, fold, Some(cursor), h.auto()); + watcher.wait_applied(artifact_rev).await; + let end = watcher.stop().await.expect("restore succeeded"); + assert_eq!(end.as_u64(), Some(artifact_rev)); + + let (cursor, watcher) = h.open("watcher.snap"); + assert_eq!(cursor.as_u64(), Some(artifact_rev)); + assert_converged(&watcher); + assert_eq!( + watcher + .range("") + .unwrap() + .iter() + .map(|e| (&e.key, &e.value)) + .collect::>(), + exporter + .range("") + .unwrap() + .iter() + .map(|e| (&e.key, &e.value)) + .collect::>(), + "the restored watcher is identical to the exporter" + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn discard_old_expiry_restores_from_artifact() { + restore_recovers(Eviction::DiscardOld).await; +} + +#[tokio::test(flavor = "multi_thread")] +async fn max_age_expiry_restores_from_artifact() { + restore_recovers(Eviction::MaxAge).await; +} + +/// The reported bug, pinned against a live server: the key-listing resync on +/// an evicting bucket would delete `route.keep` (NATS no longer lists it). +/// The legacy `Some(reader)` repair is now refused, and the fold is left +/// exactly as the watcher last persisted it. +#[tokio::test(flavor = "multi_thread")] +async fn relist_on_evicting_bucket_is_refused() { + let (gap, _) = build_gap(Eviction::DiscardOld, false).await; + let h = &gap.h; + let (cursor, fold) = h.open("watcher.snap"); + let legacy: Option> = Some(h.bucket.reader()); + let watcher = Node::spawn(&h.bucket, fold, Some(cursor.clone()), legacy.into()); + let err = timeout(Duration::from_secs(15), watcher.task) + .await + .expect("the watch must fail, not hang") + .unwrap() + .expect_err("relist on an evicting bucket must be refused"); + assert!(err.to_string().contains("evicts current values"), "{err}"); + + let (after, fold) = h.open("watcher.snap"); + assert_eq!(after, cursor, "cursor untouched"); + assert_eq!( + fold.get("route.keep").unwrap().map(|e| e.value), + Some(b"v1".to_vec()), + "the valid key the pre-fix resync deleted is still there" + ); +} + +/// An artifact exported before the eviction is ahead of the watcher but its +/// cursor fell out of retention with everything else: whatever was evicted +/// between it and the log's head exists nowhere. No safe recovery — the +/// watch fails and the fold stays as it was. +#[tokio::test(flavor = "multi_thread")] +async fn stale_artifact_fails_the_watch() { + let (gap, stale) = build_gap(Eviction::DiscardOld, true).await; + let h = &gap.h; + let stale = stale.expect("published before eviction"); + assert!( + stale > gap.resume.as_u64().unwrap(), + "ahead of the watcher, but stale" + ); + + let (cursor, fold) = h.open("watcher.snap"); + let watcher = Node::spawn(&h.bucket, fold, Some(cursor.clone()), h.auto()); + let err = timeout(Duration::from_secs(15), watcher.task) + .await + .expect("the watch must fail, not hang") + .unwrap() + .expect_err("a stale artifact must fail the watch"); + assert!(err.to_string().contains("retention window"), "{err}"); + let (after, fold) = h.open("watcher.snap"); + assert_eq!(after, cursor); + assert!(fold.get("route.keep").unwrap().is_some()); +} + +/// Counts fetches through to a real `ArtifactRestore` (proves the restore +/// path ran). +struct Counted { + inner: ArtifactRestore, + fetches: std::sync::atomic::AtomicUsize, +} + +#[async_trait::async_trait] +impl slipstream::RestoreSource for Counted { + async fn latest(&self) -> Result { + self.inner.latest().await + } + async fn fetch(&self) -> Result, SnapshotError> { + self.fetches.fetch_add(1, Ordering::SeqCst); + self.inner.fetch().await + } +} + +/// THE LIVE ROUTE of a mid-watch floor-guard trip into the restore, on a +/// real `discard: old` bucket. The exporter and the watcher both run the +/// resumed, floor-guarded watch. The watcher is stalled (its `apply` blocks; +/// a 1-deep channel backs up into the NATS consumer, which stops answering +/// flow control, so the server stops pushing — measured at ~2 MB), and a +/// 20 MB burst overruns it: `discard: old` evicts everything it hadn't been +/// sent, `route.keep`'s current value included. Released, the watcher +/// drains what it had, the floor guard sees the gap and trips with +/// `CursorExpired`, and `watch_applied` restores in process from the +/// exporter's artifact — then matches the exporter exactly. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn live_floor_guard_trip_restores_in_process() { + let h = Harness::new(StoreConfig { + name: "routes".into(), + max_bytes: Some(4 << 20), + discard: DiscardPolicy::Old, + ..Default::default() + }) + .await; + let w = h.bucket.writer().unwrap(); + let restore = Arc::new(Counted { + inner: ArtifactRestore::new( + Arc::clone(&h.transport), + KEY, + h.dir.path(), + |artifact: &Path, dest: &Path| AppendLogSnapshot::import(artifact, dest, u64::MAX), + ), + fetches: std::sync::atomic::AtomicUsize::new(0), + }); + let auto = |restore: &Arc| ExpiryRepair::Auto { + reader: h.bucket.reader(), + restore: Arc::clone(restore) as Arc>, + }; + + // Seed, and fold it on both nodes via a first run, so both restart on + // the resumed (floor-guarded) watch. + w.put("route.keep", b"v1").await.unwrap(); + let seed = w.put("route.seed", b"s").await.unwrap().as_u64().unwrap(); + for name in ["exporter.snap", "watcher.snap"] { + let (_c, fold) = h.open(name); + let node = Node::spawn(&h.bucket, fold, None, auto(&restore)); + node.wait_applied(seed).await; + node.stop().await.unwrap(); + } + let (c, fold) = h.open("exporter.snap"); + let exporter = Node::spawn(&h.bucket, fold, Some(c), auto(&restore)); + let (c, fold) = h.open("watcher.snap"); + let watcher = Node::spawn_with( + &h.bucket, + fold, + Some(c), + auto(&restore), + BatchConfig { + channel_capacity: 1, + max: 1, + ..BatchConfig::default() + }, + ); + + // Stall the watcher, then overrun it. + watcher.gate.store(true, Ordering::SeqCst); + let value = vec![b'x'; 1024]; + let mut head = 0; + for i in 0..20_000 { + head = w + .put(&format!("burst.{i}"), &value) + .await + .unwrap() + .as_u64() + .unwrap(); + } + let reader = h.bucket.reader(); + assert!( + reader.get("route.keep").await.unwrap().is_none(), + "route.keep must have been evicted" + ); + exporter.wait_applied(head).await; + let artifact = exporter.publish(&*h.transport, h.dir.path()).await; + assert_eq!(artifact, head, "the exporter kept up and exported the head"); + + // Release: drain → trip → restore → resume. + watcher.gate.store(false, Ordering::SeqCst); + watcher.wait_applied(head).await; + assert!( + restore.fetches.load(Ordering::SeqCst) >= 1, + "the watcher must have been overrun and repaired by a restore" + ); + let end = watcher + .stop() + .await + .expect("the trip was repaired in process"); + assert_eq!(end.as_u64(), Some(head)); + exporter.stop().await.unwrap(); + + let (wc, wfold) = h.open("watcher.snap"); + let (xc, xfold) = h.open("exporter.snap"); + assert_eq!(wc, xc); + assert_eq!( + wfold.get("route.keep").unwrap().map(|e| e.value), + Some(b"v1".to_vec()), + "the aged-out key survives the restore" + ); + let entries = |f: &AppendLogSnapshot| -> Vec<(String, Vec)> { + f.range("") + .unwrap() + .into_iter() + .map(|e| (e.key, e.value)) + .collect() + }; + let (we, xe) = (entries(&wfold), entries(&xfold)); + assert_eq!(we.len(), 20_002); + assert!( + we == xe, + "the restored watcher is identical to the exporter" + ); +} diff --git a/tests/model.rs b/tests/model.rs index 98ce74e..9fa223a 100644 --- a/tests/model.rs +++ b/tests/model.rs @@ -27,7 +27,15 @@ //! current pointer's target is never pruned. //! - The source stream with **retention**: a floor that advances freely; //! resuming below the floor is `CursorExpired`; delete markers at or below -//! the floor are evicted (the re-list cannot see them). +//! the floor are evicted (the re-list cannot see them). Under +//! `BucketRetention::EvictsCurrent` (`max_age`, `discard: old`) retention +//! ALSO evicts the sentinel key's current value (`AgeOut`, FIFO-consistent: +//! only once the floor has passed that value's revision), so the bucket +//! stops listing a key nobody deleted. +//! - The sentinel key's full history: its initial put, an optional real +//! delete, and an optional later write (`PutKey` — an update, or a +//! re-create after the delete). A write made while the importer is behind +//! that then ages out is exactly the gap write a re-list can never deliver. //! - A bootstrapping importer whose two reads (sibling manifest, then //! payload) interleave with all of the above, whose cross-check compares //! them, and whose post-import resume either replays the tail, or falls @@ -37,6 +45,23 @@ //! (`FailStop` — `applied.rs` as it ships). The checker proves `Degrade` //! breaks the convergence theorem, which is the machine-checked //! justification for the fail-stop change in `resync_stale_keys`. +//! `FailStop` repairs the way `ExpiryRepair::Auto` does: the key-listing +//! diff on a bucket that keeps current values, an ARTIFACT RESTORE on one +//! that evicts them (`Restoring` → `RestoreRead`, gated by the shared +//! `protocol::restore_allowed` kernel: ahead of the local fold and still +//! inside retention). The pre-fix behavior — the key-listing diff on an +//! evicting bucket — is the `RelistOnEvicting` mutation, which the checker +//! must catch. +//! +//! ## What "converged" means +//! +//! The fold is correct when it holds **every write minus every real delete** +//! (`key_at(head)`), not when it matches what NATS still lists. The two agree +//! on a bucket that keeps current values; on one that evicts them they +//! diverge, and the pre-fix resync — which defined correct as "matches +//! NATS" — deleted valid keys to match. `FoldStatus` judges against the +//! truth: `StaleKey` holds a really-deleted key, `Lost` lacks a write the +//! truth has (dropped a key that aged out, or missed an aged-out gap write). //! //! The empirical coupling runs both directions: the legacy configuration's //! `sometimes` hazards are the interleavings `tests/multi_export.rs` drives @@ -82,8 +107,16 @@ //! (empirical tier: tampered-artifact and multi-SST round-trip tests). //! NATS KV CAS semantics for the lease are unneeded here (see above); the //! lease layer is verified by `integration.rs` contention tests. -//! 5. Retention outlives consumer lag — NARROWED to prefix-scoped watches -//! and the fresh full watch's initial history scan. The ALL-scope resume +//! 5. Exporters are correct folds: an artifact holds `key_at(cursor)`. +//! DISCHARGED by `tests/model_fleet.rs`, where the exporters are watchers +//! that themselves expire, restore, start cursor-less, and publish their +//! actual folds, and the checker proves every published artifact is the +//! truth at its cursor. +//! 6. Retention outlives consumer lag — NARROWED to prefix-scoped watches, +//! the fresh full watch's initial history scan, and the key-listing +//! repair's window between its listing and its re-list +//! (`tests/model_repair.rs` drops it and reaches that trace; the artifact +//! restore does not depend on it). The ALL-scope resume //! watch (steady-state operation) no longer relies on it: the live floor //! guard (`tests/model_live_watch.rs`, `stream_watch_floor_guarded`) //! fail-stops on in-band evidence of retention overrunning the consumer @@ -96,13 +129,17 @@ //! ## Bounds and the small-scope argument //! //! Default: 2 exporters (a const-generic parameter), revisions ≤ 3, one -//! importer, one deletable sentinel key — and **unbounded rounds**: a +//! importer, one sentinel key (initial put, at most one delete, at most one +//! later write, and — on an evicting bucket — aging out) — and **unbounded +//! rounds**: a //! publisher re-enters the pipeline whenever it has applied past its last //! publish (`NextRound`), so every theorem quantifies over repeated rounds, //! including a node racing its own previous publish. Every hazard class //! needs at most: two distinct cursors (regression), one crash window (torn -//! pair), one delete + floor advance (stale key) — and every `sometimes` -//! witness fails loudly if a bound ever clips its scenario. +//! pair), one delete + floor advance (stale key), one write + age-out + floor +//! advance (lost write) — and every `sometimes` witness fails loudly if a +//! bound ever clips its scenario. The evicting configuration's deep tier +//! (revisions ≤ 4, ~124M unique states) is ignored by default. //! //! The ignored deep tier (release mode, scheduled runs) pushes both axes: //! revisions ≤ 5 (~154M unique states) and THREE exporters (~95M unique @@ -122,6 +159,10 @@ use stateright::{Checker, Model, Property}; +/// The convergence theorem's name (looked up by the mutation tests). +const DIVERGENCE_THEOREM: &str = + "bootstrap never silently diverges from every write minus every real delete"; + /// Default bucket-revision bound (1..=MAX_REV). 3 suffices for every hazard /// class and witness (two distinct export cursors plus a delete revision — /// each `sometimes` property fails loudly if a bound ever clips its @@ -130,13 +171,14 @@ use stateright::{Checker, Model, Property}; const MAX_REV: u8 = 3; /// An export artifact's identity: who exported, at which applied cursor, and -/// whether the sentinel key was still present at that cursor. Two artifacts -/// are byte-identical iff this identity is equal (BLAKE3 axiom). +/// the sentinel key's value at that cursor (the revision of the put it holds, +/// `None` if deleted). Two artifacts are byte-identical iff this identity is +/// equal (BLAKE3 axiom). #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] struct Artifact { node: u8, cursor: u8, - has_key: bool, + key: Option, } #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] @@ -154,13 +196,18 @@ enum ExporterPc { #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] enum FoldStatus { - /// The bootstrapped fold converges to the bucket (tail replay, or - /// fallback with resync, or fallback where the re-list happens to cover). + /// The bootstrapped fold holds every write minus every real delete (tail + /// replay, a repaired fallback, or a fallback the re-list happens to + /// cover). Synced, - /// Silent divergence: the fold holds a key the bucket deleted, and + /// Silent divergence: the fold holds a key that was really deleted, and /// nothing will ever remove it (expired cursor + evicted marker + no /// resync). StaleKey, + /// Silent divergence: the fold lacks a write the truth has — a key that + /// aged out of NATS (never deleted) was dropped, or a write made during + /// the gap aged out before the fold could see it. + Lost, } #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)] @@ -180,6 +227,12 @@ enum ImporterPc { FetchMissed, /// Resume ran; final verdict on this bootstrap. Resumed(Artifact, FoldStatus), + /// Resume hit an expired cursor on a bucket that evicts current values: + /// the fold (at this artifact) must be restored from a newer published + /// artifact still inside retention. Fail-stop until one exists — in code + /// the watch fails and the caller's restart retries; here `RestoreRead` + /// is simply disabled until the pointer qualifies. + Restoring(Artifact), } #[derive(Clone, Debug, Hash, PartialEq, Eq)] @@ -188,6 +241,12 @@ struct St { head: u8, /// Revision at which the sentinel key was deleted, if it was. delete_rev: Option, + /// Revision of the sentinel's later write (`PutKey`), if any. Its initial + /// put is revision 0, before every replica's first cursor. + reput_rev: Option, + /// `EvictsCurrent` only: retention evicted the sentinel's CURRENT value + /// (it was never deleted; the bucket just stopped listing it). + aged: bool, /// Stream retention floor: resuming from cursor < floor is CursorExpired; /// a delete marker at rev ≤ floor has been evicted. floor: u8, @@ -207,6 +266,11 @@ struct St { /// FIXED: latched when the monotonic swap refused an older publish — /// vacuity witness that the guard actually fires within the bounds. refused: bool, + /// Latched when the importer's fold was restored from a newer artifact + /// (vacuity witness for the restore theorem). + restored: bool, + /// `TrustRestoredCursor` only: the next resume skips its window check. + trust_next_resume: bool, importer: ImporterPc, } @@ -216,6 +280,11 @@ enum Act { Churn, /// The sentinel key is deleted (consumes a revision). DeleteKey, + /// The sentinel key is written again (consumes a revision): an update, or + /// a re-create after the delete. + PutKey, + /// `EvictsCurrent` only: retention evicts the sentinel's current value. + AgeOut, /// Retention floor advances by one. Compact, /// Replica n applies the next revision. @@ -240,12 +309,29 @@ enum Act { Prune, Retry, Resume, + /// `Restoring` only: read the current pointer and, if + /// `protocol::restore_allowed` accepts it, import it. Collapses the + /// payload fetch: the CURRENT pointer's target is always fetchable + /// (`pointer target always fetchable`), and the stale-read fetch miss is + /// already explored on the initial import. + RestoreRead, /// Degrade mode only: the resume completed but its resync failed /// mid-flight and the code warned-and-continued re-list-only. ResumeResyncDegraded, } /// How the bootstrapping node handles the cursor-expired stale-key resync. +/// What the bucket's retention can evict. +#[derive(Clone, Copy, PartialEq, Eq)] +enum BucketRetention { + /// Only superseded history and delete markers (`discard: new`, no + /// `max_age`): a key missing from the bucket was deleted. + KeepsCurrent, + /// Current values too (`max_age`, `discard: old`): a key missing from the + /// bucket may simply have aged out. + EvictsCurrent, +} + #[derive(Clone, Copy, PartialEq, Eq)] enum ResyncMode { /// No reader wired (`watch_applied(reader: None, ..)`): expiry falls back @@ -282,6 +368,16 @@ enum Mutation { /// silent clamp (gap skipped, resync never triggered) — the live bug /// `tests/resync.rs` pinned. SilentClamp, + /// The pre-fix repair: the key-listing diff even on a bucket that evicts + /// current values (absence from NATS taken as deletion). The bug + /// `tests/eviction.rs` reproduced live. + RelistOnEvicting, + /// The restore accepts any published artifact (no + /// `protocol::restore_allowed`) — the resume window re-check still runs. + NoRestoreGuard, + /// The resume after a restore skips its window check, trusting the + /// restored cursor (the restore guard still runs). + TrustRestoredCursor, } /// Model parameters: which protocol, the importer's resync mode, an optional @@ -290,6 +386,7 @@ enum Mutation { struct SnapshotProtocol { pointer_swap: bool, resync: ResyncMode, + retention: BucketRetention, mutation: Mutation, max_rev: u8, } @@ -299,26 +396,60 @@ impl SnapshotProtocol { Self { pointer_swap: true, resync, + retention: BucketRetention::KeepsCurrent, mutation: Mutation::None, max_rev: MAX_REV, } } + /// The shipped protocol on a bucket whose retention evicts current + /// values. + fn evicting(resync: ResyncMode) -> Self { + Self { + retention: BucketRetention::EvictsCurrent, + ..Self::shipped(resync) + } + } + fn legacy(resync: ResyncMode) -> Self { Self { pointer_swap: false, - resync, - mutation: Mutation::None, - max_rev: MAX_REV, + ..Self::shipped(resync) + } + } + + /// THE TRUTH: the sentinel's value after every write and every real + /// delete up to `cursor` — the revision of the put it holds, `None` if + /// deleted. Initial put at 0; the delete and the later write apply in + /// revision order. + fn key_at(s: &St, cursor: u8) -> Option { + let mut key = Some(0); + let mut events = [ + s.delete_rev.map(|d| (d, None)), + s.reput_rev.map(|r| (r, Some(r))), + ]; + events.sort(); + for (rev, value) in events.into_iter().flatten() { + if rev <= cursor { + key = value; + } } + key } - fn key_present_at(s: &St, cursor: u8) -> bool { - s.delete_rev.is_none_or(|d| cursor < d) + /// What NATS still lists for the sentinel (its live-key listing, and the + /// current value a re-list delivers): the truth, minus a current value + /// retention evicted. + fn listed(s: &St) -> Option { + Self::key_at(s, s.head).filter(|_| !s.aged) } - fn bucket_has_key(s: &St) -> bool { - s.delete_rev.is_none() + fn status(fold: Option, truth: Option) -> FoldStatus { + match (fold, truth) { + (f, t) if f == t => FoldStatus::Synced, + (Some(_), None) => FoldStatus::StaleKey, + _ => FoldStatus::Lost, + } } /// The expiry guard — THE SHARED KERNEL (`slipstream::protocol`), the @@ -330,17 +461,30 @@ impl SnapshotProtocol { } /// Outcome of an expired-cursor fallback WITHOUT a working resync: the - /// re-list delivers current values only, so a key deleted during the gap - /// is covered iff its delete marker survived retention (delete_rev > - /// floor — the fallback watch replays retained history, markers - /// included). + /// re-list delivers the current values NATS still holds, so a key deleted + /// during the gap is covered iff its delete marker survived retention + /// (delete_rev > floor, and no later write superseded it — the fallback + /// watch replays retained history, markers included); anything else the + /// re-list can't deliver (an aged-out current value) leaves the + /// artifact's value in place. fn relist_only_status(s: &St, a: Artifact) -> FoldStatus { - let marker_evicted = s.delete_rev.is_some_and(|d| d <= s.floor); - if a.has_key && !Self::bucket_has_key(s) && marker_evicted { - FoldStatus::StaleKey + let truth = Self::key_at(s, s.head); + let marker_delivered = truth.is_none() && s.delete_rev.is_some_and(|d| d > s.floor); + let fold = if Self::listed(s).is_some() { + truth + } else if marker_delivered { + None } else { - FoldStatus::Synced - } + a.key + }; + Self::status(fold, truth) + } + + /// Outcome of the key-listing resync: synthetic deletes for whatever NATS + /// no longer lists, then the re-list. The fold becomes the LISTING — + /// the truth only on a bucket that keeps current values. + fn relist_status(s: &St) -> FoldStatus { + Self::status(Self::listed(s), Self::key_at(s, s.head)) } } @@ -352,6 +496,8 @@ impl Model for SnapshotProtocol { vec![St { head: 0, delete_rev: None, + reput_rev: None, + aged: false, floor: 0, applied: [0; N], exporters: [ExporterPc::Idle; N], @@ -360,6 +506,8 @@ impl Model for SnapshotProtocol { uploaded: Default::default(), regressed: false, refused: false, + restored: false, + trust_next_resume: false, importer: ImporterPc::Start, }] } @@ -370,6 +518,17 @@ impl Model for SnapshotProtocol { if s.delete_rev.is_none() { acts.push(Act::DeleteKey); } + if s.reput_rev.is_none() { + acts.push(Act::PutKey); + } + } + // FIFO eviction: a current value ages out only once everything up to + // its revision has (the initial put, at 0, predates every message). + if self.retention == BucketRetention::EvictsCurrent + && !s.aged + && Self::key_at(s, s.head).is_some_and(|w| w == 0 || s.floor >= w) + { + acts.push(Act::AgeOut); } if s.floor < s.head { acts.push(Act::Compact); @@ -418,6 +577,24 @@ impl Model for SnapshotProtocol { acts.push(Act::ResumeResyncDegraded); } } + // The restore guard — THE SHARED KERNEL + // (`slipstream::protocol::restore_allowed`), the same function + // `applied.rs`'s restore path executes (`check_restore`): the + // pointer must be ahead of the local fold and still inside + // retention. Otherwise fail-stop: no enabled action until the + // fleet publishes one that is. + ImporterPc::Restoring(a) => { + if let Some(m) = s.manifest + && (self.mutation == Mutation::NoRestoreGuard + || slipstream::protocol::restore_allowed( + m.cursor as u64, + a.cursor as u64, + s.floor as u64 + 1, + )) + { + acts.push(Act::RestoreRead); + } + } _ => {} } } @@ -430,6 +607,12 @@ impl Model for SnapshotProtocol { s.head += 1; s.delete_rev = Some(s.head); } + Act::PutKey => { + s.head += 1; + s.reput_rev = Some(s.head); + s.aged = false; // a fresh current value + } + Act::AgeOut => s.aged = true, Act::Compact => s.floor += 1, Act::Apply(n) => s.applied[n as usize] += 1, Act::Export(n) => { @@ -437,7 +620,7 @@ impl Model for SnapshotProtocol { s.exporters[n as usize] = ExporterPc::Exported(Artifact { node: n, cursor, - has_key: Self::key_present_at(&s, cursor), + key: Self::key_at(&s, cursor), }); } Act::UploadPayload(n) => { @@ -561,31 +744,68 @@ impl Model for SnapshotProtocol { let ImporterPc::Imported(a) = s.importer else { return None; }; - let status = if self.resume_ok(&s, a) { + let trusted = std::mem::take(&mut s.trust_next_resume); + if trusted && !self.resume_ok(&s, a) { + // The unchecked resume from a restored cursor that + // retention overran since the restore's check: NATS + // silently clamps, skipping the evicted gap. + s.importer = ImporterPc::Resumed(a, Self::relist_only_status(&s, a)); + } else if self.resume_ok(&s, a) { // Window intact (shared kernel `resume_window_ok` — the // same guard `nats.rs` executes): tail replay from the - // embedded cursor. Any delete in the gap has its marker - // retained (delete_rev > a.cursor >= floor), so the - // replay delivers it. - FoldStatus::Synced + // embedded cursor delivers every event past it (all + // retained: > a.cursor >= floor), so the fold reaches + // the truth. A current value that aged out has revision + // <= floor <= a.cursor, so the artifact already holds it. + s.importer = ImporterPc::Resumed(a, FoldStatus::Synced); } else if self.mutation == Mutation::SilentClamp { // Expiry detection removed: the resume silently skips // the gap (NATS's native clamp behavior) — deletes whose // markers were evicted are lost and the resync never // triggers, regardless of the resync mode. - Self::relist_only_status(&s, a) - } else if self.resync != ResyncMode::None { - // CursorExpired -> full re-list + a SUCCESSFUL stale-key - // resync: live keys diffed against the fold, vanished - // keys get synthetic deletes. (Under FailStop a failed - // resync fails the watch and changes nothing — this - // action stays enabled for the retry. Under Degrade the - // failed-resync outcome is ResumeResyncDegraded.) - FoldStatus::Synced + s.importer = ImporterPc::Resumed(a, Self::relist_only_status(&s, a)); + } else if self.resync == ResyncMode::None { + s.importer = ImporterPc::Resumed(a, Self::relist_only_status(&s, a)); + } else if self.retention == BucketRetention::EvictsCurrent + && self.mutation != Mutation::RelistOnEvicting + { + // CursorExpired on a bucket that evicts current values: + // NATS's listing can't separate deleted from aged out, + // so the fold is restored from an artifact + // (`ExpiryRepair::Auto` → Restore). + s.importer = ImporterPc::Restoring(a); } else { - Self::relist_only_status(&s, a) + // CursorExpired -> full re-list + a SUCCESSFUL key- + // listing resync: live keys diffed against the fold, + // vanished keys get synthetic deletes. (Under FailStop a + // failed resync fails the watch and changes nothing — + // this action stays enabled for the retry. Under Degrade + // the failed-resync outcome is ResumeResyncDegraded.) + // Sound exactly when the listing is the truth. + s.importer = ImporterPc::Resumed(a, Self::relist_status(&s)); + } + } + Act::RestoreRead => { + let ImporterPc::Restoring(a) = s.importer else { + return None; }; - s.importer = ImporterPc::Resumed(a, status); + let m = s.manifest?; + if self.mutation != Mutation::NoRestoreGuard + && !slipstream::protocol::restore_allowed( + m.cursor as u64, + a.cursor as u64, + s.floor as u64 + 1, + ) + { + return None; + } + s.trust_next_resume = self.mutation == Mutation::TrustRestoredCursor; + // The in-scope fold becomes the artifact's; the watch then + // resumes from its cursor (the next `Resume`, which re-checks + // the window — retention can still overrun it, and the + // restore runs again for a newer pointer). + s.restored = true; + s.importer = ImporterPc::Imported(m); } Act::ResumeResyncDegraded => { let ImporterPc::Imported(a) = s.importer else { @@ -620,6 +840,7 @@ impl Model for SnapshotProtocol { // from an earlier pointer read has cursor <= m.cursor. (ImporterPc::GotManifest(g), Some(m)) | (ImporterPc::Imported(g), Some(m)) + | (ImporterPc::Restoring(g), Some(m)) | (ImporterPc::Resumed(g, _), Some(m)) => g.cursor <= m.cursor, _ => true, }, @@ -684,30 +905,37 @@ impl Model for SnapshotProtocol { props.push(Property::::always( "no mixed import: an installed fold is one exporter's exported state", |_, s| match s.importer { - ImporterPc::Imported(a) | ImporterPc::Resumed(a, _) => { + ImporterPc::Imported(a) | ImporterPc::Restoring(a) | ImporterPc::Resumed(a, _) => { a.node < N as u8 && a.cursor >= 1 && a.cursor <= s.head - // The artifact's key-set is exactly the bucket state - // at its cursor — imports never Frankenstein. - && a.has_key == s.delete_rev.is_none_or(|d| a.cursor < d) + // The artifact's key-set is exactly the truth at its + // cursor — imports never Frankenstein. + && a.key == Self::key_at(s, a.cursor) } _ => true, }, )); + fn diverged(s: &St) -> bool { + matches!( + s.importer, + ImporterPc::Resumed(_, FoldStatus::StaleKey | FoldStatus::Lost) + ) + } match self.resync { ResyncMode::FailStop => { - // ---- THE convergence claim: resync wired with fail-stop - // error semantics (`applied.rs` as it ships): bootstrap NEVER - // silently diverges — over every interleaving of churn, - // deletes, compaction, crashes, racing exporters, and resync - // failures (a failed resync fails the watch; only a - // successful one completes a bootstrap). ------------------- - props.push(Property::::always( - "bootstrap never silently diverges (stale is merely stale)", - |_, s| !matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::StaleKey)), - )); + // ---- THE convergence claim: repair wired with fail-stop + // error semantics (`applied.rs` as it ships, choosing like + // `ExpiryRepair::Auto`): a bootstrap NEVER silently diverges + // from every write minus every real delete — over every + // interleaving of churn, writes, deletes, compaction, aging + // out, crashes, racing exporters, and repair failures (a + // failed repair fails the watch; only a successful one + // completes a bootstrap). --------------------------------- + props.push(Property::::always(DIVERGENCE_THEOREM, |_, s| { + !diverged(s) + })); } ResyncMode::Degrade => { // ---- The PRE-FIX code semantics (resync failure → warn → @@ -718,7 +946,7 @@ impl Model for SnapshotProtocol { // remains an honest record of why. props.push(Property::::sometimes( "HAZARD reachable: armed resync that degrades on error diverges silently", - |_, s| matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::StaleKey)), + |_, s| diverged(s), )); } ResyncMode::None => { @@ -730,9 +958,45 @@ impl Model for SnapshotProtocol { "HAZARD reachable: silent stale-key divergence without resync", |_, s| matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::StaleKey)), )); + if self.retention == BucketRetention::EvictsCurrent { + // And the eviction hazard: the re-list can't deliver a + // gap write whose current value aged out. + props.push(Property::::sometimes( + "HAZARD reachable: the re-list alone misses a write that aged out", + |_, s| matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::Lost)), + )); + } } } + if self.retention == BucketRetention::EvictsCurrent + && self.resync == ResyncMode::FailStop + && self.mutation == Mutation::None + { + // Vacuity witnesses for the theorem on an evicting bucket: it is + // earned by restores that really happen past evicted current + // values — an aged-out key that was never deleted, and an + // aged-out later write — not by the hazard never arising. + props.push(Property::::sometimes( + "a restore resumes synced past a key that aged out but was never deleted", + |_, s| { + s.restored + && s.aged + && matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::Synced)) + }, + )); + props.push(Property::::sometimes( + "a restore recovers a later write that aged out", + |_, s| { + s.restored + && s.aged + && s.reput_rev.is_some() + && s.delete_rev.is_none() + && matches!(s.importer, ImporterPc::Resumed(_, FoldStatus::Synced)) + }, + )); + } + // Vacuity witness for every always-property above: bootstraps really // complete in this configuration (Imported and Resumed are reachable, // so the invariants quantify over live states, not an empty set). @@ -810,6 +1074,77 @@ fn shipped_protocol_pointer_swap_failstop_resync() { ); } +/// THE EVICTION FIX: on a bucket whose retention evicts current values, the +/// shipped repair (artifact restore, gated by `protocol::restore_allowed`) +/// keeps every write minus every real delete — aged-out keys kept, aged-out +/// gap writes recovered — and every maximal run still ends synced. +#[test] +fn evicting_bucket_restores_from_artifact() { + check( + SnapshotProtocol::<2>::evicting(ResyncMode::FailStop), + "evicting: pointer-swap + restore", + ); +} + +/// The pre-fix repair on an evicting bucket — the key-listing diff, which +/// takes "NATS no longer lists it" as "deleted" — MUST be caught: the checker +/// produces a fold that lost a write (a valid key deleted, or an aged-out gap +/// write never seen). +#[test] +fn mutation_relist_on_evicting_bucket_is_caught() { + let mut model = SnapshotProtocol::<2>::evicting(ResyncMode::FailStop); + model.mutation = Mutation::RelistOnEvicting; + let checker = run(model, "mutation: relist on evicting bucket"); + let path = checker + .discovery(DIVERGENCE_THEOREM) + .expect("the checker must find the relist-on-evicting divergence"); + assert!( + matches!( + path.last_state().importer, + ImporterPc::Resumed(_, FoldStatus::Lost) + ), + "the counterexample is a LOST write (not a stale key): {:?}", + path.last_state().importer + ); +} + +/// The restore's two defenses, separated. The window re-check on the resume +/// that follows a restore is LOAD-BEARING: retention can overrun the +/// restored cursor between the restore's check and the resume, and without +/// the re-check that gap is skipped silently. The restore guard +/// (`restore_allowed`) is FAIL-FAST: without it a stale artifact is imported +/// and the resume refuses it, so the theorem still holds (the guard turns a +/// pointless download-and-retry into an immediate, explained refusal). +#[test] +fn restore_defenses_resume_recheck_is_load_bearing_guard_is_fail_fast() { + // One exporter suffices: the race is retention vs the importer. + let mut model = SnapshotProtocol::<1>::evicting(ResyncMode::FailStop); + model.mutation = Mutation::TrustRestoredCursor; + let checker = run(model, "mutation: trust restored cursor"); + assert!( + checker.discovery(DIVERGENCE_THEOREM).is_some(), + "skipping the post-restore window check must be caught" + ); + + let mut model = SnapshotProtocol::<1>::evicting(ResyncMode::FailStop); + model.mutation = Mutation::NoRestoreGuard; + let checker = run(model, "mutation: no restore guard"); + assert!( + checker.discovery(DIVERGENCE_THEOREM).is_none(), + "the restore guard is fail-fast, not the safety gate: without it the resume re-check \ + still refuses a stale artifact" + ); +} + +/// No repair on an evicting bucket: both divergence classes reachable. +#[test] +fn evicting_bucket_without_repair_diverges() { + check( + SnapshotProtocol::<2>::evicting(ResyncMode::None), + "evicting: no repair", + ); +} + /// The shipped transport still requires the resync reader — the pointer-swap /// fix does not absolve the convergence obligation. #[test] @@ -895,9 +1230,7 @@ fn mutation_silent_clamp_is_caught() { model.mutation = Mutation::SilentClamp; let checker = run(model, "mutation: silent clamp"); assert!( - checker - .discovery("bootstrap never silently diverges (stale is merely stale)") - .is_some(), + checker.discovery(DIVERGENCE_THEOREM).is_some(), "the checker must produce a silent-divergence counterexample when \ expiry detection is removed (the live NATS clamp bug class)" ); @@ -917,6 +1250,16 @@ fn deep_more_revisions() { check(model, "deep: 2 exporters, rev <= 5"); } +/// The eviction axis at more revisions: restores racing more churn, aging, +/// and compaction. +#[test] +#[ignore = "deep bounds: run in release"] +fn deep_evicting_more_revisions() { + let mut model = SnapshotProtocol::<2>::evicting(ResyncMode::FailStop); + model.max_rev = 4; + check(model, "deep: evicting, 2 exporters, rev <= 4"); +} + /// THREE exporters: the classic check that nothing in the protocol is /// accidentally pairwise — three-way publish races, two stalled rounds /// landing after a third's takeover, prune racing two concurrent uploads. diff --git a/tests/model_fleet.rs b/tests/model_fleet.rs new file mode 100644 index 0000000..393f2d3 --- /dev/null +++ b/tests/model_fleet.rs @@ -0,0 +1,421 @@ +//! Exhaustive model-check of a FLEET repairing from each other's artifacts +//! (Stateright) — discharging the "exporters are correct folds" axiom the +//! other models assume. +//! +//! `tests/model.rs`, `tests/model_live_watch.rs` and `tests/model_repair.rs` +//! take an artifact at cursor `c` to hold the truth at `c`. In a real fleet +//! the exporter is just another watcher: it can itself expire, restore from +//! an older artifact, start with no cursor, and publish whatever its fold +//! holds. If any of those paths ever produced a wrong fold, the error would +//! propagate through artifacts to every node that restores. Here the nodes +//! ARE the exporters, and the checker proves the induction: +//! +//! - **Every published artifact is the truth at its cursor.** +//! - **Every live node's fold is the truth at its cursor.** +//! - **Every maximal run ends with every live node converged** to every +//! write minus every real delete. A node may instead be fail-stopped — and +//! then only because a write aged out before ANY node folded it: no fold +//! anywhere holds it, so there is nothing safe to restore. The checker +//! reaches that state (witness): it is the one data loss retention can +//! cause, and it is loud. +//! +//! Three nodes: two start synced, one starts with no cursor and an empty +//! fold (it seeds from an artifact once the bucket has evicted current +//! values). One key; writes, eviction (head eviction on an evicting bucket, +//! marker purge on one that keeps current values), floor-guarded delivery, +//! expiry, repair (restore via `protocol::restore_allowed`, or the +//! key-listing diff), and monotone publishes (`protocol:: +//! pointer_publish_allowed`). Repairs are atomic here; their steps are +//! checked in `tests/model_repair.rs`. +//! +//! Mutations the checker must catch: the key-listing repair on an evicting +//! bucket, and a cursor-less start that re-lists an already-evicted bucket — +//! each poisons the fleet's artifacts. + +use slipstream::protocol::{ + PointerState, pointer_publish_allowed, restore_allowed, resume_window_ok, +}; +use stateright::{Checker, Model, Property}; + +const CAP: usize = 8; +const N: usize = 3; + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +struct Node { + /// Not yet started (no cursor, empty fold). + fresh: bool, + /// The fold's value for the key (`Some(put revision)`). + fold: Option, + /// The watch frontier / fold cursor. + cursor: u8, + /// Expired and awaiting a repair (fail-stopped until one is possible). + expired: bool, + /// This node's fold came (at some point) from a restore. + restored: bool, +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq)] +struct St { + head: u8, + ops: [bool; CAP], + evicted: bool, + /// The published pointer: (cursor, value the artifact holds). + pointer: Option<(u8, Option)>, + nodes: [Node; N], + bad_publish: bool, + restored_exporter_published: bool, + seeded_from_artifact: bool, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum Act { + Put, + Delete, + Evict, + Start(usize), + Deliver(usize), + Expire(usize), + Repair(usize), + Publish(usize), +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Mutation { + None, + RelistOnEvicting, + FreshStartRelist, +} + +#[derive(Clone)] +struct Fleet { + evicting: bool, + mutation: Mutation, + /// The key's revisions run 1..=max_rev. + max_rev: u8, +} + +impl Fleet { + fn truth_at(s: &St, c: u8) -> Option { + let r = c.min(s.head); + (r >= 1 && s.ops[r as usize]).then_some(r) + } + fn truth(s: &St) -> Option { + Self::truth_at(s, s.head) + } + fn first_revision(s: &St) -> u64 { + if s.evicted { + s.head as u64 + 1 + } else { + s.head as u64 + } + } + fn listing_is_truth(s: &St) -> bool { + resume_window_ok(0, Self::first_revision(s)) + } + fn listed(s: &St) -> Option { + if s.evicted { None } else { Self::truth(s) } + } + /// The repair a node would take: the artifact restore, or the + /// key-listing diff (shipped: `ExpiryRepair::Auto`). + fn restores(&self, s: &St) -> bool { + self.evicting && self.mutation != Mutation::RelistOnEvicting && !Self::listing_is_truth(s) + } + fn restore_target(s: &St, n: &Node) -> Option<(u8, Option)> { + let (c, v) = s.pointer?; + restore_allowed(c as u64, n.cursor as u64, Self::first_revision(s)).then_some((c, v)) + } +} + +impl Model for Fleet { + type State = St; + type Action = Act; + + fn init_states(&self) -> Vec { + let mut ops = [false; CAP]; + ops[1] = true; + let synced = Node { + fresh: false, + fold: Some(1), + cursor: 1, + expired: false, + restored: false, + }; + let fresh = Node { + fresh: true, + fold: None, + cursor: 0, + expired: false, + restored: false, + }; + vec![St { + head: 1, + ops, + evicted: false, + pointer: None, + nodes: [synced, synced, fresh], + bad_publish: false, + restored_exporter_published: false, + seeded_from_artifact: false, + }] + } + + fn actions(&self, s: &St, acts: &mut Vec) { + if s.head < self.max_rev { + acts.push(Act::Put); + acts.push(Act::Delete); + } + let evictable = self.evicting || !s.ops[s.head as usize]; + if !s.evicted && evictable { + acts.push(Act::Evict); + } + for (i, n) in s.nodes.iter().enumerate() { + if n.fresh { + acts.push(Act::Start(i)); + continue; + } + if n.expired { + let possible = !self.restores(s) || Self::restore_target(s, n).is_some(); + if possible { + acts.push(Act::Repair(i)); + } + continue; + } + let window = resume_window_ok(n.cursor as u64, Self::first_revision(s)); + if window && s.head > n.cursor && !s.evicted { + acts.push(Act::Deliver(i)); + } + if !window { + acts.push(Act::Expire(i)); + } + let current = s.pointer.map(|(c, _)| PointerState::Present { + rank: Some(c as u64), + }); + if n.cursor >= 1 + && s.pointer.is_none_or(|(c, _)| c < n.cursor) + && pointer_publish_allowed( + ¤t.unwrap_or(PointerState::Absent), + n.cursor as u64, + ) + { + acts.push(Act::Publish(i)); + } + } + } + + fn next_state(&self, s: &St, a: Act) -> Option { + let mut s = s.clone(); + match a { + Act::Put | Act::Delete => { + s.head += 1; + s.ops[s.head as usize] = a == Act::Put; + s.evicted = false; + } + Act::Evict => s.evicted = true, + Act::Start(i) => { + let relist = self.mutation == Mutation::FreshStartRelist || !self.restores(&s); + let (listed, head) = (Fleet::listed(&s), s.head); + let n = &mut s.nodes[i]; + n.fresh = false; + if relist { + // Full re-list: the latest retained message, then live. + n.fold = listed; + n.cursor = head; + } else { + // An empty fold on an already-evicted bucket seeds from + // an artifact (`ExpiryRepair::Auto`). + n.expired = true; + } + } + Act::Deliver(i) => { + let (truth, head) = (Self::truth(&s), s.head); + let n = &mut s.nodes[i]; + n.fold = truth; + n.cursor = head; + } + Act::Expire(i) => s.nodes[i].expired = true, + Act::Repair(i) => { + if self.restores(&s) { + let (c, v) = Self::restore_target(&s, &s.nodes[i])?; + let n = &mut s.nodes[i]; + if n.cursor == 0 { + s.seeded_from_artifact = true; + } + n.fold = v; + n.cursor = c; + n.restored = true; + } else { + // Key-listing diff + re-list, positioned at the head. + let (listed, head) = (Self::listed(&s), s.head); + let n = &mut s.nodes[i]; + n.fold = listed; + n.cursor = head; + } + s.nodes[i].expired = false; + } + Act::Publish(i) => { + let n = s.nodes[i]; + if n.fold != Self::truth_at(&s, n.cursor) { + s.bad_publish = true; + } + if n.restored { + s.restored_exporter_published = true; + } + s.pointer = Some((n.cursor, n.fold)); + } + } + Some(s) + } + + fn properties(&self) -> Vec> { + let mut props = vec![ + Property::::always( + "every published artifact is the truth at its cursor", + |_, s| !s.bad_publish && s.pointer.is_none_or(|(c, v)| v == Fleet::truth_at(s, c)), + ), + Property::::always( + "every live node's fold is the truth at its cursor", + |_, s| { + s.nodes + .iter() + .filter(|n| !n.fresh && !n.expired) + .all(|n| n.fold == Fleet::truth_at(s, n.cursor)) + }, + ), + Property::::always( + "every maximal run ends with every live node converged, and a node fail-stops only if no fold anywhere holds the head", + |m, s| { + let mut acts = Vec::new(); + m.actions(s, &mut acts); + if !acts.is_empty() { + return true; + } + let truth = Fleet::truth(s); + let live_converged = s + .nodes + .iter() + .filter(|n| !n.expired) + .all(|n| n.fold == truth && n.cursor == s.head); + let any_expired = s.nodes.iter().any(|n| n.expired); + let no_fold_has_head = s.nodes.iter().all(|n| n.cursor < s.head) + && s.pointer.is_none_or(|(c, _)| c < s.head); + live_converged && (!any_expired || no_fold_has_head) + }, + ), + ]; + if self.mutation == Mutation::None { + // (A keeps-current fleet repairs by the key-listing diff and + // never restores; its witnesses are the repairs themselves.) + if self.evicting { + props.push(Property::::sometimes( + "a node that restored from an artifact publishes one (the induction is exercised)", + |_, s| s.restored_exporter_published, + )); + props.push(Property::::sometimes( + "a cursor-less node seeds from an artifact", + |_, s| s.seeded_from_artifact, + )); + props.push(Property::::sometimes( + "a write ages out before any node folds it: the fleet fail-stops, loudly", + |m, s| { + let mut acts = Vec::new(); + m.actions(s, &mut acts); + acts.is_empty() && s.nodes.iter().all(|n| n.expired) + }, + )); + } + } + props + } +} + +fn run(model: Fleet, label: &str) -> impl Checker { + let checker = model.checker().spawn_bfs().join(); + println!( + "{label}: {} states, {} unique", + checker.state_count(), + checker.unique_state_count() + ); + checker +} + +#[test] +fn evicting_fleet_artifacts_stay_correct() { + run( + Fleet { + evicting: true, + mutation: Mutation::None, + max_rev: 4, + }, + "fleet: evicting", + ) + .assert_properties(); +} + +#[test] +fn keeps_current_fleet_artifacts_stay_correct() { + run( + Fleet { + evicting: false, + mutation: Mutation::None, + max_rev: 4, + }, + "fleet: keeps-current", + ) + .assert_properties(); +} + +#[test] +fn mutations_poison_the_fleet_and_are_caught() { + for (mutation, label) in [ + ( + Mutation::RelistOnEvicting, + "fleet mutation: relist on evicting", + ), + ( + Mutation::FreshStartRelist, + "fleet mutation: cursor-less start re-lists", + ), + ] { + let checker = run( + Fleet { + evicting: true, + mutation, + max_rev: 4, + }, + label, + ); + assert!( + checker + .discovery("every published artifact is the truth at its cursor") + .is_some() + || checker + .discovery("every live node's fold is the truth at its cursor") + .is_some(), + "{label}: the checker must find a wrong fold or a poisoned artifact" + ); + println!( + " poisoned artifact reachable: {}", + checker + .discovery("every published artifact is the truth at its cursor") + .is_some() + ); + } +} + +/// Deeper bounds: one more revision. +/// `cargo test --release --test model_fleet -- --ignored` +#[test] +#[ignore = "deep bounds: run in release"] +fn deep_fleet() { + for evicting in [true, false] { + run( + Fleet { + evicting, + mutation: Mutation::None, + max_rev: 6, + }, + &format!("deep fleet: evicting={evicting}, rev <= 6"), + ) + .assert_properties(); + } +} diff --git a/tests/model_live_watch.rs b/tests/model_live_watch.rs index a62b6c5..dbbe925 100644 --- a/tests/model_live_watch.rs +++ b/tests/model_live_watch.rs @@ -1,32 +1,50 @@ //! Exhaustive model-check of the LIVE-WATCH retention race (Stateright) — -//! the axiom-5 boundary of `tests/model.rs`, now mechanized. +//! the axiom-6 boundary of `tests/model.rs`, now mechanized. //! //! The hazard: a live consumer's position can fall behind the stream's //! retention floor. JetStream then silently skips evicted messages (the same //! clamp behavior `tests/resync.rs` pins for resumes — consumers never error //! on evicted messages, they just never see them). A skipped DELETE marker -//! leaves the fold holding a key the bucket deleted, permanently and -//! silently: the exact failure class the resume-time `check_resume_window` -//! eliminates, alive again mid-watch. +//! leaves the fold holding a key that was deleted, permanently and silently: +//! the exact failure class the resume-time `check_resume_window` eliminates, +//! alive again mid-watch. //! -//! The fix this model dictates: a periodic **floor guard** during live -//! consumption — the same shared kernel guard (`protocol::resume_window_ok`) -//! applied to the delivered frontier instead of a resume cursor. A trip -//! fails the watch; the caller's restart routes into the resume → expiry → -//! resync path whose convergence the main model already proves. The repair -//! composite here (fold := bucket truth, frontier := floor) is exactly that -//! verified subprotocol, collapsed to one transition. +//! The fix this model dictates: a **floor guard** during live consumption — +//! the same shared kernel guard (`protocol::resume_window_ok`) applied to the +//! delivered frontier instead of a resume cursor. A trip ends the watch with +//! `CursorExpired`, and `watch_applied` repairs it in process through the +//! same expiry path a resume takes. `GuardRepair` is that repair, collapsed to +//! one transition, in the form the bucket's retention calls for: +//! +//! - **Keeps current values** (`discard: new`, no `max_age`): the key-listing +//! diff — fold := what the bucket lists, consumption re-entitled from the +//! floor. Sound, because a key missing from the bucket was deleted. +//! - **Evicts current values** (`max_age`, `discard: old`): an ARTIFACT +//! RESTORE — fold := the newest published artifact's state, consumption +//! resumed from its cursor — gated by the shared `protocol::restore_allowed` +//! kernel (ahead of the frontier, still inside retention). The key-listing +//! diff here is the pre-fix behavior, kept as the `RelistOnEvicting` +//! mutation the checker must catch: the bucket stops listing a key that +//! merely aged out, and the diff deletes it. +//! +//! Correct means **every write minus every real delete** (`key_at(head)`), +//! not "matches what NATS lists" — the two differ exactly when retention +//! evicts current values. //! //! Checked, exhaustively within bounds: -//! - GUARDED: every maximal run ends with the fold equal to the bucket — -//! divergence is at worst transient (bounded by the guard cadence), never -//! permanent. The guard genuinely trips (witness) and markers are also -//! delivered normally (witness), so the theorem is not vacuous. +//! - GUARDED (both retentions): every maximal run ends with the fold equal to +//! the truth — divergence is at worst transient, never permanent. The guard +//! genuinely trips (witness), restores genuinely happen past aged-out values +//! (witness), and markers are also delivered normally (witness), so the +//! theorems are not vacuous. //! - UNGUARDED (the pre-guard code, kept as the machine-checked record): -//! permanent terminal divergence is REACHABLE — fold holds a deleted key -//! forever with nothing left to deliver and no error anywhere. -//! - BOTH: divergence is only ever in the stale direction — the fold never -//! drops a key the bucket still has (no phantom deletes). +//! permanent terminal divergence is REACHABLE, in both directions (a +//! skipped delete marker, a skipped write). +//! - GUARDED: the fold never drops a value that is still current (no +//! phantom deletes), and a repair never moves the fold backward. +//! - MUTATIONS: `RelistOnEvicting` reaches a phantom delete; +//! `RestoreUnguarded` (restore any published artifact) reaches a backward +//! repair. //! //! Scope, honestly: this models the ALL-scope watch, where every stream //! message is deliverable to the consumer and a frontier-vs-floor gap @@ -39,17 +57,19 @@ //! - **No client buffer.** `Compact` means eviction of messages the //! consumer has NOT received — the hazardous kind. In reality a message //! pushed to the client before eviction is still processed; that is -//! received data, not loss, and needs no guard. (The code's dense-pass -//! fast path folds such messages without probing — sound because a -//! delivered message was by definition retained when pushed.) -//! - **`GuardRepair` is a composite** of fail-stop → restart → resume → -//! `CursorExpired` → resync, each verified separately (`tests/model.rs`, -//! `tests/resync.rs`); the composition is by argument. Repair LIVENESS is -//! conditional on the caller restarting the failed watch (standard -//! supervision); the safety half — never folding past unexamined -//! evidence — is unconditional in the code path itself. +//! received data, not loss, and needs no guard. (The code folds that +//! delivered backlog before the repair — `applied.rs` drains the channel — +//! which this model's atomic repair assumes.) +//! - **`GuardRepair` is a composite** of trip → `CursorExpired` → repair → +//! resume, each half verified separately (`tests/model.rs` for the repairs, +//! `tests/resync.rs` and `tests/eviction.rs` live); the composition is by +//! argument. +//! - **Exporters are correct folds**: a published artifact at cursor `c` +//! holds `key_at(c)` (axiom 5 of `tests/model.rs`, which proves it of every +//! bootstrapped fold). `Publish` may lag arbitrarily — a stale pointer is +//! exactly what the restore guard must refuse. -use slipstream::protocol::resume_window_ok; +use slipstream::protocol::{restore_allowed, resume_window_ok}; use stateright::{Checker, Model, Property}; /// Stream revisions run 1..=MAX_REV. @@ -63,13 +83,26 @@ struct St { floor: u8, /// Revision of the sentinel key's delete marker, if deleted. delete_rev: Option, + /// Revision of the sentinel's later write (an update, or a re-create + /// after the delete), if any. Its initial put is revision 0. + reput_rev: Option, + /// Evicting retention only: the sentinel's current value aged out (never + /// deleted; the bucket just stopped listing it). + aged: bool, + /// The newest published artifact's cursor, if any (monotone). + pointer: Option, /// The consumer's delivered frontier (max revision handed to the fold). frontier: u8, - /// The fold's view of the sentinel key (bucket truth: `delete_rev` none). - fold_has_key: bool, + /// The fold's view of the sentinel: the revision of the put it holds, + /// `None` if absent. + fold_key: Option, /// Latched when the floor guard tripped (witness that the guarded /// theorem is earned by repair, not by the race never occurring). tripped: bool, + /// Latched when a repair restored from an artifact. + restored: bool, + /// Latched when a repair moved the frontier backward. + regressed: bool, } #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] @@ -78,26 +111,85 @@ enum Act { Churn, /// The sentinel key is deleted (consumes a revision: the marker). DeleteKey, + /// The sentinel key is written again (consumes a revision). + PutKey, + /// Evicting retention only: the sentinel's current value ages out + /// (FIFO-consistent: once the floor has passed its revision). + AgeOut, /// Retention evicts the oldest retained revision. Compact, + /// A (correct) exporter publishes an artifact at this cursor. + Publish(u8), /// The consumer receives the next RETAINED revision after its frontier — /// evicted revisions are silently skipped, which is the hazard. Deliver, - /// Guarded variant only: the periodic floor check fires, finds the - /// frontier behind the floor, and fail-stops; the restart's resume → - /// expiry → resync (verified by tests/model.rs) repairs the fold. Free - /// interleaving of this action is a superset of every periodic cadence. + /// Guarded variant only: the floor guard finds the frontier behind the + /// floor and the expiry repair runs (see the module docs). Free + /// interleaving of this action is a superset of every guard cadence. GuardRepair, } +#[derive(Clone, Copy, PartialEq, Eq)] +enum BucketRetention { + KeepsCurrent, + EvictsCurrent, +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Mutation { + None, + /// The pre-fix repair on an evicting bucket: the key-listing diff. + RelistOnEvicting, + /// The restore without `restore_allowed`: any published artifact. + RestoreUnguarded, +} + #[derive(Clone)] struct LiveWatch { guarded: bool, + retention: BucketRetention, + mutation: Mutation, } impl LiveWatch { - fn bucket_has_key(s: &St) -> bool { - s.delete_rev.is_none() + /// THE TRUTH: the sentinel after every write and every real delete up to + /// `cursor` (initial put at 0; the delete and later write in revision + /// order). + fn key_at(s: &St, cursor: u8) -> Option { + let mut key = Some(0); + let mut events = [ + s.delete_rev.map(|d| (d, None)), + s.reput_rev.map(|r| (r, Some(r))), + ]; + events.sort(); + for (rev, value) in events.into_iter().flatten() { + if rev <= cursor { + key = value; + } + } + key + } + + fn truth(s: &St) -> Option { + Self::key_at(s, s.head) + } + + /// What the bucket lists: the truth minus an evicted current value. + fn listed(s: &St) -> Option { + Self::truth(s).filter(|_| !s.aged) + } + + fn restores(&self) -> bool { + self.retention == BucketRetention::EvictsCurrent + && self.mutation != Mutation::RelistOnEvicting + } + + /// The restore this repair may take from the current pointer, if any. + fn restore_target(&self, s: &St) -> Option { + let p = s.pointer?; + (self.mutation == Mutation::RestoreUnguarded + || restore_allowed(p as u64, s.frontier as u64, s.floor as u64 + 1)) + .then_some(p) } } @@ -111,9 +203,14 @@ impl Model for LiveWatch { head: 0, floor: 0, delete_rev: None, + reput_rev: None, + aged: false, + pointer: None, frontier: 0, - fold_has_key: true, + fold_key: Some(0), tripped: false, + restored: false, + regressed: false, }] } @@ -123,10 +220,25 @@ impl Model for LiveWatch { if s.delete_rev.is_none() { acts.push(Act::DeleteKey); } + if s.reput_rev.is_none() { + acts.push(Act::PutKey); + } + } + if self.retention == BucketRetention::EvictsCurrent + && !s.aged + && Self::truth(s).is_some_and(|w| w == 0 || s.floor >= w) + { + acts.push(Act::AgeOut); } if s.floor < s.head { acts.push(Act::Compact); } + // Artifacts only matter where a restore can use them. + if self.restores() { + for c in s.pointer.map_or(1, |p| p + 1)..=s.head { + acts.push(Act::Publish(c)); + } + } // Something retained remains beyond the frontier. if s.frontier.max(s.floor) < s.head { // GUARDED: delivery never silently jumps a gap. A delivered @@ -143,8 +255,13 @@ impl Model for LiveWatch { // behind the first retained revision (floor + 1). Reached either by // the gapped-delivery check (above: Deliver disabled, this is the // only progress) or by the periodic backstop when no deliveries - // arrive at all; free interleaving covers every cadence of both. - if self.guarded && !resume_window_ok(s.frontier as u64, s.floor as u64 + 1) { + // arrive at all; free interleaving covers every cadence of both. A + // restore additionally needs an artifact the guard accepts: + // fail-stop until the fleet publishes one. + if self.guarded + && !resume_window_ok(s.frontier as u64, s.floor as u64 + 1) + && (!self.restores() || self.restore_target(s).is_some()) + { acts.push(Act::GuardRepair); } } @@ -157,28 +274,47 @@ impl Model for LiveWatch { s.head += 1; s.delete_rev = Some(s.head); } + Act::PutKey => { + s.head += 1; + s.reput_rev = Some(s.head); + s.aged = false; + } + Act::AgeOut => s.aged = true, Act::Compact => s.floor += 1, + Act::Publish(c) => s.pointer = Some(c), Act::Deliver => { // Next retained revision; anything evicted in between is - // SKIPPED — if the skipped range held the delete marker, the - // fold silently keeps the key. + // SKIPPED — if the skipped range held a sentinel event, the + // fold silently misses it. let next = s.frontier.max(s.floor) + 1; if next > s.head { return None; } s.frontier = next; if s.delete_rev == Some(next) { - s.fold_has_key = false; + s.fold_key = None; + } + if s.reput_rev == Some(next) { + s.fold_key = Some(next); } } Act::GuardRepair => { - // Fail-stop + restart + resume(frontier) + CursorExpired + - // resync, collapsed to its verified outcome: the fold equals - // the bucket's live keys, and consumption is re-entitled - // from the floor. s.tripped = true; - s.fold_has_key = Self::bucket_has_key(&s); - s.frontier = s.floor; + let before = s.frontier; + if self.restores() { + // Artifact restore: the fold becomes the artifact's, + // consumption resumes from its cursor. + let p = self.restore_target(&s)?; + s.fold_key = Self::key_at(&s, p); + s.frontier = p; + s.restored = true; + } else { + // Key-listing diff + re-list: the fold becomes what the + // bucket lists, consumption re-entitled from the floor. + s.fold_key = Self::listed(&s); + s.frontier = s.floor; + } + s.regressed |= s.frontier < before; } } Some(s) @@ -187,49 +323,77 @@ impl Model for LiveWatch { fn properties(&self) -> Vec> { let mut props: Vec> = Vec::new(); - // BOTH variants: divergence is only ever stale-direction — the fold - // never drops a key the bucket still has. - props.push(Property::::always( - "no phantom deletes: the fold never drops a live key", - |_, s| s.fold_has_key || !LiveWatch::bucket_has_key(s), - )); - - // Vacuity witnesses shared by both variants. + // Vacuity witness shared by every variant. props.push(Property::::sometimes( "the marker is delivered normally and the fold drops the key", - |_, s| !s.fold_has_key && !LiveWatch::bucket_has_key(s), + |_, s| s.fold_key.is_none() && LiveWatch::truth(s).is_none() && !s.tripped, )); if self.guarded { + // Divergence is only ever stale-direction: the fold never drops + // a value that is still current (unchanged since the fold's + // cursor). On an evicting bucket the key-listing diff breaks + // exactly this — the `RelistOnEvicting` mutation. + props.push(Property::::always( + "no phantom deletes: the fold never drops a value that is still current", + |_, s| { + let truth = LiveWatch::truth(s); + truth.is_none() + || LiveWatch::key_at(s, s.frontier) != truth + || s.fold_key == truth + }, + )); + props.push(Property::::always( + "a repair never moves the fold backward", + |_, s| !s.regressed, + )); // THE theorem: every maximal run ends with the fold equal to the - // bucket — divergence is at worst transient (a trip away), never + // truth — divergence is at worst transient (a trip away), never // permanent. Terminal-state invariant (cycle-proof): a state // with no enabled actions must be converged. props.push(Property::::always( - "every maximal run ends with the fold equal to the bucket", + "every maximal run ends with the fold equal to every write minus every real delete", |m, s| { let mut acts = Vec::new(); m.actions(s, &mut acts); - !acts.is_empty() || s.fold_has_key == LiveWatch::bucket_has_key(s) + !acts.is_empty() || s.fold_key == LiveWatch::truth(s) }, )); // The theorem is earned: the guard genuinely trips within the // bounds (the race occurs and is repaired, not avoided). props.push(Property::::sometimes( "the floor guard trips and repairs a real divergence", - |_, s| s.tripped && s.fold_has_key == LiveWatch::bucket_has_key(s), + |_, s| s.tripped && s.fold_key == LiveWatch::truth(s), )); + if self.restores() { + props.push(Property::::sometimes( + "a trip restores from an artifact past a value that aged out", + |_, s| s.restored && s.aged && s.fold_key == LiveWatch::truth(s), + )); + } } else { // The pre-guard code, machine-checked: PERMANENT silent // divergence is reachable — a terminal state where the fold - // holds a deleted key, nothing remains to deliver, and no error + // misses an event, nothing remains to deliver, and no error // occurred anywhere. props.push(Property::::sometimes( "HAZARD reachable: permanent silent divergence (marker evicted unseen)", |m, s| { let mut acts = Vec::new(); m.actions(s, &mut acts); - acts.is_empty() && s.fold_has_key && !LiveWatch::bucket_has_key(s) + acts.is_empty() && s.fold_key.is_some() && LiveWatch::truth(s).is_none() + }, + )); + // ...and in the other direction: a skipped WRITE leaves the fold + // without a value the truth has. + props.push(Property::::sometimes( + "HAZARD reachable: permanent silent divergence (write evicted unseen)", + |m, s| { + let mut acts = Vec::new(); + m.actions(s, &mut acts); + acts.is_empty() + && LiveWatch::truth(s).is_some() + && s.fold_key != LiveWatch::truth(s) }, )); } @@ -238,22 +402,46 @@ impl Model for LiveWatch { } } -fn check(guarded: bool, label: &str) { - let model = LiveWatch { guarded }; +fn run(model: LiveWatch, label: &str) -> impl Checker { let checker = model.checker().spawn_bfs().join(); println!( "{label}: {} states, {} unique", checker.state_count(), checker.unique_state_count(), ); - checker.assert_properties(); + checker +} + +fn shipped(guarded: bool, retention: BucketRetention) -> LiveWatch { + LiveWatch { + guarded, + retention, + mutation: Mutation::None, + } } -/// The SHIPPED behavior: All-scope watches carry the periodic floor guard -/// (`nats.rs`), gated by the same shared kernel this model executes. +/// The SHIPPED behavior on a bucket that keeps current values: All-scope +/// watches carry the floor guard (`nats.rs`), gated by the same shared kernel +/// this model executes, repaired by the key-listing diff. #[test] fn guarded_live_watch_always_converges() { - check(true, "live watch: guarded"); + run( + shipped(true, BucketRetention::KeepsCurrent), + "live watch: guarded", + ) + .assert_properties(); +} + +/// The SHIPPED behavior on a bucket that evicts current values: a trip is +/// repaired from an artifact (`restore_allowed`-gated), keeping values that +/// aged out of NATS without being deleted. +#[test] +fn guarded_live_watch_restores_on_evicting_bucket() { + run( + shipped(true, BucketRetention::EvictsCurrent), + "live watch: guarded, evicting, restore", + ) + .assert_properties(); } /// The pre-guard behavior, kept as the machine-checked record of the hazard: @@ -261,5 +449,48 @@ fn guarded_live_watch_always_converges() { /// the fold. #[test] fn unguarded_live_watch_diverges_permanently() { - check(false, "live watch: unguarded"); + run( + shipped(false, BucketRetention::KeepsCurrent), + "live watch: unguarded", + ) + .assert_properties(); +} + +/// The pre-fix repair of a floor-guard trip on an evicting bucket — the +/// key-listing diff — MUST be caught: it deletes a value that merely aged out. +#[test] +fn mutation_relist_on_evicting_trip_is_caught() { + let checker = run( + LiveWatch { + mutation: Mutation::RelistOnEvicting, + ..shipped(true, BucketRetention::EvictsCurrent) + }, + "live watch mutation: relist on evicting", + ); + assert!( + checker + .discovery("no phantom deletes: the fold never drops a value that is still current") + .is_some(), + "the checker must find the key-listing diff deleting an aged-out value" + ); +} + +/// Without `restore_allowed`, a trip can restore an artifact older than what +/// the watch already delivered, moving the fold (and the consumer's domain +/// state) backward. The guard is load-bearing; the checker must catch it. +#[test] +fn mutation_unguarded_restore_is_caught() { + let checker = run( + LiveWatch { + mutation: Mutation::RestoreUnguarded, + ..shipped(true, BucketRetention::EvictsCurrent) + }, + "live watch mutation: unguarded restore", + ); + assert!( + checker + .discovery("a repair never moves the fold backward") + .is_some(), + "the checker must find a restore regressing the fold" + ); } diff --git a/tests/model_repair.rs b/tests/model_repair.rs new file mode 100644 index 0000000..e7f91cb --- /dev/null +++ b/tests/model_repair.rs @@ -0,0 +1,971 @@ +//! Exhaustive model-check of the cursor-expiry repair AS STEPS (Stateright). +//! +//! `tests/model.rs` and `tests/model_live_watch.rs` collapse a repair into +//! one transition. That abstraction hid a real bug: a transient store +//! failure on the flush just before a repair left updates re-queued in +//! memory, invisible to the diff — the diff then missed them and a re-queued +//! put resurrected a key deleted during the gap. This model takes the repair +//! apart into the steps `applied.rs` executes and interleaves them with +//! everything else: +//! +//! - the watch task: resume (window check), delivery, the floor-guard trip, +//! the repair plan, the listing / artifact checks, the ack/reply wait; +//! - the main loop: receive, flush (domain `apply`, cursor advance, store +//! apply), and each repair step — drain, pre-repair flush (retried until +//! the store holds everything), diff, chunk flush, cursor commit, reply; +//! - the store: every apply can succeed, fail transiently (re-queue; a +//! second consecutive failure fail-stops), or crash before / after / torn; +//! the process can also crash between any two steps (bounded budgets); +//! - the environment, at any moment: writes (put or delete), retention +//! eviction (head eviction on an evicting bucket; marker purge on one that +//! keeps current values), and artifact publishes. +//! +//! One key. Retained messages are the latest per key (history 1), so the +//! floor guard here is maximally trigger-happy — every superseding write +//! gaps the frontier — which only exercises the repair more. Writes to other +//! keys don't change this key's fold; the multi-key behavior of the same +//! code is covered by `repair::tests` (exhaustive diff) and +//! `tests/repair_dst.rs` (fault injection over the real loop). +//! +//! Checked, exhaustively within bounds, for both bucket kinds: +//! - **While the store's cursor is resumable, the store plus the tail is the +//! truth**: what a NATS cursor promises — every RETAINED message at or below +//! it is applied, so a resume from it reaches every write minus every real +//! delete. (An expired cursor promises nothing; the repair that must then +//! run fixes it, which the terminal property checks.) +//! - **The store's cursor never moves backward.** +//! - **The same for the domain state at the applied cursor**, and the domain +//! never sees a key's revision go backward or a repair delete its current +//! value — no transient regressions or phantom deletes. +//! - **Every maximal run ends with the store and the domain state equal to +//! every write minus every real delete.** +//! +//! Mutations the checker must catch, one per step: no drain, no pre-repair +//! flush retry (the bug), cursor committed before the restore diff, no ack +//! barrier, no restore guard, key-listing diff on an evicting bucket. +//! +//! Axioms: correct exporters (an artifact at `c` holds the truth at `c` — +//! discharged by `tests/model_fleet.rs`); a full re-list is not overrun by +//! retention before it delivers what it found (axiom 6 of `tests/model.rs`). + +use slipstream::protocol::{restore_allowed, resume_window_ok}; +use stateright::{Checker, Model, Property}; + +/// Capacity of the revision array (bounds are per model: `max_rev`). +const CAP: usize = 8; +/// Consecutive transient failures before the watch fail-stops (16 in code; +/// 2 reaches the same branch). +const FAILSTOP_STREAK: u8 = 2; + +/// An update on the key: its value (`Some(put revision)`, `None` deleted) and +/// the stream position it carries (`None`: a repair's synthetic update). +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +struct Upd { + pos: Option, + value: Option, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum Req { + /// Key-listing diff; what the listing showed for the key. + Relist { listed: Option }, + /// Artifact restore to this cursor, with what the live-key listing (taken + /// after the fetch) showed for the key. + Restore { target: u8, listed: Option }, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum Reply { + Ack, + Restored(u8), + Refused, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum WPhase { + /// Process start: resume from the store's cursor (or a fresh start). + Start, + /// Delivering from frontier `f`. `guarded`: the `_from` resume watch + /// (floor guard on); otherwise the full re-list watch. + Watch { + f: u8, + guarded: bool, + }, + /// The cursor (or frontier) `local` expired. + Expired { + local: u8, + }, + AwaitAck, + AwaitReply, + /// The watch task failed (a refused restore): the process ends. + Failed, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum MPhase { + Idle, + /// Repair received; next: drain the channel. + Drain(Req), + /// Pre-repair flush (retried until the store holds everything). + PreFlush(Req, bool), + /// Key-listing diff: one flush of the synthetic delete, then ack. + RelistFlush(bool), + /// Restore: one flush of the diff (under the old cursor). + RestoreChunk(u8, bool), + /// Restore: the cursor commit, then reply. + RestoreCommit(u8, bool), +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq)] +struct St { + // --- the log + head: u8, + /// `ops[r]`: the write at revision r was a put (true) or delete. + ops: [bool; CAP], + /// The key's latest (only retained) message was evicted. + evicted: bool, + /// Newest published artifact's cursor (0: none). + pointer: u8, + // --- durable + store_val: Option, + store_cur: u8, + // --- in memory + chan: Vec, + batch: Vec, + raw: Vec, + batch_high: Option, + applied: u8, + domain: Option, + streak: u8, + req: Option, + reply: Option, + w: WPhase, + m: MPhase, + // --- budgets and latches + crashes: u8, + transients: u8, + regressed: bool, + /// The domain saw the key's revision go backward. + domain_regressed: bool, + /// The domain saw a repair delete the key while it was live. + phantom_delete: bool, + /// The highest revision the domain has seen for the key. + domain_max: u8, + restored: bool, + relisted: bool, + drained_backlog: bool, + requeued_at_repair: bool, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum FlushOutcome { + Ok, + Transient, + CrashBefore, + CrashAfter, + CrashTorn, +} + +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +enum Act { + Put, + Delete, + Evict, + Publish(u8), + WStart, + WDeliver, + WTrip, + WPlan, + WGotReply, + Restart, + Crash, + MRecv, + MStartRepair, + MDrain, + Flush(FlushOutcome), + MNext, +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Mutation { + None, + NoDrain, + NoPreFlushRetry, + CursorBeforeDiff, + NoAckBarrier, + NoRestoreGuard, + RelistOnEvicting, + /// A fold with data but no cursor re-listed blind (the pre-fix start). + UnanchoredRelist, + /// The restore takes the artifact's value even when the local one is + /// newer (the pre-fix diff). + RestoreIgnoresVersions, + /// The restore deletes keys the bucket lists live (the pre-fix diff). + RestoreIgnoresListing, +} + +#[derive(Clone)] +struct RepairModel { + evicting: bool, + fresh: bool, + mutation: Mutation, + /// Let retention evict during a key-listing repair (see `actions`). + drop_axiom_6: bool, + /// The key's revisions run 1..=max_rev (revision 1 is its creation). + max_rev: u8, + /// Crash budget (process deaths, including store-apply crashes). + max_crashes: u8, + /// Transient store-failure budget. + max_transient: u8, +} + +impl RepairModel { + /// THE TRUTH at `c`: every revision is a write to this key, so the + /// latest write at or before `c` is revision `min(c, head)`. + fn truth_at(s: &St, c: u8) -> Option { + let r = c.min(s.head); + (r >= 1 && s.ops[r as usize]).then_some(r) + } + + fn truth(s: &St) -> Option { + Self::truth_at(s, s.head) + } + + fn first_revision(s: &St) -> u64 { + if s.evicted { + s.head as u64 + 1 + } else { + s.head as u64 + } + } + + /// `listing_is_truth` in `applied.rs`: retention never evicts current + /// values, or has never evicted anything yet. + fn listing_is_truth(s: &St) -> bool { + resume_window_ok(0, Self::first_revision(s)) + } + + fn listed(s: &St) -> Option { + if s.evicted { None } else { Self::truth(s) } + } + + /// What a NATS cursor promises: every RETAINED message at or below it is + /// applied. So a value at cursor `c` is correct when a resume from `c` + /// reaches the truth — the key's latest message, if after `c` and still + /// retained, is delivered; otherwise the value itself must be the truth. + /// Vacuous once `c` is outside retention: that cursor is expired, and the + /// repair that must then run is what makes it right (the terminal + /// property checks that it does). + fn resumes_to_truth(s: &St, c: u8, value: Option) -> bool { + if c == 0 || !resume_window_ok(c as u64, Self::first_revision(s)) { + return true; + } + let tail_delivers = s.head > c && !s.evicted; + tail_delivers || value == Self::truth(s) + } + + fn flushable(s: &St) -> bool { + !s.batch.is_empty() || !s.raw.is_empty() || s.batch_high.is_some() + } + + /// Process death: everything in memory is gone; the restart rebuilds the + /// domain from the store and resumes from its cursor. + fn restart(s: &mut St) { + s.chan.clear(); + s.batch.clear(); + s.raw.clear(); + s.batch_high = None; + s.applied = s.store_cur; + s.domain = s.store_val; + s.domain_max = s.store_val.unwrap_or(0); + s.streak = 0; + s.req = None; + s.reply = None; + s.w = WPhase::Start; + s.m = MPhase::Idle; + } + + fn ingest(s: &mut St, u: Upd) { + if let Some(p) = u.pos { + s.batch_high = Some(p); + } + s.batch.push(u); + s.raw.push(u); + } + + /// `flush!()`: domain apply, cursor advance, store apply with `outcome`. + /// Returns false if the process died. + fn flush(s: &mut St, outcome: FlushOutcome) -> bool { + for u in std::mem::take(&mut s.batch) { + match u.value { + Some(rev) => { + if rev < s.domain_max { + s.domain_regressed = true; + } + s.domain_max = s.domain_max.max(rev); + } + // A PHANTOM delete: a repair's revisionless delete of the value + // the domain holds while that value is still the current one. + // (Deleting a value a newer write has replaced is just the + // domain being behind; the newer write arrives later.) + None if u.pos.is_none() && s.domain.is_some() && s.domain == Self::truth(s) => { + s.phantom_delete = true + } + None => {} + } + s.domain = u.value; + } + let advanced = s.batch_high.is_some(); + if let Some(h) = s.batch_high.take() { + s.applied = h; + } + if s.raw.is_empty() && !advanced { + return true; + } + let commit = |s: &mut St, cursor: u8| { + if let Some(last) = s.raw.last() { + s.store_val = last.value; + } + s.raw.clear(); + if cursor < s.store_cur { + s.regressed = true; + } + s.store_cur = cursor; + }; + match outcome { + FlushOutcome::Ok => { + commit(s, s.applied); + s.streak = 0; + true + } + FlushOutcome::Transient => { + s.transients += 1; + s.streak += 1; + if matches!(s.m, MPhase::PreFlush(..)) { + s.requeued_at_repair = true; + } + if s.streak >= FAILSTOP_STREAK { + Self::restart(s); + return false; + } + true + } + FlushOutcome::CrashBefore => { + s.crashes += 1; + Self::restart(s); + false + } + FlushOutcome::CrashAfter => { + s.crashes += 1; + commit(s, s.applied); + Self::restart(s); + false + } + FlushOutcome::CrashTorn => { + s.crashes += 1; + let old = s.store_cur; + commit(s, old); + Self::restart(s); + false + } + } + } + + /// `restore_diff` on the key: the artifact at `target` holds the truth + /// there. Take it unless the store's value is the same or NEWER; delete + /// (the artifact lacks the key) unless the listing shows it live. + fn restore_op(&self, s: &St, target: u8, listed: Option) -> Option { + let want = Self::truth_at(s, target); + match (s.store_val, want) { + (Some(l), Some(a)) if l >= a && self.mutation != Mutation::RestoreIgnoresVersions => { + None + } + (_, Some(a)) if s.store_val != Some(a) => Some(Upd { + pos: None, + value: Some(a), + }), + (Some(_), None) + if listed.is_none() || self.mutation == Mutation::RestoreIgnoresListing => + { + Some(Upd { + pos: None, + value: None, + }) + } + _ => None, + } + } + + fn restores(&self) -> bool { + self.evicting && self.mutation != Mutation::RelistOnEvicting + } + + /// Resume from cursor `c` (> 0): the window check. + fn resume(s: &St, c: u8) -> WPhase { + if resume_window_ok(c as u64, Self::first_revision(s)) { + WPhase::Watch { + f: c, + guarded: true, + } + } else { + WPhase::Expired { local: c } + } + } + + fn restore_ok(&self, s: &St, target: u8, local: u8) -> bool { + self.mutation == Mutation::NoRestoreGuard + || (target > 0 && restore_allowed(target as u64, local as u64, Self::first_revision(s))) + } +} + +impl Model for RepairModel { + type State = St; + type Action = Act; + + fn init_states(&self) -> Vec { + let mut ops = [false; CAP]; + ops[1] = true; + let (store_val, store_cur) = if self.fresh { (None, 0) } else { (Some(1), 1) }; + vec![St { + head: 1, + ops, + evicted: false, + pointer: 0, + store_val, + store_cur, + chan: vec![], + batch: vec![], + raw: vec![], + batch_high: None, + applied: store_cur, + domain: store_val, + streak: 0, + req: None, + reply: None, + w: WPhase::Start, + m: MPhase::Idle, + crashes: 0, + transients: 0, + regressed: false, + domain_regressed: false, + phantom_delete: false, + domain_max: store_val.unwrap_or(0), + restored: false, + relisted: false, + drained_backlog: false, + requeued_at_repair: false, + }] + } + + fn actions(&self, s: &St, acts: &mut Vec) { + // Environment. + if s.head < self.max_rev { + acts.push(Act::Put); + acts.push(Act::Delete); + } + // Axiom 6: retention doesn't overrun a key-listing repair or a full + // re-list while it is in flight — from the moment the listing is + // taken until the re-list has delivered what the bucket holds. (On a + // bucket that keeps current values, the only eviction is an admin + // purge of delete markers; NATS's `purge_deletes` keeps markers + // younger than 30 minutes by default. Without this axiom the checker + // finds the delete-and-purge-inside-the-window trace — kept as + // `relist_window_needs_axiom_6`.) + let relist_in_flight = !self.drop_axiom_6 + && (matches!(s.w, WPhase::Watch { f, guarded: false } if f < s.head) + || matches!(s.w, WPhase::AwaitAck) + || matches!(s.req, Some(Req::Relist { .. })) + || matches!( + s.m, + MPhase::Drain(Req::Relist { .. }) + | MPhase::PreFlush(Req::Relist { .. }, _) + | MPhase::RelistFlush(_) + )); + let evictable = if self.evicting { + true + } else { + !s.ops[s.head as usize] + }; + if !s.evicted && evictable && !relist_in_flight { + acts.push(Act::Evict); + } + if self.evicting { + for c in (s.pointer + 1)..=s.head { + acts.push(Act::Publish(c)); + } + } + + // Watch task. + match s.w { + WPhase::Start => acts.push(Act::WStart), + WPhase::Watch { f, guarded } => { + if !s.evicted && s.head > f { + // Guarded: the in-band check precedes delivery. + if !guarded || resume_window_ok(f as u64, Self::first_revision(s)) { + acts.push(Act::WDeliver); + } + } + if guarded && !resume_window_ok(f as u64, Self::first_revision(s)) { + acts.push(Act::WTrip); + } + } + WPhase::Expired { .. } => acts.push(Act::WPlan), + WPhase::AwaitAck | WPhase::AwaitReply => { + if s.reply.is_some() { + acts.push(Act::WGotReply); + } + } + WPhase::Failed => acts.push(Act::Restart), + } + + // Main loop. + let flush_outcomes = |acts: &mut Vec| { + acts.push(Act::Flush(FlushOutcome::Ok)); + if s.transients < self.max_transient { + acts.push(Act::Flush(FlushOutcome::Transient)); + } + if s.crashes < self.max_crashes { + acts.push(Act::Flush(FlushOutcome::CrashBefore)); + acts.push(Act::Flush(FlushOutcome::CrashAfter)); + acts.push(Act::Flush(FlushOutcome::CrashTorn)); + } + }; + match s.m { + MPhase::Idle => { + // Superset of the biased select: receive is allowed even with + // a repair pending (the drain makes either order safe). + if !s.chan.is_empty() { + acts.push(Act::MRecv); + } + if s.req.is_some() { + acts.push(Act::MStartRepair); + } + if Self::flushable(s) { + flush_outcomes(acts); + } + } + MPhase::Drain(_) => acts.push(Act::MDrain), + MPhase::PreFlush(_, attempted) => { + let retry = self.mutation != Mutation::NoPreFlushRetry || !attempted; + if Self::flushable(s) && retry { + flush_outcomes(acts); + } + let done = if self.mutation == Mutation::NoPreFlushRetry { + attempted || !Self::flushable(s) + } else { + !Self::flushable(s) + }; + if done { + acts.push(Act::MNext); + } + } + MPhase::RelistFlush(attempted) + | MPhase::RestoreChunk(_, attempted) + | MPhase::RestoreCommit(_, attempted) => { + if !attempted && Self::flushable(s) { + flush_outcomes(acts); + } + if attempted || !Self::flushable(s) { + acts.push(Act::MNext); + } + } + } + if s.crashes < self.max_crashes { + acts.push(Act::Crash); + } + } + + fn next_state(&self, s: &St, a: Act) -> Option { + let mut s = s.clone(); + match a { + Act::Put | Act::Delete => { + s.head += 1; + s.ops[s.head as usize] = a == Act::Put; + s.evicted = false; + } + Act::Evict => s.evicted = true, + Act::Publish(c) => s.pointer = c, + Act::WStart => { + s.w = if s.store_cur == 0 { + if s.store_val.is_some() && self.mutation != Mutation::UnanchoredRelist { + // Data but no cursor (a torn first checkpoint): a + // re-list alone never removes what it doesn't + // deliver — repair like an expiry at 0. + WPhase::Expired { local: 0 } + } else if self.restores() && !Self::listing_is_truth(&s) { + // Empty fold, bucket already evicted current values: + // seed from an artifact. + WPhase::Expired { local: 0 } + } else { + WPhase::Watch { + f: 0, + guarded: false, + } + } + } else { + Self::resume(&s, s.store_cur) + }; + } + Act::WDeliver => { + let WPhase::Watch { guarded, .. } = s.w else { + return None; + }; + s.chan.push(Upd { + pos: Some(s.head), + value: Self::truth(&s), + }); + s.w = WPhase::Watch { f: s.head, guarded }; + } + Act::WTrip => { + let WPhase::Watch { f, .. } = s.w else { + return None; + }; + s.w = WPhase::Expired { local: f }; + } + Act::WPlan => { + let WPhase::Expired { local } = s.w else { + return None; + }; + if self.restores() && !Self::listing_is_truth(&s) { + // Peek + fetch collapsed onto the CURRENT pointer (the + // code re-checks what it fetched against now). + let target = s.pointer; + if self.restore_ok(&s, target, local) && target > 0 { + s.req = Some(Req::Restore { + target, + listed: Self::listed(&s), + }); + s.w = WPhase::AwaitReply; + } else { + s.w = WPhase::Failed; + } + } else { + s.req = Some(Req::Relist { + listed: Self::listed(&s), + }); + s.w = if self.mutation == Mutation::NoAckBarrier { + WPhase::Watch { + f: 0, + guarded: false, + } + } else { + WPhase::AwaitAck + }; + } + } + Act::WGotReply => { + s.w = match s.reply.take()? { + Reply::Ack => WPhase::Watch { + f: 0, + guarded: false, + }, + Reply::Restored(c) => Self::resume(&s, c), + Reply::Refused => WPhase::Failed, + }; + } + Act::Restart => Self::restart(&mut s), + Act::Crash => { + s.crashes += 1; + Self::restart(&mut s); + } + Act::MRecv => { + let u = s.chan.remove(0); + Self::ingest(&mut s, u); + } + Act::MStartRepair => { + let req = s.req.take()?; + s.m = MPhase::Drain(req); + } + Act::MDrain => { + let MPhase::Drain(req) = s.m else { return None }; + if self.mutation != Mutation::NoDrain { + if !s.chan.is_empty() { + s.drained_backlog = true; + } + for u in std::mem::take(&mut s.chan) { + Self::ingest(&mut s, u); + } + } + s.m = MPhase::PreFlush(req, false); + } + Act::Flush(outcome) => { + let before = s.m; + if Self::flush(&mut s, outcome) { + s.m = match before { + MPhase::PreFlush(r, _) => MPhase::PreFlush(r, true), + MPhase::RelistFlush(_) => MPhase::RelistFlush(true), + MPhase::RestoreChunk(t, _) => MPhase::RestoreChunk(t, true), + MPhase::RestoreCommit(t, _) => MPhase::RestoreCommit(t, true), + other => other, + }; + } + } + Act::MNext => match s.m { + MPhase::PreFlush(Req::Relist { listed }, _) => { + // Diff the STORE against the listing. + if s.store_val.is_some() && listed.is_none() { + let u = Upd { + pos: None, + value: None, + }; + s.batch.push(u); + s.raw.push(u); + } + s.relisted = true; + s.m = MPhase::RelistFlush(false); + } + MPhase::PreFlush(Req::Restore { target, listed }, _) => { + // The authoritative ahead check. + if self.mutation != Mutation::NoRestoreGuard && target <= s.applied { + s.reply = Some(Reply::Refused); + s.m = MPhase::Idle; + } else if self.mutation == Mutation::CursorBeforeDiff { + s.batch_high = Some(target); + s.m = MPhase::RestoreCommit(target, false); + } else { + if let Some(u) = self.restore_op(&s, target, listed) { + s.batch.push(u); + s.raw.push(u); + } + s.m = MPhase::RestoreChunk(target, false); + } + } + MPhase::RelistFlush(_) => { + s.reply = Some(Reply::Ack); + s.m = MPhase::Idle; + } + MPhase::RestoreChunk(target, _) => { + s.batch_high = Some(target); + s.m = MPhase::RestoreCommit(target, false); + } + MPhase::RestoreCommit(target, _) => { + if self.mutation == Mutation::CursorBeforeDiff && !s.restored { + // The diff, after the cursor already moved. + s.restored = true; + if let Some(u) = self.restore_op(&s, target, Self::listed(&s)) { + s.batch.push(u); + s.raw.push(u); + } + s.m = MPhase::RestoreChunk(target, false); + return Some(s); + } + s.restored = true; + s.reply = Some(Reply::Restored(target)); + s.m = MPhase::Idle; + } + _ => return None, + }, + } + Some(s) + } + + fn properties(&self) -> Vec> { + let mut props: Vec> = vec![ + Property::::always( + "while the store's cursor is resumable, the store plus the tail is the truth", + |_, s| RepairModel::resumes_to_truth(s, s.store_cur, s.store_val), + ), + Property::::always("the store's cursor never moves backward", |_, s| { + !s.regressed + }), + Property::::always( + "the domain never sees the key's revision go backward", + |_, s| !s.domain_regressed, + ), + Property::::always("a repair never deletes the key's current value", |_, s| { + !s.phantom_delete + }), + Property::::always( + "while the applied cursor is resumable, the domain plus the tail is the truth", + |_, s| RepairModel::resumes_to_truth(s, s.applied, s.domain), + ), + Property::::always( + "every maximal run ends with the store and domain equal to every write minus every real delete", + |m, s| { + let mut acts = Vec::new(); + m.actions(s, &mut acts); + !acts.is_empty() + || (s.store_val == RepairModel::truth(s) && s.domain == s.store_val) + }, + ), + ]; + if self.mutation == Mutation::None { + // The theorems are earned: the repairs run, the hazards they + // guard against occur. + if self.restores() { + props.push(Property::::sometimes( + "a restore completes past an evicted current value", + |_, s| s.restored && s.evicted && RepairModel::truth(s).is_some(), + )); + } else { + props.push(Property::::sometimes( + "a key-listing repair runs", + |_, s| s.relisted, + )); + } + props.push(Property::::sometimes( + "a repair drains a delivered backlog", + |_, s| s.drained_backlog, + )); + props.push(Property::::sometimes( + "a pre-repair flush fails transiently and is retried", + |_, s| s.requeued_at_repair, + )); + } + props + } +} + +fn run(model: RepairModel, label: &str) -> impl Checker { + let checker = model.checker().spawn_bfs().join(); + println!( + "{label}: {} states, {} unique", + checker.state_count(), + checker.unique_state_count() + ); + checker +} + +fn shipped(evicting: bool, fresh: bool) -> RepairModel { + RepairModel { + evicting, + fresh, + mutation: Mutation::None, + drop_axiom_6: false, + max_rev: 4, + max_crashes: 1, + max_transient: 2, + } +} + +#[test] +fn keeps_current_repair_steps_are_correct() { + run(shipped(false, false), "repair steps: keeps-current").assert_properties(); +} + +#[test] +fn evicting_repair_steps_are_correct() { + run(shipped(true, false), "repair steps: evicting").assert_properties(); +} + +#[test] +fn fresh_start_repair_steps_are_correct() { + run(shipped(false, true), "repair steps: keeps-current, fresh").assert_properties(); + run(shipped(true, true), "repair steps: evicting, fresh").assert_properties(); +} + +/// Every mutation must produce a counterexample to at least one safety or +/// terminal property, on the bucket kind it applies to. +#[test] +fn every_repair_step_is_load_bearing() { + let cases = [ + (Mutation::NoDrain, true, false), + (Mutation::NoDrain, false, false), + (Mutation::NoPreFlushRetry, false, false), + (Mutation::NoPreFlushRetry, true, false), + (Mutation::CursorBeforeDiff, true, false), + (Mutation::NoAckBarrier, false, false), + (Mutation::NoRestoreGuard, true, false), + (Mutation::RelistOnEvicting, true, false), + (Mutation::UnanchoredRelist, false, true), + (Mutation::RestoreIgnoresVersions, true, false), + (Mutation::RestoreIgnoresListing, true, false), + ]; + let names = [ + "while the store's cursor is resumable, the store plus the tail is the truth", + "the store's cursor never moves backward", + "the domain never sees the key's revision go backward", + "a repair never deletes the key's current value", + "while the applied cursor is resumable, the domain plus the tail is the truth", + "every maximal run ends with the store and domain equal to every write minus every real delete", + ]; + for (mutation, evicting, fresh) in cases { + let label = format!( + "mutation {}: {}", + match mutation { + Mutation::None => "none", + Mutation::NoDrain => "no drain", + Mutation::NoPreFlushRetry => "no pre-repair flush retry", + Mutation::CursorBeforeDiff => "cursor before diff", + Mutation::NoAckBarrier => "no ack barrier", + Mutation::NoRestoreGuard => "no restore guard", + Mutation::RelistOnEvicting => "relist on evicting", + Mutation::UnanchoredRelist => "unanchored fold re-listed blind", + Mutation::RestoreIgnoresVersions => "restore ignores versions", + Mutation::RestoreIgnoresListing => "restore ignores the live listing", + }, + if evicting { + "evicting" + } else { + "keeps-current" + } + ); + let checker = run( + RepairModel { + mutation, + ..shipped(evicting, fresh) + }, + &label, + ); + let found: Vec<&str> = names + .iter() + .copied() + .filter(|n| checker.discovery(n).is_some()) + .collect(); + assert!( + !found.is_empty(), + "{label}: the checker found no counterexample" + ); + println!(" caught by: {found:?}"); + } +} + +/// The axiom the key-listing repair rests on, made explicit: drop it and the +/// checker finds a delete whose marker is purged between the listing and the +/// re-list. The artifact restore resumes on a floor-guarded watch and does +/// not need it — the evicting configuration holds without the axiom. +#[test] +fn relist_window_needs_axiom_6() { + let checker = run( + RepairModel { + drop_axiom_6: true, + ..shipped(false, false) + }, + "repair steps: keeps-current, axiom 6 dropped", + ); + assert!( + checker + .discovery( + "every maximal run ends with the store and domain equal to every write minus every real delete" + ) + .is_some(), + "without axiom 6 the relist window must be reachable" + ); + run( + RepairModel { + drop_axiom_6: true, + ..shipped(true, false) + }, + "repair steps: evicting, axiom 6 dropped", + ) + .assert_properties(); +} + +/// Deeper bounds: one more revision, two crashes, three transient failures. +/// `cargo test --release --test model_repair -- --ignored` +#[test] +#[ignore = "deep bounds: run in release"] +fn deep_repair_steps() { + for (evicting, fresh) in [(false, false), (true, false), (false, true), (true, true)] { + run( + RepairModel { + max_rev: 5, + max_crashes: 2, + max_transient: 3, + ..shipped(evicting, fresh) + }, + &format!("deep repair steps: evicting={evicting} fresh={fresh}"), + ) + .assert_properties(); + } +} diff --git a/tests/repair_dst.rs b/tests/repair_dst.rs new file mode 100644 index 0000000..01f9993 --- /dev/null +++ b/tests/repair_dst.rs @@ -0,0 +1,1429 @@ +//! Deterministic fault injection over the REAL `watch_applied`: every +//! scenario × every fault schedule, against the shipped code, on every +//! snapshot backend. +//! +//! The Stateright models check protocols over abstractions; this checks the +//! implementation. A simulated NATS log (latest message per key, head +//! eviction on an evicting bucket, marker purges on one that keeps current +//! values, an optional mid-watch floor-guard trip) feeds the real +//! `watch_applied` with a real snapshot backend wrapped in a fault store. A +//! schedule injects, at chosen store-apply calls of chosen runs: +//! +//! - a TRANSIENT failure (the batch is re-queued; the watch continues), or +//! - a CRASH — process death — before the apply, after it, or TORN (the +//! batch's data durable, its cursor not: the append log's torn-write shape, +//! and a superset state for the atomic-batch LSM backends). The run dies +//! there; the harness reopens the fold from disk and restarts, as a +//! supervisor would. +//! +//! Runs repeat until one completes with no fault left to inject. Then: +//! - the fold equals every write minus every real delete in scope, at the +//! head's cursor, and every out-of-scope key exactly as it started; +//! - the consumer's domain state — rebuilt from the fold at each start, then +//! driven only by `apply` — equals the fold; +//! - the fold's cursor never moved backward across any crash, and +//! `on_applied` never reported a backward step within a run; +//! - `apply` never saw a key's revision go backward, and no repair deleted a +//! key that is live in the log (no regressions, no phantom deletes — not +//! even transiently); +//! - every artifact exported mid-run (an `ExportRequest` sent from +//! `on_applied`) records the watch's scope and, with a resume from its +//! cursor, reaches the truth. This is the real-code half of +//! `tests/model_fleet.rs`'s induction. Restores also run from artifacts +//! exported mid-catch-up — what a real exporter publishes when an export +//! round lands during its own resume or initial re-list. +//! +//! Scenarios cover both the expiry repairs (resume-time expiry, floor-guard +//! trip mid-watch, cursor-less starts with and without data; key-listing +//! diff and artifact restore; All, Prefix and multi-prefix scopes) and the +//! normal data path (resume inside the window with deletes and re-creates, +//! a fresh full re-list). Each runs in both delivery orders and with batches +//! of 1 and 100. The append log gets the full schedule space: no fault; +//! every set of ≤ 2 transient failures; every single crash (each call × each +//! kind); every pair of crashes in consecutive runs; every transient failure +//! followed by a crash; a persistent store failure. The LSM backends get +//! every single fault and the persistent failure (their per-run cost is +//! dominated by opening a database). +//! +//! Reverting any of seven repair steps (pre-repair flush retry, drain, +//! cursor-after-diff, `Auto`'s retention check, the diff's deletes, the ack +//! barrier, the data-without-cursor repair) fails this file. + +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex, Once}; + +use async_trait::async_trait; +use slipstream::protocol::resume_window_ok; +use slipstream::snapshot::{SnapshotError, SnapshotStore}; +use slipstream::{ + ARTIFACT_SCHEMA_VERSION, AppendLogSnapshot, BatchConfig, ExpiryRepair, ExportManifest, + ExportRequest, KvEntry, KvError, KvReader, KvUpdate, KvWatcher, RestoreSource, RestoredFold, + Retention, VersionToken, WatchCursor, WatchScope, watch_applied, +}; +use tokio::sync::mpsc::Sender; +use tokio::sync::{mpsc, oneshot}; + +// --- Backends ------------------------------------------------------------- + +trait Backend: Send + Sync + 'static { + type S: SnapshotStore + Send + 'static; + const NAME: &'static str; + fn open(path: &Path) -> (WatchCursor, Self::S); + fn import(artifact: &Path, dest: &Path) -> (WatchCursor, Self::S); +} + +struct AppendLog; +impl Backend for AppendLog { + type S = AppendLogSnapshot; + const NAME: &'static str = "append-log"; + fn open(path: &Path) -> (WatchCursor, AppendLogSnapshot) { + AppendLogSnapshot::open(path, u64::MAX).unwrap() + } + fn import(artifact: &Path, dest: &Path) -> (WatchCursor, AppendLogSnapshot) { + AppendLogSnapshot::import(artifact, dest, u64::MAX).unwrap() + } +} + +#[cfg(feature = "fjall")] +struct Fjall; +#[cfg(feature = "fjall")] +impl Backend for Fjall { + type S = slipstream::FjallSnapshot; + const NAME: &'static str = "fjall"; + fn open(path: &Path) -> (WatchCursor, Self::S) { + slipstream::FjallSnapshot::open(path, Self::cfg()).unwrap() + } + fn import(artifact: &Path, dest: &Path) -> (WatchCursor, Self::S) { + slipstream::FjallSnapshot::import(artifact, dest, Self::cfg()).unwrap() + } +} +#[cfg(feature = "fjall")] +impl Fjall { + fn cfg() -> slipstream::FjallConfig { + slipstream::FjallConfig { + sync: false, + cache_size_bytes: 4 << 20, + } + } +} + +#[cfg(feature = "rocksdb")] +struct Rocks; +#[cfg(feature = "rocksdb")] +impl Backend for Rocks { + type S = slipstream::RocksDbSnapshot; + const NAME: &'static str = "rocksdb"; + fn open(path: &Path) -> (WatchCursor, Self::S) { + slipstream::RocksDbSnapshot::open(path, Self::cfg()).unwrap() + } + fn import(artifact: &Path, dest: &Path) -> (WatchCursor, Self::S) { + slipstream::RocksDbSnapshot::import(artifact, dest, Self::cfg()).unwrap() + } +} +#[cfg(feature = "rocksdb")] +impl Rocks { + fn cfg() -> slipstream::RocksDbConfig { + slipstream::RocksDbConfig { + sync: false, + cache_size_bytes: 4 << 20, + } + } +} + +// --- The simulated log ---------------------------------------------------- + +#[derive(Clone, Debug)] +struct Event { + rev: u64, + key: &'static str, + value: Option<&'static str>, +} + +/// A mid-watch floor-guard trip: the first `*_from` call delivers `deliver` +/// messages; then, while the watch is behind, the bucket moves on (`append`) +/// and retention overruns the consumer (`evict_to` on an evicting bucket, a +/// marker purge on one that keeps current values); the watch ends with +/// `CursorExpired`. +#[derive(Clone, Debug)] +struct Trip { + deliver: usize, + append: Vec, + evict_to: Option, + purge_markers: bool, +} + +#[derive(Debug)] +struct LogState { + events: Vec, + evicts_current: bool, + /// Revisions <= floor are head-evicted (evicting buckets only). + floor: u64, + /// Delete markers removed by an admin purge. + purged: BTreeSet, + trip: Option, + /// Yield between messages, so the main loop ingests them before a repair + /// request arrives (the other delivery order). + yield_between: bool, +} + +impl LogState { + fn head(&self) -> u64 { + self.events.last().map_or(0, |e| e.rev) + } + + /// The stream's retained messages: the latest per key (history 1), minus + /// head eviction and purged markers, in revision order. + fn retained(&self) -> Vec { + let mut latest: BTreeMap<&str, &Event> = BTreeMap::new(); + for e in &self.events { + latest.insert(e.key, e); + } + let mut out: Vec = latest + .into_values() + .filter(|e| !(self.evicts_current && e.rev <= self.floor)) + .filter(|e| !(e.value.is_none() && self.purged.contains(&e.rev))) + .cloned() + .collect(); + out.sort_by_key(|e| e.rev); + out + } + + fn first_revision(&self) -> u64 { + self.retained().first().map_or(self.head() + 1, |e| e.rev) + } + + fn apply_trip(&mut self, trip: Trip) { + self.events.extend(trip.append); + if let Some(rev) = trip.evict_to { + self.floor = self.floor.max(rev); + } + if trip.purge_markers { + for e in &self.events { + if e.value.is_none() { + self.purged.insert(e.rev); + } + } + } + } +} + +fn in_scope(prefixes: &[String], key: &str) -> bool { + prefixes.iter().any(|p| key.starts_with(p.as_str())) +} + +fn to_update(e: &Event) -> KvUpdate { + let version = VersionToken::from_u64(e.rev); + match e.value { + Some(v) => KvUpdate::Put(KvEntry { + key: e.key.to_string(), + value: v.as_bytes().to_vec(), + version, + }), + None => KvUpdate::Delete { + key: e.key.to_string(), + version, + }, + } +} + +/// THE TRUTH at `cursor`: every write minus every real delete. +fn truth(events: &[Event], cursor: u64) -> BTreeMap, u64)> { + let mut out = BTreeMap::new(); + for e in events.iter().filter(|e| e.rev <= cursor) { + match e.value { + Some(v) => { + out.insert(e.key.to_string(), (v.as_bytes().to_vec(), e.rev)); + } + None => { + out.remove(e.key); + } + } + } + out +} + +struct SimLog(Mutex); + +impl SimLog { + async fn deliver(&self, msgs: Vec, tx: &Sender) { + let yield_between = self.0.lock().unwrap().yield_between; + for m in msgs { + if tx.send(to_update(&m)).await.is_err() { + return; + } + if yield_between { + tokio::task::yield_now().await; + } + } + } + + async fn from( + &self, + prefixes: &[String], + cursor: &WatchCursor, + tx: Sender, + ) -> Result<(), KvError> { + let c = cursor.as_u64().unwrap_or(0); + let (msgs, trip) = { + let mut st = self.0.lock().unwrap(); + if !resume_window_ok(c, st.first_revision()) { + return Err(KvError::CursorExpired); + } + let msgs: Vec = st + .retained() + .into_iter() + .filter(|e| e.rev > c && in_scope(prefixes, e.key)) + .collect(); + (msgs, st.trip.take()) + }; + match trip { + Some(trip) => { + self.deliver(msgs.into_iter().take(trip.deliver).collect(), &tx) + .await; + self.0.lock().unwrap().apply_trip(trip); + Err(KvError::CursorExpired) + } + None => { + self.deliver(msgs, &tx).await; + Ok(()) + } + } + } + + async fn full(&self, prefixes: &[String], tx: Sender) -> Result<(), KvError> { + let msgs: Vec = { + let st = self.0.lock().unwrap(); + st.retained() + .into_iter() + .filter(|e| in_scope(prefixes, e.key)) + .collect() + }; + self.deliver(msgs, &tx).await; + Ok(()) + } +} + +fn owned(p: &[&str]) -> Vec { + p.iter().map(|s| s.to_string()).collect() +} + +#[async_trait] +impl KvWatcher for SimLog { + async fn watch_all(&self, tx: Sender) -> Result<(), KvError> { + self.full(&owned(&[""]), tx).await + } + async fn watch_prefix(&self, prefix: &str, tx: Sender) -> Result<(), KvError> { + self.full(&owned(&[prefix]), tx).await + } + async fn watch_prefixes(&self, prefixes: &[&str], tx: Sender) -> Result<(), KvError> { + self.full(&owned(prefixes), tx).await + } + async fn watch_all_from(&self, c: &WatchCursor, tx: Sender) -> Result<(), KvError> { + self.from(&owned(&[""]), c, tx).await + } + async fn watch_prefix_from( + &self, + prefix: &str, + c: &WatchCursor, + tx: Sender, + ) -> Result<(), KvError> { + self.from(&owned(&[prefix]), c, tx).await + } + async fn watch_prefixes_from( + &self, + prefixes: &[&str], + c: &WatchCursor, + tx: Sender, + ) -> Result<(), KvError> { + self.from(&owned(prefixes), c, tx).await + } + async fn retention(&self) -> Result, KvError> { + let st = self.0.lock().unwrap(); + Ok(Some(Retention { + evicts_current_values: st.evicts_current, + first_revision: st.first_revision(), + })) + } +} + +#[async_trait] +impl KvReader for SimLog { + async fn get(&self, _k: &str) -> Result, KvError> { + unreachable!("repairs only list keys") + } + async fn entry(&self, _k: &str) -> Result, KvError> { + unreachable!("repairs only list keys") + } + async fn keys(&self, prefix: &str) -> Result, KvError> { + let st = self.0.lock().unwrap(); + Ok(st + .retained() + .into_iter() + .filter(|e| e.value.is_some() && e.key.starts_with(prefix)) + .map(|e| e.key.to_string()) + .collect()) + } + async fn scan(&self, _p: &str) -> Result, KvError> { + unreachable!("repairs only list keys") + } +} + +// --- The fault store ------------------------------------------------------ + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Crash { + /// Dies before anything of this apply is durable. + Before, + /// Dies after the apply is fully durable, before returning. + After, + /// The batch's data is durable, its cursor is not. + Torn, +} + +/// Faults for one run: store-apply call indices to fail transiently, and an +/// optional crash. +#[derive(Clone, Debug, Default)] +struct RunFaults { + transient: Vec, + crash: Option<(usize, Crash)>, +} + +impl RunFaults { + fn is_empty(&self) -> bool { + self.transient.is_empty() && self.crash.is_none() + } +} + +const CRASH_MSG: &str = "simulated crash (repair_dst)"; + +struct FaultStore { + inner: S, + faults: RunFaults, + calls: Arc, +} + +impl SnapshotStore for FaultStore { + fn load(_path: &Path) -> Result<(WatchCursor, Self), SnapshotError> { + unreachable!("constructed directly") + } + fn apply(&mut self, batch: &[KvUpdate], cursor: &WatchCursor) -> Result<(), SnapshotError> { + let call = self.calls.fetch_add(1, Ordering::SeqCst); + if let Some((at, kind)) = self.faults.crash + && at == call + { + match kind { + Crash::Before => {} + Crash::After => self.inner.apply(batch, cursor)?, + Crash::Torn => { + let old = self.inner.cursor(); + self.inner.apply(batch, &old)?; + } + } + panic!("{CRASH_MSG}"); + } + if self.faults.transient.contains(&call) { + return Err(SnapshotError::Backend("injected transient failure".into())); + } + self.inner.apply(batch, cursor) + } + fn get(&self, key: &str) -> Result, SnapshotError> { + self.inner.get(key) + } + fn range(&self, prefix: &str) -> Result, SnapshotError> { + self.inner.range(prefix) + } + fn for_each_in_range( + &self, + prefix: &str, + f: impl FnMut(KvEntry) -> Result<(), SnapshotError>, + ) -> Result<(), SnapshotError> { + self.inner.for_each_in_range(prefix, f) + } + fn cursor(&self) -> WatchCursor { + self.inner.cursor() + } + fn export_to(&mut self, dest: &Path) -> Result { + self.inner.export_to(dest) + } +} + +// --- The artifact source -------------------------------------------------- + +/// The newest published artifact: a correct exporter's fold at `cursor` (an +/// exporter that saw the whole log, so it reads the log's events). Built on +/// the backend under test, like a real `ArtifactRestore` import. +struct SimRestore { + log: Arc, + cursor: u64, + /// How the exporter came to its fold: `None` — complete (the truth at + /// `cursor`); `Some(e)` — it held the truth at `e`, then caught up + /// through `cursor` by a resume (latest message per key, superseded + /// revisions skipped); `Some(0)` is a fresh full re-list. Mid-catch-up + /// artifacts lack keys whose latest write is after `cursor`, and hold + /// older values for keys whose intermediate revisions were superseded. + exporter_from: Option, + dir: PathBuf, + fetches: AtomicUsize, + _b: std::marker::PhantomData B>, +} + +impl SimRestore { + fn manifest(&self) -> ExportManifest { + ExportManifest { + schema_version: ARTIFACT_SCHEMA_VERSION, + backend: "sim".into(), + backend_version: "0".into(), + cursor: WatchCursor::from_u64(self.cursor), + created_at_unix: 0, + files: vec![], + scope: Some(vec![String::new()]), + } + } +} + +#[async_trait] +impl RestoreSource> for SimRestore { + async fn latest(&self) -> Result { + Ok(self.manifest()) + } + async fn fetch(&self) -> Result>, SnapshotError> { + let n = self.fetches.fetch_add(1, Ordering::SeqCst); + let (_c, mut fold) = B::open(&self.dir.join(format!("artifact-{n}"))); + let events = self.log.0.lock().unwrap().events.clone(); + let state = match self.exporter_from { + None => truth(&events, self.cursor), + Some(e) => { + let mut state = truth(&events, e); + let mut latest: BTreeMap<&str, &Event> = BTreeMap::new(); + for ev in &events { + latest.insert(ev.key, ev); + } + for ev in latest + .into_values() + .filter(|ev| ev.rev > e && ev.rev <= self.cursor) + { + match ev.value { + Some(v) => { + state.insert(ev.key.to_string(), (v.as_bytes().to_vec(), ev.rev)); + } + None => { + state.remove(ev.key); + } + } + } + state + } + }; + let updates: Vec = state + .into_iter() + .map(|(key, (value, rev))| { + KvUpdate::Put(KvEntry { + key, + value, + version: VersionToken::from_u64(rev), + }) + }) + .collect(); + fold.apply(&updates, &WatchCursor::from_u64(self.cursor))?; + Ok(RestoredFold::new( + self.manifest(), + FaultStore { + inner: fold, + faults: RunFaults::default(), + calls: Arc::new(AtomicUsize::new(0)), + }, + )) + } +} + +// --- Scenarios ------------------------------------------------------------ + +#[derive(Clone, Copy, Debug)] +enum Mode { + Auto, + Relist, +} + +#[derive(Clone, Debug)] +struct Scenario { + name: &'static str, + events: Vec, + /// The fold's starting state: every event up to here (`None`: a fresh + /// node with no cursor and an empty fold). + local_at: Option, + /// Seed `local_at`'s data WITHOUT its cursor: a torn first checkpoint. + unanchored: bool, + evicts_current: bool, + floor: u64, + purged: Vec, + trip: Option, + artifact_at: u64, + /// See `SimRestore::exporter_from`. + exporter_from: Option, + scope: WatchScope, + mode: Mode, +} + +impl Scenario { + fn prefixes(&self) -> Vec { + match &self.scope { + WatchScope::All => vec![String::new()], + WatchScope::Prefix(p) => vec![p.clone()], + WatchScope::Prefixes(ps) => ps.clone(), + } + } +} + +fn ev(rev: u64, key: &'static str, value: Option<&'static str>) -> Event { + Event { rev, key, value } +} + +/// k1..k3 written at 1..3; the local fold has them at cursor 3. Then, while +/// the node is away: an update, a real delete, two new keys, a tail write. +fn evicting_log() -> Vec { + vec![ + ev(1, "n.k1", Some("a")), + ev(2, "n.k2", Some("b")), + ev(3, "n.k3", Some("c")), + ev(4, "n.k2", Some("b2")), + ev(5, "n.k1", None), + ev(6, "n.k4", Some("d")), + ev(7, "n.k5", Some("e")), + ev(8, "n.k6", Some("f")), + ] +} + +/// Three key families: two in a multi-prefix scope (n., m.), one outside +/// (x.), interleaved. +fn multi_log() -> Vec { + vec![ + ev(1, "n.k1", Some("a")), + ev(2, "m.k1", Some("b")), + ev(3, "x.k1", Some("out")), + ev(4, "m.k1", None), + ev(5, "x.k1", Some("out2")), + ev(6, "n.k2", Some("c")), + ev(7, "m.k2", Some("d")), + ev(8, "n.k1", Some("a2")), + ] +} + +/// The normal path: a resume inside the window with an update, a delete, a +/// re-create, a create-then-delete, and a tail. +fn churn_log() -> Vec { + vec![ + ev(1, "n.k1", Some("a")), + ev(2, "n.k2", Some("b")), + ev(3, "n.k3", Some("c")), + ev(4, "n.k1", Some("a2")), + ev(5, "n.k2", None), + ev(6, "n.k2", Some("b3")), + ev(7, "n.tmp", Some("t")), + ev(8, "n.tmp", None), + ev(9, "n.k3", Some("c2")), + ] +} + +/// Keys created or updated just before the artifact's cursor (7) and +/// written again after it: a mid-catch-up exporter at 7 never saw their +/// pre-7 revisions, while the local fold (at 3, or with n.k2's 5 via a +/// torn tail) did. +fn straddle_log() -> Vec { + vec![ + ev(1, "n.k1", Some("a")), + ev(2, "n.k2", Some("b")), + ev(3, "n.k3", Some("c")), + ev(4, "n.k3", Some("c2")), + ev(5, "n.k4", Some("d")), + ev(6, "n.k5", Some("e")), + ev(7, "n.k6", Some("f")), + ev(8, "n.k3", Some("c3")), + ev(9, "n.k4", Some("d2")), + ev(10, "n.k1", None), + ] +} + +fn relist_log() -> Vec { + vec![ + ev(1, "n.k1", Some("a")), + ev(2, "n.k2", Some("b")), + ev(3, "n.k3", Some("c")), + ev(4, "n.new", Some("recreated")), + ev(5, "n.new", None), + ev(6, "n.k1", Some("a2")), + ev(7, "n.k2", Some("b2")), + ev(8, "n.k3", Some("c2")), + ev(9, "n.k4", Some("d")), + ] +} + +fn relist_trip() -> Trip { + Trip { + deliver: 1, + append: relist_log()[4..].to_vec(), + evict_to: None, + purge_markers: true, + } +} + +fn evict_trip(deliver: usize, to: u64) -> Trip { + Trip { + deliver, + append: vec![], + evict_to: Some(to), + purge_markers: false, + } +} + +fn base(name: &'static str, events: Vec) -> Scenario { + Scenario { + name, + events, + local_at: Some(3), + unanchored: false, + evicts_current: false, + floor: 0, + purged: vec![], + trip: None, + artifact_at: 0, + exporter_from: None, + scope: WatchScope::All, + mode: Mode::Auto, + } +} + +fn scenarios() -> Vec { + let evicting = |name, trip, floor, scope| Scenario { + evicts_current: true, + floor, + trip, + artifact_at: 7, + scope, + ..base(name, evicting_log()) + }; + vec![ + // --- Expiry repairs --- + // Resume finds the cursor expired: k3, the k2 update, k4 all aged out + // (never deleted); the k1 delete marker too. Restore from 7. + evicting( + "evicting: resume expired → restore", + None, + 6, + WatchScope::All, + ), + // A live watch delivers 4 and 5, then retention overruns it. + evicting( + "evicting: floor-guard trip mid-watch → restore", + Some(evict_trip(2, 6)), + 3, + WatchScope::All, + ), + evicting( + "evicting: prefix scope, trip → restore", + Some(evict_trip(1, 6)), + 3, + WatchScope::Prefix("n.".into()), + ), + Scenario { + local_at: None, + ..evicting("evicting: fresh start → restore", None, 6, WatchScope::All) + }, + Scenario { + unanchored: true, + ..evicting( + "evicting: data but no cursor → restore", + None, + 6, + WatchScope::All, + ) + }, + // Two prefixes on one consumer, x.* out of scope; restore after a + // trip. The local fold also holds x.k1 (out of scope, untouched). + Scenario { + name: "evicting: multi-prefix, trip → restore", + events: multi_log(), + evicts_current: true, + floor: 3, + trip: Some(evict_trip(1, 6)), + artifact_at: 7, + scope: WatchScope::Prefixes(vec!["n.".into(), "m.".into()]), + ..base("", vec![]) + }, + // The artifact was exported while its exporter was still catching + // up: a fresh full re-list (from 0), or a resume from 2. It lacks + // keys whose latest write is after its cursor and holds superseded + // values for others; the resume from its cursor completes them. + Scenario { + exporter_from: Some(0), + ..evicting( + "evicting: trip → restore from a re-list-hydrating artifact", + Some(evict_trip(2, 6)), + 3, + WatchScope::All, + ) + }, + Scenario { + name: "evicting: expired → restore from a mid-catch-up artifact", + events: straddle_log(), + evicts_current: true, + floor: 6, + artifact_at: 7, + exporter_from: Some(2), + ..base("", vec![]) + }, + Scenario { + name: "keeps-current: resume expired → relist", + purged: vec![5], + mode: Mode::Relist, + ..base("", relist_log()) + }, + // THE RE-QUEUE BUG's shape: the live watch delivers the re-create at + // 4; then the key is deleted again, everything else superseded, and + // the marker purged, overrunning the watch. + Scenario { + name: "keeps-current: trip mid-watch → relist", + trip: Some(relist_trip()), + mode: Mode::Relist, + ..base("", relist_log()[..4].to_vec()) + }, + Scenario { + name: "keeps-current: trip mid-watch → auto picks relist", + trip: Some(relist_trip()), + artifact_at: 9, + ..base("", relist_log()[..4].to_vec()) + }, + Scenario { + name: "keeps-current: data but no cursor → relist", + local_at: Some(4), + unanchored: true, + purged: vec![5], + ..base("", relist_log()) + }, + // Overlapping prefixes (a key matched twice) on a key-listing repair. + Scenario { + name: "keeps-current: overlapping prefixes, expired → relist", + purged: vec![5], + scope: WatchScope::Prefixes(vec!["n.".into(), "n.k".into()]), + mode: Mode::Relist, + ..base("", relist_log()) + }, + // --- The normal data path (no expiry) --- + Scenario { + name: "keeps-current: resume inside the window, churn", + ..base("", churn_log()) + }, + // Evicting bucket, resume inside the window; k3 (at 3) has aged out + // of NATS but the live fold keeps it — never deleted. + Scenario { + name: "evicting: resume inside the window keeps an aged-out key", + evicts_current: true, + floor: 3, + artifact_at: 7, + ..base("", evicting_log()) + }, + Scenario { + name: "keeps-current: fresh full re-list", + local_at: None, + ..base("", churn_log()) + }, + Scenario { + name: "keeps-current: multi-prefix resume inside the window", + scope: WatchScope::Prefixes(vec!["n.".into(), "m.".into()]), + ..base("", multi_log()) + }, + ] +} + +// --- The harness ---------------------------------------------------------- + +struct World { + log: Arc, + restore: Arc>, + path: PathBuf, + dir: tempfile::TempDir, + /// The out-of-scope part of the fold at the start (must never change). + out_of_scope: BTreeMap>, +} + +fn world(sc: &Scenario, yield_between: bool) -> World { + let dir = tempfile::TempDir::new().unwrap(); + let path = dir.path().join("fold"); + let (_c, mut fold) = B::open(&path); + if let Some(at) = sc.local_at { + let seed: Vec = sc + .events + .iter() + .filter(|e| e.rev <= at) + .map(to_update) + .collect(); + let cursor = if sc.unanchored { + WatchCursor::none() + } else { + WatchCursor::from_u64(at) + }; + fold.apply(&seed, &cursor).unwrap(); + } + let prefixes = sc.prefixes(); + let out_of_scope = fold + .range("") + .unwrap() + .into_iter() + .filter(|e| !in_scope(&prefixes, &e.key)) + .map(|e| (e.key, e.value)) + .collect(); + drop(fold); + let log = Arc::new(SimLog(Mutex::new(LogState { + events: sc.events.clone(), + evicts_current: sc.evicts_current, + floor: sc.floor, + purged: sc.purged.iter().copied().collect(), + trip: sc.trip.clone(), + yield_between, + }))); + let restore = Arc::new(SimRestore { + log: Arc::clone(&log), + cursor: sc.artifact_at, + exporter_from: sc.exporter_from, + dir: dir.path().to_path_buf(), + fetches: AtomicUsize::new(0), + _b: std::marker::PhantomData, + }); + World { + log, + restore, + path, + dir, + out_of_scope, + } +} + +fn repair(sc: &Scenario, w: &World) -> ExpiryRepair> { + match sc.mode { + Mode::Relist => ExpiryRepair::Relist(Arc::clone(&w.log) as Arc), + Mode::Auto => ExpiryRepair::Auto { + reader: Arc::clone(&w.log) as Arc, + restore: Arc::clone(&w.restore) as Arc>>, + }, + } +} + +enum RunEnd { + Clean, + Crashed, +} + +type Exports = Arc< + Mutex< + Vec<( + oneshot::Receiver>, + PathBuf, + )>, + >, +>; + +struct RunResult { + end: RunEnd, + reported: Vec, + domain: HashMap>, + /// Keys whose revision `apply` saw go backward. + regressions: Vec, + /// Keys `apply` saw deleted by a repair (revisionless delete). + synthetic_deletes: Vec, + /// Store-apply calls this run made. + calls: usize, +} + +/// One process lifetime: open the fold, rebuild domain state from it, run +/// `watch_applied` to the end of the (simulated) stream or a crash. With +/// `export`, the first `on_applied` of the run requests an export. +async fn run_once( + sc: &Scenario, + w: &World, + faults: RunFaults, + max: usize, + export: bool, + exports: &Exports, + run: usize, +) -> RunResult { + let (cursor, inner) = B::open(&w.path); + let start = inner.range("").unwrap(); + let domain: Arc>>> = Arc::new(Mutex::new( + start + .iter() + .map(|e| (e.key.clone(), e.value.clone())) + .collect(), + )); + // The highest revision `apply` has seen per key (seeded from the fold). + let seen_rev: Arc>> = Arc::new(Mutex::new( + start + .iter() + .filter_map(|e| Some((e.key.clone(), e.version.as_u64()?))) + .collect(), + )); + let regressions = Arc::new(Mutex::new(Vec::new())); + let synthetic = Arc::new(Mutex::new(Vec::new())); + let reported = Arc::new(Mutex::new(Vec::new())); + let calls = Arc::new(AtomicUsize::new(0)); + let store = FaultStore { + inner, + faults, + calls: Arc::clone(&calls), + }; + let (_sd_tx, sd_rx) = tokio::sync::watch::channel(false); + let (ex_tx, ex_rx) = mpsc::channel::(4); + let (d, r) = (Arc::clone(&domain), Arc::clone(&reported)); + let (seen, regr, synth) = ( + Arc::clone(&seen_rev), + Arc::clone(®ressions), + Arc::clone(&synthetic), + ); + let (exports_c, export_dir) = ( + Arc::clone(exports), + w.dir.path().join(format!("export-{run}")), + ); + let mut requested = false; + let res = watch_applied( + Arc::clone(&w.log) as Arc, + sc.scope.clone(), + (!cursor.is_none()).then_some(cursor), + repair(sc, w), + Some(store), + Some(ex_rx), + BatchConfig { + window: std::time::Duration::from_secs(3600), + max, + ..BatchConfig::default() + }, + |u: &KvUpdate| { + Some(match u { + KvUpdate::Put(e) => (e.key.clone(), Some(e.value.clone()), e.version.as_u64()), + KvUpdate::Delete { key, version } | KvUpdate::Purge { key, version } => { + (key.clone(), None, version.as_u64()) + } + }) + }, + move |batch: Vec<(String, Option>, Option)>| { + let mut d = d.lock().unwrap(); + let mut seen = seen.lock().unwrap(); + for (k, v, rev) in batch { + match rev { + Some(rev) => { + let max = seen.entry(k.clone()).or_insert(0); + if rev < *max { + regr.lock().unwrap().push(format!("{k}: {max} → {rev}")); + } + *max = (*max).max(rev); + } + None if v.is_none() => synth.lock().unwrap().push(k.clone()), + None => {} + } + match v { + Some(v) => { + d.insert(k, v); + } + None => { + d.remove(&k); + } + } + } + }, + move |c: WatchCursor| { + r.lock().unwrap().push(c.as_u64().unwrap()); + if export && !requested { + requested = true; + let (reply, rx) = oneshot::channel(); + let dest = export_dir.join("artifact"); + std::fs::create_dir_all(&export_dir).unwrap(); + if ex_tx + .try_send(ExportRequest { + dest_dir: dest.clone(), + reply, + }) + .is_ok() + { + exports_c.lock().unwrap().push((rx, dest)); + } + } + }, + sd_rx, + ) + .await; + let reported = reported.lock().unwrap().clone(); + let domain = domain.lock().unwrap().clone(); + let end = match res { + Ok(_) => RunEnd::Clean, + // A crash, or the fail-stop after a persistent store failure: either + // way the process restarts. + Err(KvError::WatchError(msg)) + if msg.contains("panicked") || msg.contains("consecutive times") => + { + RunEnd::Crashed + } + Err(e) => panic!("[{}] unexpected watch error: {e}", sc.name), + }; + RunResult { + end, + reported, + domain, + regressions: regressions.lock().unwrap().clone(), + synthetic_deletes: synthetic.lock().unwrap().clone(), + calls: calls.load(Ordering::SeqCst), + } +} + +/// The artifact plus a resume from its cursor reaches the truth: applying, +/// on top of `fold`, the latest message per in-scope key written after +/// `cursor` yields every write minus every real delete. This is what a +/// NATS cursor means (every RETAINED message at or below it is applied — +/// not "the truth at the cursor": a key whose latest write is after it +/// hasn't been delivered yet); the restore's freshness check guarantees the +/// tail is retained. +fn replay_complete( + events: &[Event], + prefixes: &[String], + fold: &BTreeMap>, + cursor: u64, + head: u64, +) -> Result<(), String> { + let mut state: BTreeMap> = fold + .iter() + .filter(|(k, _)| in_scope(prefixes, k)) + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + let mut latest: BTreeMap<&str, &Event> = BTreeMap::new(); + for e in events.iter().filter(|e| e.rev <= head) { + latest.insert(e.key, e); + } + for e in latest + .into_values() + .filter(|e| e.rev > cursor && in_scope(prefixes, e.key)) + { + match e.value { + Some(v) => { + state.insert(e.key.to_string(), v.as_bytes().to_vec()); + } + None => { + state.remove(e.key); + } + } + } + let want: BTreeMap> = truth(events, head) + .into_iter() + .filter(|(k, _)| in_scope(prefixes, k)) + .map(|(k, (v, _))| (k, v)) + .collect(); + if state == want { + Ok(()) + } else { + Err(format!( + "artifact + tail after {cursor} = {state:?}, truth = {want:?}" + )) + } +} + +#[derive(Clone, Copy)] +struct Cfg { + yield_between: bool, + max: usize, + export: bool, +} + +/// Drive a schedule (faults per run) to convergence and check the outcome. +/// Returns the number of store-apply calls the first run made. +async fn check_schedule(sc: &Scenario, cfg: Cfg, schedule: &[RunFaults]) -> usize { + let w = world::(sc, cfg.yield_between); + let ctx = format!( + "[{} / {}] yield={} max={} schedule={schedule:?}", + B::NAME, + sc.name, + cfg.yield_between, + cfg.max + ); + let prefixes = sc.prefixes(); + let exports: Exports = Arc::new(Mutex::new(Vec::new())); + let mut last_cursor = B::open(&w.path).0.as_u64().unwrap_or(0); + let mut first_calls = None; + let mut run = 0usize; + loop { + let faults = schedule.get(run).cloned().unwrap_or_default(); + let faultless = faults.is_empty() && run >= schedule.len(); + let res = run_once::(sc, &w, faults, cfg.max, cfg.export, &exports, run).await; + first_calls.get_or_insert(res.calls); + assert!( + res.reported.windows(2).all(|p| p[0] <= p[1]), + "{ctx}: on_applied went backward within a run: {:?}", + res.reported + ); + // The consumer never sees a key's revision go backward, and a repair + // never deletes a key that is live in the log (the log is static + // once a repair can run). + assert!( + res.regressions.is_empty(), + "{ctx}: apply saw revisions go backward: {:?}", + res.regressions + ); + { + let st = w.log.0.lock().unwrap(); + let live = truth(&st.events, st.head()); + let phantom: Vec<&String> = res + .synthetic_deletes + .iter() + .filter(|k| live.contains_key(k.as_str())) + .collect(); + assert!( + phantom.is_empty(), + "{ctx}: a repair deleted keys that are live: {phantom:?}" + ); + } + let (cursor, fold) = B::open(&w.path); + let cursor = cursor.as_u64().unwrap_or(0); + assert!( + cursor >= last_cursor, + "{ctx}: the fold's cursor went backward across run {run}: {last_cursor} → {cursor}" + ); + last_cursor = cursor; + run += 1; + assert!(run < 20, "{ctx}: no convergence after {run} runs"); + if !(faultless && matches!(res.end, RunEnd::Clean)) { + drop(fold); + continue; + } + + // Converged run: judge it. + let (head, events) = { + let st = w.log.0.lock().unwrap(); + (st.head(), st.events.clone()) + }; + let want: BTreeMap, u64)> = truth(&events, head) + .into_iter() + .filter(|(k, _)| in_scope(&prefixes, k)) + .collect(); + let all: Vec = fold.range("").unwrap(); + let got: BTreeMap, u64)> = all + .iter() + .filter(|e| in_scope(&prefixes, &e.key)) + .map(|e| { + ( + e.key.clone(), + (e.value.clone(), e.version.as_u64().unwrap()), + ) + }) + .collect(); + assert_eq!( + got, want, + "{ctx}: the fold is not every write minus every real delete" + ); + let in_scope_head = events + .iter() + .filter(|e| in_scope(&prefixes, e.key)) + .map(|e| e.rev) + .max() + .unwrap_or(0); + assert!( + cursor >= in_scope_head && cursor <= head, + "{ctx}: cursor {cursor} short of the scope's head {in_scope_head}" + ); + let out: BTreeMap> = all + .iter() + .filter(|e| !in_scope(&prefixes, &e.key)) + .map(|e| (e.key.clone(), e.value.clone())) + .collect(); + assert_eq!(out, w.out_of_scope, "{ctx}: out-of-scope keys changed"); + let fold_map: HashMap> = all + .iter() + .map(|e| (e.key.clone(), e.value.clone())) + .collect(); + assert_eq!( + res.domain, fold_map, + "{ctx}: the domain state (apply) diverged from the fold" + ); + drop(fold); + + // Every export that completed: scope recorded, never behind its cursor. + let pending = std::mem::take(&mut *exports.lock().unwrap()); + for (mut rx, artifact) in pending { + let Ok(Ok(manifest)) = rx.try_recv() else { + continue; + }; + assert_eq!( + manifest.scope.as_ref(), + Some(&prefixes), + "{ctx}: export scope" + ); + let dest = artifact.with_file_name("imported"); + let (c, imported) = B::import(&artifact, &dest); + assert_eq!(c, manifest.cursor, "{ctx}: imported cursor"); + let contents: BTreeMap> = imported + .range("") + .unwrap() + .into_iter() + .map(|e| (e.key, e.value)) + .collect(); + replay_complete(&events, &prefixes, &contents, c.as_u64().unwrap_or(0), head) + .unwrap_or_else(|e| { + panic!("{ctx}: exported artifact can't resume to the truth: {e}") + }); + } + return first_calls.unwrap(); + } +} + +fn quiet_simulated_crashes() { + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let default = std::panic::take_hook(); + std::panic::set_hook(Box::new(move |info| { + let msg = info + .payload() + .downcast_ref::() + .map(String::as_str) + .or_else(|| info.payload().downcast_ref::<&str>().copied()) + .unwrap_or(""); + if !msg.contains(CRASH_MSG) { + default(info); + } + })); + }); +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Depth { + /// Every ≤2-transient set, single and paired crashes, transient + crash. + Full, + /// Every single fault. + Single, +} + +/// Every schedule for one scenario configuration. Returns how many ran. +async fn exhaust(sc: &Scenario, cfg: Cfg, depth: Depth) -> usize { + let k = check_schedule::(sc, cfg, &[]).await; + // A crash index can also land past the first run's calls (a later run — + // e.g. a restore re-run after a crash — makes more): probe beyond. + let calls: Vec = (0..=k).collect(); + let later_calls: Vec = (0..=k + 4).collect(); + let kinds = [Crash::Before, Crash::After, Crash::Torn]; + let mut schedules: Vec> = Vec::new(); + for a in &calls { + schedules.push(vec![RunFaults { + transient: vec![*a], + crash: None, + }]); + for kind in kinds { + schedules.push(vec![RunFaults { + transient: vec![], + crash: Some((*a, kind)), + }]); + } + if depth == Depth::Single { + continue; + } + for b in calls.iter().filter(|b| *b > a) { + schedules.push(vec![RunFaults { + transient: vec![*a, *b], + crash: None, + }]); + } + for kind in kinds { + let crash = RunFaults { + transient: vec![], + crash: Some((*a, kind)), + }; + for b in &later_calls { + for kind2 in kinds { + schedules.push(vec![ + crash.clone(), + RunFaults { + transient: vec![], + crash: Some((*b, kind2)), + }, + ]); + } + } + for t in calls.iter().filter(|t| *t < a) { + schedules.push(vec![RunFaults { + transient: vec![*t], + crash: Some((*a, kind)), + }]); + } + } + } + // A persistent store failure: every apply fails until the watch + // fail-stops; the restart must still converge. + schedules.push(vec![RunFaults { + transient: (0..64).collect(), + crash: None, + }]); + for s in &schedules { + check_schedule::(sc, cfg, s).await; + } + schedules.len() + 1 +} + +async fn exhaust_backend(depth: Depth, maxes: &[usize], yields: &[bool]) -> usize { + quiet_simulated_crashes(); + let mut total = 0usize; + for sc in scenarios() { + for &yield_between in yields { + for &max in maxes { + let cfg = Cfg { + yield_between, + max, + export: true, + }; + let n = exhaust::(&sc, cfg, depth).await; + println!( + "{} / {} yield={yield_between} max={max}: {n} schedules", + B::NAME, + sc.name + ); + total += n; + } + } + } + println!("{}: {total} schedules", B::NAME); + total +} + +#[tokio::test(flavor = "current_thread")] +async fn append_log_every_fault_schedule_converges() { + let total = exhaust_backend::(Depth::Full, &[1, 100], &[false, true]).await; + assert!( + total > 20_000, + "the schedule space is not vacuous ({total})" + ); +} + +// The LSM backends: every single fault, every scenario, batches of one (the +// most store applies, so the most crash points) in one delivery order — a +// CI-sized slice. The deep tier adds batches of 100 and the other delivery +// order (~11 min for both backends, dominated by filesystem syncs). + +#[cfg(feature = "fjall")] +#[tokio::test(flavor = "current_thread")] +async fn fjall_every_single_fault_converges() { + let total = exhaust_backend::(Depth::Single, &[1], &[false]).await; + assert!(total > 400, "the schedule space is not vacuous ({total})"); +} + +#[cfg(feature = "rocksdb")] +#[tokio::test(flavor = "current_thread")] +async fn rocksdb_every_single_fault_converges() { + let total = exhaust_backend::(Depth::Single, &[1], &[false]).await; + assert!(total > 400, "the schedule space is not vacuous ({total})"); +} + +#[cfg(feature = "fjall")] +#[tokio::test(flavor = "current_thread")] +#[ignore = "deep tier: ~6 min"] +async fn deep_fjall_every_single_fault_both_orders_both_batch_sizes() { + exhaust_backend::(Depth::Single, &[1, 100], &[false, true]).await; +} + +#[cfg(feature = "rocksdb")] +#[tokio::test(flavor = "current_thread")] +#[ignore = "deep tier: ~6 min"] +async fn deep_rocksdb_every_single_fault_both_orders_both_batch_sizes() { + exhaust_backend::(Depth::Single, &[1, 100], &[false, true]).await; +}