Skip to content

Retire the "fold is a cache of NATS" model the store layer still assumed - #22

Merged
jaredLunde merged 2 commits into
mainfrom
jared/replica-of-record
Oct 1, 2026
Merged

jaredLunde merged 2 commits into
mainfrom
jared/replica-of-record

Conversation

@jaredLunde

@jaredLunde jaredLunde commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Why

slipstream's design is a bounded log: NATS keeps only the retained tail, and folds are the replicas of record. The store layer and its docs still assumed an older model, "NATS is the source of truth, the snapshot is a cache". Every cursor-expiry bug fixed in #21 was a component running on that model:

  • the resync deleted whatever NATS no longer listed;
  • a fold with data but no cursor was treated as empty;
  • a restore assumed an artifact at C is the truth at C;
  • on_applied's cursor was treated as durable.

This PR removes the remaining instances, so the code and docs state one model.

Behavior changes

  • delete_with_version tombstones reach watchers, and so folds, as deletes. A tombstone is an empty-value write, and it now arrives as a Delete carrying its own revision. Previously the fold kept the key as present-but-empty while get/scan/keys and the repairs' key listings treated it as deleted, so the key-listing repair deleted it and its re-list put it straight back. entry() still exposes the raw tombstone for CAS callers (the export lease's takeover depends on it).
  • watch_applied refuses ExpiryRepair::Restore/Auto without a store. A restore needs a fold to restore into; such a consumer silently held only NATS's retained view. ExpiryRepair::None remains the explicit way to accept that view.
  • A warning when a cursor-less start with no repair armed re-lists a bucket that has already evicted current values.

Docs

  • The store invariants are now fold + tail = truth (a cursor means every retained message at or below it is applied), cursor-after-apply, and replica of record.
  • The durability rationale and recovery runbooks no longer say "delete the snapshot and replay NATS", which on a bounded log loses everything evicted. A lost or corrupt fold is rebuilt by importing an artifact.

New edge, pinned and documented, not fixed

The resume check is conservative: it sees only the stream's first retained revision. When every message after a node's cursor has been superseded (every key rewritten while the node was down), the cursor looks expired even though nothing was evicted. On an evicting bucket with no artifact ahead of the node, the watch fail-stops until the next export round. That's safe but unavailable, and a small hot bucket (heartbeat-style) hits it on a short restart.

tests/eviction.rs::supersession_only_restart_fail_stops_without_a_newer_artifact pins it against a live server. Telling supersession from eviction needs information the stream doesn't expose today (e.g. the cursor's message timestamp against max_age); that's a follow-up design.

Tests

  • Live tombstone-as-delete through the plain watch and the multi-prefix resume consumer. A race in the first version was fixed (attach the resume consumer before anything supersedes its cursor); it then passed 100/100 runs under 4× parallel load.
  • Store-less restore refused (unit).
  • The supersession-only fail-stop, pinned live.

🤖 Generated with Claude Code

https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr

Base automatically changed from jared/correctness to main October 1, 2026 23:14
jaredLunde and others added 2 commits October 1, 2026 16:14
slipstream's design is a bounded log: folds are the replicas of record and
NATS holds only the retained tail. The store layer and its docs still
assumed the older model — "NATS is the source of truth, the snapshot is a
cache" — and every cursor-expiry bug fixed in the previous PR was a
component running on it. This removes the remaining instances.

Behavior:
- A delete_with_version tombstone (an empty-value Put) now reaches watchers,
  and so folds, as a Delete carrying the tombstone's revision. The fold held
  it as a present-but-empty key while get/scan/keys and the repairs' key
  listings treated it as deleted, so the key-listing repair deleted it and
  then re-listed it. entry() still exposes the raw tombstone for CAS.
- watch_applied refuses ExpiryRepair::Restore/Auto without a store (a
  restore needs a fold to restore into; such a consumer silently held
  NATS's retained view), and warns when a cursor-less start with no repair
  armed re-lists a bucket that has already evicted current values.

Docs: the store invariants are now fold + tail = truth, cursor-after-apply,
and replica of record. The durability rationale and recovery runbooks no
longer say "delete the snapshot and replay NATS" (on a bounded log that
loses everything evicted); a lost or corrupt fold is rebuilt by importing
an artifact.

Tests: live tombstone-as-delete through the plain watch and the
multi-prefix resume consumer (made race-free: attach the resume consumer
before anything supersedes its cursor; 100/100 under parallel load);
store-less restore refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
The resume check sees only the stream's first retained revision, so a node
whose post-cursor messages were all superseded (every key rewritten while
it was down) looks expired though nothing was evicted. On an evicting
bucket with no artifact ahead of the node, the restore refuses and the
watch fail-stops: safe, but unavailable until the next export round.
Pinned live and documented as a failure mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
@jaredLunde
jaredLunde force-pushed the jared/replica-of-record branch from 22ec039 to d7c4932 Compare October 1, 2026 23:14
@jaredLunde
jaredLunde merged commit 6102135 into main Oct 1, 2026
2 checks passed
@jaredLunde
jaredLunde deleted the jared/replica-of-record branch October 1, 2026 23:54
@jaredLunde jaredLunde mentioned this pull request Oct 2, 2026
jaredLunde added a commit that referenced this pull request Oct 2, 2026
Cursor-expiry repair for bounded logs (#21) and the replica-of-record
cleanup (#22). Breaking: ExpiryRepair replaces the reader argument (the old
Option<reader> still compiles), Relist is refused on buckets that evict
current values, Restore/Auto require a store, artifact manifests move to
schema 2 (upgrade importers before exporters), delete_with_version
tombstones reach watchers as deletes, and the floor guard reports
CursorExpired.

Also moves the lockfile off yanked fjall/lsm-tree 3.1.4 to 3.1.10 (and
chacha20, spin), so the suite runs on what dependents resolve, and
refreshes the README install snippets to 0.8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
jaredLunde added a commit that referenced this pull request Oct 2, 2026
Cursor-expiry repair for bounded logs (#21) and the replica-of-record
cleanup (#22). Breaking: ExpiryRepair replaces the reader argument (the old
Option<reader> still compiles), Relist is refused on buckets that evict
current values, Restore/Auto require a store, artifact manifests move to
schema 2 (upgrade importers before exporters), delete_with_version
tombstones reach watchers as deletes, and the floor guard reports
CursorExpired.

Also moves the lockfile off yanked fjall/lsm-tree 3.1.4 to 3.1.10 (and
chacha20, spin), so the suite runs on what dependents resolve, and
refreshes the README install snippets to 0.8.


Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant