Skip to content

Fix three cursor-expiry repair gaps (eviction history, overwrite-only gaps, restore memory) - #25

Merged
jaredLunde merged 1 commit into
mainfrom
fix/repair-eviction-history-and-memory
Oct 3, 2026
Merged

jaredLunde merged 1 commit into
mainfrom
fix/repair-eviction-history-and-memory

Conversation

@jaredLunde

Copy link
Copy Markdown
Contributor

Summary

  • Eviction turned off no longer deletes aged-out keys. Whether NATS's key list could be trusted was decided from the bucket's current config. Turn max_age / discard: old off (or recreate the bucket), and Auto relisted and deleted keys that had aged out while eviction was on, though the local copies held the only remaining record of them. A fold now remembers, on its cursor, that it saw eviction (once the log is past its first revision), and never trusts the listing again. The flag rides in exports, and a cursor without it keeps the old encoding byte for byte. Retention is read at every start, every expiry, and every 60 s while a relist is still possible.
  • Overwrite-only gaps resume instead of fail-stopping. On a max_age-only bucket, a cursor whose gap is all overwrites looked expired and needed an export newer than the node. A flagged cursor now keeps its message's server timestamp. If the cursor's message is at most half of max_age old by NATS's clock (ts in stream info), nothing after it can have aged out, so the watch resumes from the first retained revision. discard: old limits and per-message TTLs still fail-stop.
  • Restore memory is bounded. The restore held every live key name in memory. Rewrites now spill to scratch, only delete candidates are held, and the listing is streamed past them (KvReader::for_each_key, a provided method). The model checker caught that moving the listing after the diff needs a retention recheck, so that's added and pinned (mutation NoListingRecheck).

All API changes are additive (provided trait method, builder method, pub WatchCursor::to_bytes/from_bytes), so this fits a patch release. On-disk caveat: a fold on an evicting bucket that gets the new flag can't be read by 0.8.0 (it refuses loudly, it doesn't misread). Folds on buckets that never evict are unchanged.

Test plan

  • cargo clippy --all-targets --all-features -D warnings, cargo doc
  • cargo test --lib --tests --features fjall,rocksdb,transport. Two live eviction tests flaked once under machine memory pressure, then passed 3/3 standalone.
  • Model checker: new eviction_turned_off_repair_steps_are_correct; mutations NoEvictionMemo and NoListingRecheck are each caught
  • Live NATS: eviction_turned_off_never_trusts_the_listing_again, supersession_only_restart_resumes_on_a_max_age_bucket (flips the old pinned fail-stop), supersession_gap_still_expires_when_eviction_is_not_only_by_age

🤖 Generated with Claude Code

https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr

…nly gaps, restore memory

- A fold now remembers, on its cursor, that it saw its bucket evict current
  values. Turning max_age / discard:old off later (or recreating the bucket)
  no longer makes the key listing look trustworthy, so Auto keeps restoring
  instead of deleting keys that aged out. The memo is carried in artifacts.
- On a max_age-only bucket, a resume whose gap was all overwrites (nothing
  young enough to have aged out) resumes from the first retained revision
  instead of fail-stopping for want of a newer artifact. Uses the cursor
  message's server timestamp and the server's clock, with half of max_age as
  headroom.
- The restore no longer holds every live key name: rewrites spill to scratch,
  only delete candidates stay in memory, and the listing is streamed past
  them (and re-checked against retention, which the model checker showed is
  needed once the listing moves after the diff).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
@jaredLunde
jaredLunde merged commit fd1bff6 into main Oct 3, 2026
1 check passed
@jaredLunde
jaredLunde deleted the fix/repair-eviction-history-and-memory branch October 3, 2026 19:58
@jaredLunde jaredLunde mentioned this pull request Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant