Skip to content

Bound the key-listing resync's memory (external sort + merge-join) - #27

Merged
jaredLunde merged 1 commit into
mainfrom
fix/relist-bounded-memory
Oct 3, 2026
Merged

jaredLunde merged 1 commit into
mainfrom
fix/relist-bounded-memory

Conversation

@jaredLunde

Copy link
Copy Markdown
Contributor

Summary

  • The cursor-expiry key-listing diff (Relist, and Auto on non-evicting buckets) held every live key name in memory. That's tens of GB at a billion keys, on exactly the RocksDB/fjall deployments built for that size.
  • The listing now streams (KvReader::for_each_key) into an external sort: 64 MiB runs, spilled and k-way merged. It's walked alongside the fold's own key-ordered scan, which all three backends provide in byte order. Memory is one run plus the stale keys.
  • Scratch is the restore source's directory under Auto (new provided method RestoreSource::scratch_dir, which ArtifactRestore implements) and TMPDIR under Relist. Stale deletes now flush in batch-sized chunks, still all before the ack.
  • Model: the listing moved into the main loop, after the drain. The ack barrier is no longer load-bearing (axiom 6 suffices), so it's kept as defense in depth, and a new check relist_holds_without_the_ack_barrier pins that.

All changes are additive, so this fits a patch release.

Test plan

  • clippy -D warnings (all targets, all features), docs
  • key_sorter_matches_sort_dedup_at_every_run_size (one run, a few, one per key; duplicates within and across runs)
  • model_repair: all properties hold, every mutation is still caught
  • full suite with fjall, rocksdb and transport (CI)

🤖 Generated with Claude Code

https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr

…e-join the fold

The cursor-expiry key-listing diff held every live key name (a Vec plus a
HashSet), which at a billion keys exhausts memory on exactly the on-disk
backends built for that size. The listing now streams into an external sort
(64 MiB runs spilled to scratch) and is merge-joined against the fold's own
key-ordered scan; only the stale keys are held. Scratch is the restore
source's directory under Auto, TMPDIR under Relist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
@jaredLunde
jaredLunde merged commit f2c7eb0 into main Oct 3, 2026
1 check passed
@jaredLunde
jaredLunde deleted the fix/relist-bounded-memory branch October 3, 2026 21:54
@jaredLunde jaredLunde mentioned this pull request Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant