Repository navigation
Bound the key-listing resync's memory (external sort + merge-join) - #27
Merged
Merged
Conversation
…e-join the fold The cursor-expiry key-listing diff held every live key name (a Vec plus a HashSet), which at a billion keys exhausts memory on exactly the on-disk backends built for that size. The listing now streams into an external sort (64 MiB runs spilled to scratch) and is merge-joined against the fold's own key-ordered scan; only the stale keys are held. Scratch is the restore source's directory under Auto, TMPDIR under Relist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Relist, andAutoon non-evicting buckets) held every live key name in memory. That's tens of GB at a billion keys, on exactly the RocksDB/fjall deployments built for that size.KvReader::for_each_key) into an external sort: 64 MiB runs, spilled and k-way merged. It's walked alongside the fold's own key-ordered scan, which all three backends provide in byte order. Memory is one run plus the stale keys.Auto(new provided methodRestoreSource::scratch_dir, whichArtifactRestoreimplements) andTMPDIRunderRelist. Stale deletes now flush in batch-sized chunks, still all before the ack.relist_holds_without_the_ack_barrierpins that.All changes are additive, so this fits a patch release.
Test plan
-D warnings(all targets, all features), docskey_sorter_matches_sort_dedup_at_every_run_size(one run, a few, one per key; duplicates within and across runs)model_repair: all properties hold, every mutation is still caught🤖 Generated with Claude Code
https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr