[Store] Add snapshot-validated batch OpLog pruning - #4142
Conversation
he-yufeng
left a comment
There was a problem hiding this comment.
Mechanism review of the coordinator and its tests (static, line level; no local build, per my review scope):
The contract chain from #4112 is honored end to end. The floor advance precedes deletion, it is computed at the fallback snapshot's batch id (so the latest snapshot's own batches are never in scope), the monotonicity guard (candidate < floor returns OK) keeps concurrent or repeated passes from walking the floor backwards, and the advance itself is a compare-and-swap on the old floor value (or its absence) under the maintenance lease, so two coordinators cannot clobber each other. The corrupt-floor branch (parse must consume the full stored value) fails closed instead of treating garbage as a floor. Only after the CAS does DeleteBatchesThrough run, and its idempotency is what makes the acknowledged crash window (floor advanced, delete failed) recoverable by the next successful publication instead of a leak.
Two integration questions, neither blocking this PR:
- The floor is written here, but who reads it? A
BuildBatchOpLogSnapshotCompactionFloorKeyconsumer on the standby restore/rebootstrap path is what makes the floor reader-visible in practice. If that reader lands separately, the rollout note in the docs (promoted standbys must understand the floor protocol before pruning starts) is doing real work and is worth keeping bold. - The fallback-anchored floor means pruning always trails one publication behind. That is the safe choice and reads deliberately conservative; just confirming the first snapshot intentionally never prunes is the documented behavior, since an operator watching disk growth after publication one might otherwise read it as a bug.
Boundary note: no overlap with #3806 (standby index restore tolerance) or #3135 (re-offload idempotency) on my side; the layers are different.
Thanks for the review. To clarify the two integration points:
|
Description
Adds the N12 batch OpLog pruning coordinator. After a successful snapshot publication, it independently revalidates latest/fallback descriptors and artifacts, atomically advances the reader-visible compaction floor under the maintenance lease, then invokes bounded batch deletion. GC and pruning failures remain best-effort and never invalidate an already published snapshot.
Module
mooncake-store)Type of Change
How Has This Been Tested?
Test commands:
Test results:
Checklist
./scripts/code_format.shAI Assistance Disclosure