[Store] Freeze restored snapshot test eviction worker - #4156
Merged
Merged
Conversation
Icedcoco
marked this pull request as ready for review
September 16, 2026 07:40
Icedcoco
requested review from
00fish0,
Aionw,
Libotry and
YiXR
as code owners
September 16, 2026 07:40
Aionw
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The restored MasterService starts an eviction worker before the snapshot test can freeze it. When
EvictObjectleaves memory above the watermark with expired leases, that worker changes replica metadata before the second snapshot and state comparison. In the failing PR #4142 CI run, the restored worker evicted 1042 replicas and the metadata snapshot shrank from 737277 to 723791 bytes.Set the restored instance's memory eviction watermark to 1.0 before construction, then stop and join its eviction worker before snapshot comparison, matching the original instance's existing freeze. This prevents watermark eviction during startup as well as subsequent worker mutations during comparison. Uses the upstream
MasterServiceTestPeeraccessors; only the shared snapshot test fixture changes.Related: #4136. This addresses the restore-side eviction race observed in run 34979380422, not the separate tenant-quota scenario mentioned in that issue. PR #4143 records the same flaky snapshot test but does not modify it. No N12 OpLog pruning changes are included.
Module
mooncake-store)Type of Change
How Has This Been Tested?
Rebuilt locally after rebasing onto upstream/main
f86289c90, with RelWithDebInfo, USE_CUDA=OFF, STORE_USE_ETCD=ON.Test commands:
cmake --build /tmp/mooncake-n12-build --target master_service_ssd_test_for_snapshot master_service_test_for_snapshot master_service_promotion_test_for_snapshot -j 16 /tmp/mooncake-n12-build/mooncake-store/tests/master_service_ssd_test_for_snapshot --gtest_filter=MasterServiceSSDSnapshotTest.EvictObject --gtest_repeat=10 --gtest_break_on_failure ctest --test-dir /tmp/mooncake-n12-build --output-on-failure -R '^(master_service_ssd_test_for_snapshot|master_service_test_for_snapshot|master_service_promotion_test_for_snapshot)$' pre-commit run --files mooncake-store/tests/ha/snapshot/master_service_test_for_snapshot_base.hTest results:
Checklist
AI Assistance Disclosure