[Store] Rebootstrap standbys behind the compaction floor

[Icedcoco]

Description

Implement N09 compaction-floor awareness and online rebootstrap for batch OpLog standbys, following RFC #3167.

When a standby cursor falls behind /oplog/<cluster>/snapshot/compaction_floor, polling returns REBOOTSTRAP_REQUIRED. The service pauses normal apply and becomes non-promotable while restoring an eligible latest/fallback snapshot and replaying its suffix into temporary state. Successful recovery atomically replaces the store, registry, applier, reader and cursor; failed attempts retain the old state and retry with backoff.

The reader uses the current floor without retaining deleted keys, rechecks the floor when pruning races with polling, and accepts a proven cursor at the floor even if its historical batch has been deleted. Unknown suffix gaps remain fatal. Promotion and stop are guarded against recovery/swap races.

This PR does not advance the floor, prune batches, or collect snapshot objects. Production pruning remains disabled until all eligible standbys have deployed and validated N09.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Debug build with BUILD_UNIT_TESTS=ON, MOONCAKE_ENABLE_TEST_FAILPOINTS=ON, USE_CUDA=OFF, USE_ETCD=OFF, STORE_USE_ETCD=OFF, and WITH_STORE_RUST=OFF. Tests use an in-memory KV backend and local snapshot objects.

Test commands:

cmake --build build-n09 --target oplog_batch_standby_reader_test batch_oplog_snapshot_provider_test batch_oplog_snapshot_rebootstrap_test hot_standby_service_test batch_oplog_snapshot_promotion_test -j32
ctest --test-dir build-n09 -R '^(oplog_batch_standby_reader_test|batch_oplog_snapshot_provider_test|batch_oplog_snapshot_rebootstrap_test|hot_standby_service_test|batch_oplog_snapshot_promotion_test)$' --output-on-failure
pre-commit run --files $(git diff --name-only --diff-filter=ACMR upstream/main...HEAD)

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done (describe below)

All five test targets pass. Coverage includes absent/deleted/invalid floors, cursor boundaries, unknown gaps, floor advancement during poll and before swap, corrupt latest with healthy fallback, suffix replay, failed recovery preserving old objects/segments, online retry after repair, promotion during recovery, successful promotion after recovery, and stop during swap. PR-scoped pre-commit hooks pass. Real-etcd E2E and production rollout validation have not been run.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

Formatting ran through the repository pre-commit hook. Production changes are below 500 LOC excluding tests; related design discussion is RFC #3167. Human review remains pending while this PR is a draft.

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

OpenAI Codex assisted with implementation, tests, code review, and this description. The human submitter must review every changed line and be able to defend the change end-to-end before marking it ready for review.