[Store] Wire opt-in batch OpLog snapshots into standby runtime

[Icedcoco]

Description

The batch OpLog snapshot components are not yet reachable from the production standby factory. This draft adds default-off enable_oplog_snapshot, propagates chunk sizing through master configuration, and wires the batch provider/coordinator into the standby lifecycle. Artifacts use a per-cluster root. It also includes the N07 legacy promotion timeout compatibility fix and initial E2E launcher options.

Related to #3167 and #3808; follows the components merged in #3794 and #3841. This implements the batch snapshot production switch, distinct from the legacy child-process shutdown work in #3942 and the broader HA refactor in #1776. It does not complete the N08 production-readiness gate.

Scope and follow-up

This PR supplies the default-off production wiring and a real-etcd primary-path gate. Static batch snapshot dependency errors now terminate supervisor startup before the admin server or leadership acquisition. The serving primary bypasses legacy snapshot stores/restore in the new mode, even when legacy flags are set. Controller shutdown joins workers before releasing dependencies.

The new run_oplog_snapshot_smoke.sh uses two real master processes, a local etcd and a persistent storage provider. It verifies two publications, multiple chunks, snapshot cold restore with a nonempty suffix, repeated standby restarts, promotion, 19 surviving objects and one removed object. It also exercises invalid local-store startup configurations and the controller lifecycle test. Launcher regression tests cover failpoint environment on/off and old run-directory defaults.

The following remain for the N08 follow-up: corruption/fallback/full-replay fault matrix, crash points, competing snapshot leases, storage outage, target-hardware freeze/catch-up/RSS/upload measurements, documentation and CI/nightly integration. Passing this primary-path smoke does not establish full N08 production readiness. Human review remains pending; this PR stays draft.

Module

• Mooncake Store (mooncake-store)
• CI/CD

Type of Change

• Bug fix
• New feature

How Has This Been Tested?

Tested on the current worktree with STORE_USE_ETCD=ON, local filesystem snapshots and TCP:

cmake --build build-n08 --target mooncake_master oplog_batch_inspector oplog_ha_client hot_standby_snapshot_bootstrap_test hot_standby_service_test -j8
./build-n08/mooncake-store/tests/hot_standby_snapshot_bootstrap_test
GTEST_FILTER='PromotionCatchUpTest.*' ./build-n08/mooncake-store/tests/hot_standby_service_test
./mooncake-store/tests/e2e/run_oplog_batch_cluster_test.sh
./mooncake-store/tests/e2e/run_oplog_snapshot_smoke.sh --build-dir "$PWD/build-n08" --run-dir /tmp/n08-snapshot-20260909-b --timeout-sec 60
./mooncake-store/tests/e2e/run_oplog_snapshot_smoke.sh --build-dir "$PWD/build-n08" --run-dir /tmp/n08-snapshot-20260909-d --timeout-sec 60 --failpoint-dir /tmp/n08-snapshot-20260909-d/failpoints

Results: both real-etcd runs PASS (failpoint environment disabled/enabled; no crash point armed). The second run includes the new real-etcd controller repeated Start/Stop/promotion/destructor test: 1/1. Focused bootstrap/controller tests: 9/9 before that optional test was added; final catch-up: 8/8. Launcher tests: PASS. Earlier focused results: configuration 7/7, metadata 8/8, promotion 7/7.

Both E2E runs restored snapshot seq 18 and replayed to seq 23 before promotion. Data audit: 19 successful reads, zero mismatches, one removed object absent. Test clusters were stopped and logs retained under the run directories above. The existing inspector still warns about unrecognized producer-view/snapshot control keys; this smoke separately checks snapshot pointers/artifacts and restored cursors.

• Unit tests pass (focused targets above)
• Integration tests pass (real-etcd primary path; full fault matrix deferred)
• Manual testing done (local E2E runs above; not registered in CI/nightly)

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• AI tools were used (specify below)

Codex assisted with implementation, focused tests, and this draft description. Human line-by-line review and end-to-end sign-off are pending; the human submitter remains responsible for understanding and defending all changes.

[Aionw]

LGTM