[Store] Persist and recover DFS allocator metadata

[fcczzz]

Description

Implements the standalone DFS allocator recovery proposed in #3819.

Previously, DfsGlobalAllocator lost allocation ownership on Master restart, allowing ranges referenced by restored metadata to be reused. This change persists allocator state in per-shard checkpoints and WALs, then reconciles recovered allocations with standalone Master snapshots before serving requests.

• Persist ALLOC before exposing a descriptor and RELEASE before reusing a deferred extent.
• Recover allocator state from checksummed checkpoints and WAL records, with configurable compaction.
• Keep the allocator sealed during snapshot reconciliation. Retain matching DFS replicas, prune stale descriptors, and quarantine orphan allocations for a fresh deferred-free interval.
• Reject incomplete, corrupt, or incompatible metadata.
• Preserve online shard expansion and recover both existing and newly added shard allocations after restart.

Two details adapt the RFC to the current implementation:

• Compatibility validation uses stable per-shard identity rather than the total shard count, so online expansion does not invalidate existing checkpoints.
• The WAL compaction setting is named MOONCAKE_DFS_METADATA_WAL_COMPACTION_THRESHOLD_BYTES.

Compatibility and scope

The DFS descriptor format and client data path remain unchanged. Existing roots containing only .data files cannot be migrated in place; upgrade and rollback require a new empty DFS root.

This supports standalone Master recovery. HA, OpLog recovery, standby promotion, and multi-tenant DFS remain unsupported. Allocator metadata persistence does not add data-file fsync durability.

Module

• Mooncake Store (mooncake-store)
• Docs

Type of Change

• New feature
• Breaking change
• Documentation update

How Has This Been Tested?

Test commands:

cmake --build build --target \
  dfs_posix_test \
  master_snapshot_codec_test \
  distributed_storage_config_test \
  master_service_dfs_scenario_test \
  -j"$(nproc)"

ctest --test-dir build --output-on-failure \
  -R '^(dfs_posix_test|master_snapshot_codec_test|distributed_storage_config_test|master_service_dfs_scenario_test)$'

All four test targets passed, including 55 DFS tests. Coverage includes allocation recovery, durable release, torn WAL tails, corrupt checkpoints, namespace mismatches, orphan reconciliation, and recovery after online expansion.

Test results:

• Unit tests pass
• Master service scenario tests pass
• Manual deployment testing done

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue ([RFC]: [Store] Persist and Recover DFS Allocator Metadata #3819)

AI Assistance Disclosure

• AI tools were used