[Store] Add WeightRevision metadata registry (RFC #4017 )

[huangxiaobai-ydmy]

Description

Partial implementation of RFC #4017 (revision-level Weight Management), slice PR1.

Adds an in-memory WeightMetadataStore and Master/client/RPC APIs so a weight revision can be:

1. registered via BeginWeightImport (IMPORTING, invisible to Get/List)
2. published via CommitWeightImport as READY + HOT after payload/manifest keys exist
3. discovered via GetWeightMetadata / ListWeightRevisions
4. policy-updated with generation fencing via UpdateWeightPolicy

Related: #4017

Out of scope for this PR

Python weight_* API convergence, HA OpLog/snapshot, residency migration, and revision leases.

Follow-up plan (3+1)

PR2 - safe read path + protect managed groups

1. Revision lease for weight_get (TTL; blocks delete / release of existing readable replicas while held; allows creating new replicas)
2. Master + client wiring for lease acquire/release on the get path
3. Managed-group guard so generic BatchEvict / single-key Remove / cleanup skip members of a published weight group (with a skip metric)
4. Tests: get-with-lease vs remove returns BUSY; eviction cannot partially dismantle a READY revision

PR3 - residency migration

1. Explicit weight_migrate (HOT / COLD / MIXED) with operation records + batched progress
2. AUTO reconciliation under memory pressure (bounded per round, cooldown)

PR4 (optional / later) - HA + API convergence

1. Persist weight metadata / leases / operations via OpLog + master snapshot
2. weight_management_oplog_capability_confirmed rolling-upgrade gate
3. Converge public API to the RFC weight_* surface (remove unmanaged entry points last)

Happy to adjust these boundaries based on maintainer feedback.

Module

• Mooncake Store (mooncake-store)
• Transfer Engine (mooncake-transfer-engine)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

# Store unit tests added:
#   weight_metadata_store_test
#   master_service_weight_import_test
ctest --test-dir build -R 'weight_metadata_store_test|master_service_weight_import_test' --output-on-failure

Test results:

• Unit tests added (registry + MasterService mount/put/begin/commit/get/list/update)
• Integration tests pass (if applicable) - not run in this environment (Windows host without full native Store build)
• Manual testing done (describe below)

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue ([RFC]: Managing Model Weights as First-Class Citizens in Mooncake Store #4017)

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Cursor / Composer assisted with implementing the WeightMetadataStore, Master RPC wiring, and tests against RFC #4017. The submitter reviewed the changes.

[Aionw]

Will there be an API to revoke or delete? If we encounter an error when transferring model weights, we may need to delete it to leave space for another model weight transfer.

Correct me if I missed some context or if this failure pattern is handled by other mechanisms.

[huangxiaobai-ydmy]

Thanks @Aionw — good catch. That failure path was missing in PR1.

I added:

1. AbortWeightImport — revoke an IMPORTING revision and optionally delete partial keys via keys_to_remove (for mid-transfer failures).
2. RemoveWeightRevision — delete a published (READY/DEGRADED) or still-importing revision, removing stored payload_keys + manifest_key.

CommitWeightImport now also persists payload_keys so remove can reclaim them later.

Happy to adjust naming / semantics if you prefer folding abort into a single weight_remove-style API.