[MoonEP] Harden BF16 reference adapter contracts and validation

[charliechenye]

• fix(moonep): enforce BF16 reference weight contracts
• fix(moonep): use physical VM group rows for reference compute
• fix(moonep): make communication resources lifecycle-safe
• fix(moonep): reject unsupported runtime mutation and overlap

This stacks directly on sg-moonep-upstream-draft-20260802 and keeps the scope intentionally narrow: harden the current BF16/unquantized MoonEP reference adapter before moving on to Kimi-K3 SiTU, MXFP4, optimized expert compute, or model-level A2A integration.

What this fixes

1. Enforce the BF16 reference weight contract

• Centralize the current MoonEP reference-path configuration checks.
• Reject quantized weights, non-BF16 params, fused shared experts, expert bias, and non-SiLU activation for this reference path.
• Require canonical [Gate, Up] expert storage and reject transformed Triton / FlashInfer TRTLLM layouts and CUTLASS [Up, Gate] loading order.
• Reject heterogeneous/KTEP expert ownership that is incompatible with the current replicated global expert layout.
• Add a real FusedMoE.weight_loader() regression with distinguishable Gate/Up checkpoint values.

2. Use MoonEP physical VM-group rows for reference compute

The original PoC mapped prefetched physical group E+b back through plan.experts_to_copy[b] and computed from source row e.

MoonEP's actual contract is:

physical group g -> physical weight row g

For a prefetched expert, experts_to_copy[b] tells prefetch_weight which source expert to copy into physical slot E+b; it is not a compute-row mapping.

This stack:

• removes the unnecessary expert_ids / rank-remapping layer
• computes directly from the physical VM-group row
• adds a planner-realistic numerical regression
• strengthens dispatch/prefetch/combine contract coverage
• updates distributed validation so slot consumption cannot be masked by replicated source weights

The validation now verifies the prefetched slot, poisons the original source row after prefetch, runs the real physical-row compute + combine path, and compares the final result against an independent logical top-k PyTorch reference.

3. Make MoonEP communication resources lifecycle-safe

• Add a small generic distributed-resource cleanup registry.
• Run registered cleanup before process-group teardown and during runtime-context reset.
• Register MoonEP buffer cleanup only after allocation succeeds.
• Preserve MoonEP buffer ownership if destroy() fails so cleanup can be retried.
• Add focused retry-order and reset lifecycle tests.

4. Reject unsupported runtime mutation and overlap

• Reject online weight updates because the current path caches copied expert layouts.
• Reject EPLB because the replicated global layout has no rebalancing/cache-invalidation ownership contract yet.
• Reject two-batch overlap because current MoonEP buffer/plan ownership is synchronous and single-inflight.
• Scope SGLANG_DEEPEP_BF16_DISPATCH to DeepEP only so DeepEP-specific fallback policy cannot silently divert MoonEP.

Why this is stacked separately

I kept this as four reviewable commits rather than folding all of the hardening directly into the original draft.

The commits separate four concerns:

1. supported weight/storage contract
2. physical dispatch/prefetch/compute correctness
3. distributed resource lifetime
4. unsupported runtime mutation/overlap

Validation status

• The clean four-commit stack preserves the reviewed production and validation tree from the development branch.
• The corrected distributed validation now makes physical prefetch-slot consumption observable.
• Linux CPU tests still need to execute in a proper SGLang environment.
• The previous 8x H100 result predates the physical VM-row correctness fix and should be considered superseded. The corrected H100/NVSwitch validation still needs to be rerun.

This remains a narrow BF16 reference-adapter PoC.

It does not yet claim production Kimi-K3 support. SiTU, K3 model-level _ep_a2a, MXFP4, optimized expert compute, and production symmetric expert residency remain follow-up work.

---

CI States

Latest PR Test (Base): ❌ Run #31779992496
Latest PR Test (Extra): ❌ Run #31779992393