[MoonEP] Kimi-K3 production integration roadmap

[wirybeaver]

Motivation

Track MoonEP integration for Kimi-K3 as a small set of independently reviewable changes under the broader Kimi-K3 roadmap in #32607.

MoonEP uses static-shape dispatch/combine plus dynamic redundant experts and remote expert prefetch to balance expert-parallel traffic. The first completion boundary is a reproducible Kimi-K3 MXFP4 configuration; broader runtime and model capabilities can follow independently.

Related roadmap: vllm-project/vllm#52095.

Current implementation

• BF16/eager integration seam and correctness path — [MoonEP] BF16 PoC integration for Kimi-K3 #33249
  Distinct moonep backend, dispatch/combine contracts, static buffer facade, BF16 expert layout and runner, FusedMoE wiring, and distributed validation harness.
• Synthetic BF16 correctness validation
  8x H100 SXM; all eight ranks passed with zero numerical error.
• Kimi-K3 MXFP4 symmetric-memory + DeepGEMM path — [MoonEP] MXFP4 experts for Kimi K3 on the DeepGEMM runner in symmetric memory #34874
  Author-validated end to end on 4x GB200, TP16/EP16, but still needs rebase, CI, and upstream dependency pinning.

The merged BF16 path remains intentionally diagnostic: eager-only, replicated/global BF16 weights, and a correctness-first expert loop.

Roadmap

Productionization

• Land and pin the compatible MoonEP dependency
  Merge or supersede update prefetch api and support rotating local to first MoonshotAI/MoonEP#32, then provide an exact commit, wheel, or container build. The integration must not depend on an undocumented development checkout.
• Rebase and land [MoonEP] MXFP4 experts for Kimi K3 on the DeepGEMM runner in symmetric memory #34874
  Rebase onto the latest moonep, isolate the MXFP4/DeepGEMM/symmetric-memory delta, fix pre-commit, and run the required SGLang CPU/GPU CI.
• Close correctness and support-contract gaps @charliechenye
  Remove silent rank-0 fallback; keep padded routes out of planner statistics; reject unsupported bias; and fail fast for unsupported overlap/CUDA-graph combinations. [MoonEP] Fix BF16 physical row semantics and unsupported config checks #36133
• Add focused automated coverage
  Cover symmetric-pool row mapping and lifecycle, MXFP4 weight/scale residency, remote-prefetch remapping, MoonEP-to-DeepGEMM pre/post-permute, empty/padded groups, route weights, and capacity boundaries.
• Add a real Kimi-K3 integration test
  Instantiate the actual FusedMoE/MXFP4 loading path without import stubs and exercise dispatch → prefetch → DeepGEMM → combine. Keep the BF16 harness as a component test.
• Run reproducible acceptance validation
  Record exact SGLang/MoonEP/DeepGEMM revisions, image and launch command, topology, TP/EP, memory, correctness, prefill/decode coverage, capacity failures, and stability.
• Benchmark against DeepEP
  Compare identical Kimi-K3 MXFP4 configurations across decode/small-batch and larger-batch workloads. Report TTFT, ITL, throughput, memory, and dispatch/planning/prefetch/GEMM/combine breakdown against a pre-agreed success threshold.
• Package and document the supported configuration
  Provide installation, hardware/fabric prerequisite checks, launch recipe, support matrix, and troubleshooting guidance.

Follow-up capabilities

Each capability should be independently designed, benchmarked, and reviewed. They should not block the first narrow Kimi-K3 production configuration unless explicitly moved into its acceptance contract.

• Async communication and scheduler overlap
  Use MoonEP's communication stream and async_finish=True event chain across dispatch, prefetch, compute, and combine. Define buffer/plan ownership for multiple in-flight batches, then integrate SBO/TBO rather than inheriting DeepEP overlap semantics accidentally.

  • [MoonEP] Add single-flight async event chaining #35935

• CUDA graph support and static-capacity classes
  Capture stable prefill/decode capacity buckets, eliminate capture-time allocations and host synchronization, and guarantee rank-consistent bucket selection. Capacity policy should minimize padding without permitting collective disagreement.

• Zero-copy dispatch and combine
  Integrate MoonEP's zero-copy controls with SGLang-owned static buffers so dispatch/combine can avoid avoidable allocation and copies. Coordinate this with CUDA graphs and output-buffer ownership. — [MoonEP] Add zero-copy eager expert output path #35934

• Quantized communication and additional expert formats
  MXFP4 expert weights are the first target, but dispatched activations are still BF16. Scope FP8/other communication formats and additional expert-weight formats such as FP8 or NVFP4, including scale layout and remote-prefetch contracts.

• Tuned expert compute and kernel fusion
  Tune the DeepGEMM contiguous grouped path over MoonEP cu_seqlens; evaluate Hopper and Blackwell separately. Consider fusing activation quantization, route-weight application, skipped-row zeroing, and other adapter work only when profiling identifies a meaningful boundary.

• EPLB and expert-placement interaction
  MoonEP's dynamic redundant experts and SGLang EPLB both alter where expert work executes. Define compose-or-exclude semantics, placement-versioning, and how plans/prefetch row mappings react to expert relocation.

• General EP groups and topology support
  Remove Kimi-K3's current EP=TP/TP-group assumptions where practical. Cover independent EP groups, multi-node NVLink fabric/IMEX domains, mixed DP+EP, and topology-aware capability probing.

• Serving-mode compatibility
  Validate pipeline parallelism, PD disaggregation, speculative decoding/MTP, overlap scheduling, and memory-saver/offload behavior. Each combination needs an explicit support contract rather than inheriting generic MoE behavior.

• Broader model semantics
  Support or explicitly reject fused/shared experts, expert bias, alternative gated activations, different gate/up layouts, router top-k values, and non-Kimi expert shapes. Promote MoonEP from a Kimi-K3 path to a generic backend only after these semantics are separated from model-specific assumptions.

• Resource lifecycle, observability, and autotuning
  Integrate buffer and symmetric-pool teardown with model reload and runtime reset. Expose capacity utilization, padding, duplicated experts, prefetch volume/bandwidth, and per-stage timing; use those signals to tune capacity buckets, prefetch slots, and communication SMs.

Initial completion boundary

The Kimi-K3 productionization item is complete when:

• the compatible MoonEP dependency and [MoonEP] MXFP4 experts for Kimi K3 on the DeepGEMM runner in symmetric memory #34874-equivalent integration are merged and reproducibly installable;
• focused tests and required SGLang CI pass;
• unsupported configurations fail early;
• Kimi-K3 validation is reproducible from recorded revisions and commands; and
• MoonEP is compared with DeepEP against an agreed performance threshold.

Follow-up capabilities remain open roadmap items and may graduate into their own tracking issues as designs and owners become clear.

[musclemuller]

Thanks! Since the correctness/support-contract fixes are already being worked on, I would like to take item 2: resource lifecycle and basic observability.

Proposed scope:

• Connect MoonEP communication-buffer cleanup to runtime reset.
• Add symmetric weight-pool teardown alongside the [MoonEP] MXFP4 experts for Kimi K3 on the DeepGEMM runner in symmetric memory #34874 integration.
• Add basic logging for capacity utilization, padding ratio, duplicated experts, and prefetch volume.
• Keep autotuning out of scope.

Is this scope still unclaimed? Since the symmetric pool and some of the relevant statistics currently live in #34874, would you prefer an initial PR against the moonep branch for communication-buffer lifecycle followed by a stacked follow-up for the pool and observability, or should all of the work be stacked on #34874?

[musclemuller]

I opened draft PR #35834 for the first narrow slice: deterministic MoonEP communication-buffer cleanup during runtime reset. It targets the moonep branch and intentionally leaves the #34874 symmetric pool and observability for follow-ups. Could you confirm whether you would like those follow-ups stacked on #34874?