diff --git a/.agents/specs/glm4-moe-lite-router-f32.md b/.agents/specs/glm4-moe-lite-router-f32.md new file mode 100644 index 000000000..80d75d00e --- /dev/null +++ b/.agents/specs/glm4-moe-lite-router-f32.md @@ -0,0 +1,283 @@ +# GLM-4.7-Flash routes through DeepSeek-V2's config-driven router dtype, and upstream's is a class property + +**Row:** `MODEL-TEXT-GLM4-MOE-LITE-ROUTER-F32` — new row, `ACTIVE`. +**Issue:** [#2928](https://github.com/mudler/vllm.cpp/issues/2928). +**Date:** 2026-09-04. **Base:** `c796fea41`. +**Predecessor rows:** `MODEL-TEXT-GLM4-MOE-LITE-GATE-2839` (the gate that +measures the gap, [#2839](https://github.com/mudler/vllm.cpp/issues/2839) / +[#2906](https://github.com/mudler/vllm.cpp/pull/2906)), +`MODEL-TEXT-glm4-moe-lite-glm4-moe-lite-for-causal-lm` (the port). + +## Scope + +One mirror repair on the `Glm4MoeLiteForCausalLM` forward: the MoE router logit +dtype. In scope are the GLM registry's parse, the load path that feeds the +forward, and a checkpoint-free test that measures the difference through the +model forward. + +Out of scope: re-capturing any golden (needs a GPU and a 58.2 GiB snapshot, +neither of which is on this host), the near-tie gap artifacts ([#2929](https://github.com/mudler/vllm.cpp/issues/2929)), +the `routed_scaling_factor` application point ([#2930](https://github.com/mudler/vllm.cpp/issues/2930)), +and this model's speed axis. + +## What is true at `c796fea41` + +**Which revision this was compared against.** Every upstream anchor below was +read in the local checkout at `5559679229`, which is the revision the committed +`tests/parity/goldens/glm4_moe_lite_greedy/` artifacts were captured on and the +PRIOR parity pin. The ACTIVE pin is `e126687a9a`. **One** token gate has now run +at it and passed (`b55f63ed6`, job `7386f034-246a-4af5-9a04-f98aafffce54` on +GB10, 2026-09-04, `IDS mismatched_positions 0 of 96`, `TOKENGATE_VERDICT PASS`), +and that is a different model's battery: no `Glm4MoeLiteForCausalLM` golden has +been re-captured at the active pin ([#2794](https://github.com/mudler/vllm.cpp/issues/2794)). +Re-reading `glm4_moe.py` at the active pin is owed as O5. + +**Upstream, read at `5559679229` — the pin the goldens were captured on.** +`Glm4MoeLiteForCausalLM`'s MoE block is `Glm4MoeLite`, a bare subclass of +`Glm4MoE` (`glm4_moe_lite.py:86-87`, instantiated at `:161-165`). `Glm4MoE`'s +gate is `nn.Linear(hidden_size, n_routed_experts, bias=False, +dtype=torch.float32)` (`glm4_moe.py:141-146`), it is fed +`hidden_states.to(dtype=torch.float32)` (`:218`), and the layer declares +`router_logits_dtype=torch.float32` (`:205`). No config key participates. The +fp32 router is a property of the class. + +`DeepseekV2MoE` resolves the same dtype from the config +(`deepseek_v2.py:308-314` through `_get_moe_router_dtype`, `:123-133`), which +returns fp32 only for `model_type == "glm_moe_dsa"` or an explicit +`moe_router_dtype: "float32"`. + +**Ours.** `glm4_moe_lite_registry.cpp` composes the DeepSeek-V2 forward and +loader, so GLM's router dtype comes from `ParseDeepseekV2Params`, which mirrors +`_get_moe_router_dtype` at `deepseek_v2_weights.cpp:332`, and `deepseek_v2.cpp:363` +sizes the logit buffer from the resolved flag. The published +`zai-org/GLM-4.7-Flash` `config.json` declares no `moe_router_dtype` and its +`model_type` is `glm4_moe_lite`, so the flag resolves **false** and our router +logits are rounded to bf16 before the top-k. + +**Why it bites this model and not the vehicle that gated the block.** +GLM-4.7-Flash routes top-4 of 64 experts, `topk_method: noaux_tc` (sigmoid +scores plus `e_score_correction_bias` for the selection), `norm_topk_prob: true`, +`routed_scaling_factor: 1.8`. bf16 carries an 8-bit mantissa, ~4e-3 relative, and +the rounding lands in front of a **discrete** rank-4 boundary, so the error is +bimodal rather than a tolerance. DeepSeek-V2-Lite — the vehicle the block was +gated on — is top-2 of 4, softmax, greedy, no bias. + +`deepseek_v2.h:164-168` states that "a token gate cannot see this either way". +That holds for a store that is merely too wide. It does not hold for one that is +too narrow in front of a top-k, and this row's test measures the difference. + +## Design + +**A — the GLM registration answers for GLM.** `ParseDeepseekV2Params` keeps +mirroring `_get_moe_router_dtype` and keeps refusing to read `model_type` +(`deepseek_v2_weights.cpp:321-330` argues that, and it is right: that parser +serves `DeepseekV2ForCausalLM`). The GLM registry TU gains one exported +`ParseGlm4MoeLiteParams(config)` that composes it and then sets +`router_dtype_is_f32 = true`, which is what `Glm4MoE` being a different class +from `DeepseekV2MoE` means in a port that composes them. `glm_moe_dsa.cpp:353` +already does exactly this for GLM-5.3. + +**B — every GLM entry point resolves through that one function.** The registry's +`parse_config`, `load_weights` and `make_kv_cache` hooks all call it, so the +params the forward reads and the params the config hook validates cannot drift. + +**C — the difference is measured through the forward, not asserted.** A new +checkpoint-free case builds a GLM-4.7-Flash-shaped synthetic model (`noaux_tc` +sigmoid router with the correction bias, 64 experts, top-4, `norm_topk_prob`, +`routed_scaling_factor` 1.8) and runs the CPU forward under both arms. + +## Tests and expected verdicts + +- **T1** `ParseGlm4MoeLiteParams` on a config shaped like the published + `zai-org/GLM-4.7-Flash` one — no `moe_router_dtype` key — resolves + `router_dtype_is_f32 == true`, and `ParseDeepseekV2Params` on the same bytes + resolves `false`. RED before the change (both false). +- **T2** at GLM's router shape the two arms produce **different** logits through + the model forward, so the dtype is observable rather than merely wider, and the + params the GLM production path resolves select the f32 arm. RED before the + change (the production arm is the bf16 one). +- **T3** at GLM's real router configuration, the bf16 store changes the selected + top-4 SET. This is the MECHANISM measured directly through + `vt::MoeRouterTopK`; it holds independently of the repair and is not a + regression gate. +- **T4** the PRODUCTION path — `ModelRegistry::Load` over a synthetic + GLM-4.7-Flash checkpoint on disk, then `ModelRegistry::Forward` on its default + configuration — produces the f32 arm and not the bf16 one. This is the + reachability case: it is what the loader call site owes. +- **T5** the SACRED engine gate. **NOT RUNNABLE HERE and not claimed.** No + `zai-org/GLM-4.7-Flash` snapshot is on this host or on the NAS. + +## Measured + +Built `-O0`, CPU only (no CUDA toolkit on this host), at `c796fea41`. + +RED, before the one-line repair, with the whole file in place (M1 below): + +``` +[doctest] test cases: 5 | 2 passed | 3 failed | 0 skipped +[doctest] assertions: 23 | 20 passed | 3 failed +``` + +GREEN, after: + +``` +[doctest] test cases: 5 | 5 passed | 0 failed | 0 skipped +[doctest] assertions: 16085 | 16085 passed | 0 failed +``` + +The three numbers the cases print: + +| | measurement | +|---|---| +| T2 | the router dtype is observable in the GLM forward at **6 of 16** fixed seeds; worst \|f32 - bf16\| logit `0.00100262` | +| T3 | the bf16 store changes the top-4 **SET** for **34 of 4096** tokens (0.83%) at 64 experts / top-4 / sigmoid+bias / renormalize / scale 1.8 | +| T4 | checkpoint seed 61 separates the arms; `ModelRegistry::Forward` matches the f32 arm byte for byte | + +T3's 0.83% is per MoE layer per token on `N(0,1)` synthetic logits, so it is the +magnitude of the mechanism and not a prediction for the checkpoint. Taken at face +value over GLM-4.7-Flash's 46 MoE layers it would put at least one changed +selection on roughly a third of token positions, which is the order of the +SACRED gate's 59 divergent positions — but the real logit distribution is not +this one, and nothing here measures the checkpoint. + +### The per-prompt distribution, and what it rules out + +Read from the committed artifacts alone, no checkpoint. The per-prompt mismatch +counts `[2,6,6,12,12,11,10,0]` are almost entirely explained by ONE variable — +where the prompt first diverges — because after that it barely re-converges: + +| prompt | prompt len | first divergent generated index | absolute position | mismatches | `16 - first` | re-converged | +|---|---|---|---|---|---|---| +| 0 | 5 | 14 | 19 | 2 | 2 | 0 | +| 1 | 4 | 6 | 10 | 6 | 10 | 4 | +| 2 | 14 | 6 | 20 | 6 | 10 | 4 | +| 3 | 12 | 4 | 16 | 12 | 12 | 0 | +| 4 | 6 | 4 | 10 | 12 | 12 | 0 | +| 5 | 11 | 4 | 15 | 11 | 12 | 1 | +| 6 | 6 | 6 | 12 | 10 | 10 | 0 | +| 7 | 9 | — | — | 0 | — | — | + +Two things follow, and both bear on what kind of defect this is. + +**It is not positional or structural.** The absolute positions of the first +divergence are 19, 10, 20, 16, 10, 15, 12. They share no page, block or layer +boundary — only prompt 3's 16 lands on one — and they do not track prompt length +(prompt 2 is the longest at 14 and diverges at generated index 6, prompt 1 is the +shortest at 4 and diverges at the same index). A defect keyed to a position, a +cache page or a layer count crossing a boundary would leave a common divisor +here, and there is none. + +**No prompt diverges before generated index 4, and that is not chance.** Under +any per-step flip probability `q` fitted to a first-divergence set of +`{4,4,4,6,6,6,14}` plus one 16-token survivor, all eight prompts surviving their +first four steps has probability of order `1e-3`. The natural reading is the one +a SMALL NUMERICAL perturbation predicts: while the continuation is +high-confidence the logit margin swamps the noise, and once the margin narrows +the noise decides. Prompt 7 — `To be or not to be, that is` — is a memorised +quotation whose margin never narrows, and it is the one prompt that survives all +16 tokens. + +This is consistent with the router-dtype divergence and is evidence AGAINST a +structural bug in the MLA block, the cache or the schedule, which would not wait +for the margin to narrow. It is not proof that the router dtype is the whole gap, +and O1 stands. + +### Mutations + +Each rebuilt (`BUILD_RC=0` every time, so no mutation was caught by the +compiler) and each restored, with the restore verified by `sha256sum -c` against +a baseline taken before the first mutation. + +| # | Mutation | Result | +|---|---|---| +| M1 | remove `p.router_dtype_is_f32 = true;` from `ParseGlm4MoeLiteParams` | RED `5 \| 2 passed \| 3 failed`, `assertions: 23 \| 20 passed \| 3 failed` — T1, T2 and T4 all fire | +| M2 | delete the production loader call site `weights.params = ParseGlm4MoeLiteParams(config);` | RED `5 \| 4 passed \| 1 failed`, 2 assertions — **only T4**, and it reports the registry's output as the bf16 arm. This is the reachability proof: T1, T2 and T3 stay green without the call site | +| M3 | remove the bf16 round trip T3's claim rests on (`logits_bf16 = logits_f32`) | RED — the SET-difference count falls to `0/4096`, so the 34 is caused by the store and by nothing else | +| M4 | make T2's second arm f32 as well | RED — `0/16` seeds differ, `worst 0`, so T2's comparison is not vacuous | + +### Preflight + +`scripts/agent-preflight.sh --staged` exits 1 with two failing gates, and neither +is caused by this change. + +* `test_cpu_x86_llamacpp_floor` — `test_a_contended_leg_is_discarded_and_never_summarised` + expects rc 2 and got 4 while the box was at `load=75.28 76.77 62.18` from other + sessions' builds; the harness discarded the contended leg and ran out of + retries. Re-run standalone on this same tree: `Ran 10 tests ... OK`. +* `tools suites` — `test_oracle_pin.test_metadata_and_runtime_strings_differ_on_the_pin` + asserts `VLLM_DISTRIBUTION_VERSION != VLLM_ORACLE_VERSION`, and both now read + `0.28.1rc1.dev132+ge126687a9`. Those constants come from `.agents/upstream-sync.md` + through `tools/bench/serve_low_common.py:107-108`, and this change touches + neither, so the failure is a property of `origin/main`. Already open and owned + as [#2931](https://github.com/mudler/vllm.cpp/issues/2931). + +Five gates SKIP for want of a cross-compiler, a CUDA toolkit or a CI-only input. +`check-pr-size.py` is one of them and was run by hand: `OK: every explicit path +class is within its review budget.` + +### Regression + +Run on the repaired tree; `DeepseekV2ForCausalLM` is byte-identical because +`ParseDeepseekV2Params` is untouched. + +| suite | result | +|---|---| +| `test_glm4_moe_lite_router_dtype` | 5/5, 16085 assertions | +| `test_glm4_moe_lite_load` | 3/3, 15 assertions | +| `test_deepseek_v2_forward` | 12/12, 2061 assertions | +| `test_deepseek_v2_load` | 4/4, 14 assertions | +| `test_ops_moe_router_grouped` | 14/14, 941 assertions | + +## Gates + +```sh +cmake --build build -j 3 --target test_glm4_moe_lite_router_dtype \ + test_glm4_moe_lite_load test_deepseek_v2_forward test_deepseek_v2_load \ + test_ops_moe_router_grouped +./build/tests/test_glm4_moe_lite_router_dtype +./build/tests/test_glm4_moe_lite_load +./build/tests/test_deepseek_v2_forward +./build/tests/test_deepseek_v2_load +./build/tests/test_ops_moe_router_grouped +scripts/agent-preflight.sh +``` + +## Stop conditions + +Stop and report `NEEDS_DECISION` before changing `ParseDeepseekV2Params` to read +`model_type`, and before touching any assertion in +`tests/vllm/models/test_glm4_moe_lite_paged_engine.cpp`. That gate is red because +it measures something; a forward repair cannot and must not make it green from +this host, because what it compares is a frozen `our_ids.npy`. + +## Owed + +- O1. **This repair is not verified against the oracle, and the row does not + claim it is.** The gap it addresses is 59 positions + ([#2839](https://github.com/mudler/vllm.cpp/issues/2839)); what is established + here is that the router dtype is a mirror divergence on the block that feeds + those tokens, and that it is observable in a forward. Whether it is the whole + gap is owed to whoever next holds a GPU and the snapshot, who must re-capture + `our_ids.npy` and re-run the SACRED gate. +- O2. `docs/USAGE.md` still carries no checkpoint row for + `Glm4MoeLiteForCausalLM` — no revision, no sha256 — which `CLAUDE.md` + §"Say which weights, and from where" requires. Inherited from + `MODEL-TEXT-GLM4-MOE-LITE-GATE-2839` O3 and not repaired here, because the + values have to be read off an artifact this host does not have. +- O3. The near-tie gap artifacts are identically zero in 12 of 18 golden + directories while `our_ids` diverges at 13 to 83 positions in each + ([#2929](https://github.com/mudler/vllm.cpp/issues/2929)). Established from the + committed artifacts alone; the re-capture that would settle the cause needs the + checkpoints. +- O5. The upstream anchors are read at `5559679229`, not at the active pin + `e126687a9a`. `Glm4MoE`'s fp32 gate is a class property rather than a config + key, so it is unlikely to have moved, but "unlikely" is not "read". The token + gate that passed at the active pin on 2026-09-04 (`b55f63ed6`) is a different + model's battery and says nothing about this one. Owed to the pin-reconciliation + queue ([#2611](https://github.com/mudler/vllm.cpp/issues/2611)). +- O4. `routed_scaling_factor` is applied to the routing weights rather than the + routed output ([#2930](https://github.com/mudler/vllm.cpp/issues/2930)). It is + a recorded deviation whose only gate vehicle had `routed_scaling_factor: 1.0`, + which made it bit-identical there; GLM's 1.8 makes it live. Moving it changes + `vt::MoeCombine`'s contract for every model on this block, so it needs its own + spec rather than an in-flow fix. diff --git a/include/vllm/model_executor/models/glm4_moe_lite.h b/include/vllm/model_executor/models/glm4_moe_lite.h new file mode 100644 index 000000000..90b9e6731 --- /dev/null +++ b/include/vllm/model_executor/models/glm4_moe_lite.h @@ -0,0 +1,58 @@ +// GLM-4.7-Flash (`Glm4MoeLiteForCausalLM`, HF `model_type: glm4_moe_lite`) — the +// one place this port says what GLM's MoE block is, as opposed to DeepSeek-V2's. +// Row `MODEL-TEXT-GLM4-MOE-LITE-ROUTER-F32`, issue +// [#2928](https://github.com/mudler/vllm.cpp/issues/2928). +// +// ─── WHY THIS MODEL SHARES DeepseekV2Params AND STILL NEEDS ITS OWN PARSE ──── +// Upstream at the parity pin `5559679229bc961848b121ccdeaa8fa5d79bec98`, +// `Glm4MoeLiteAttention` / `Glm4MoeLiteMLAAttention` are literal zero-override +// subclasses of `DeepseekV2Attention` / `DeepseekV2MLAAttention` +// (glm4_moe_lite.py:94-95, :98-99), and the decoder layer, model and +// `load_weights` are structural copies of deepseek_v2. So the LAYOUT is +// DeepSeek-V2's and one weight struct serves both. +// +// The MoE BLOCK is not. `Glm4MoeLite` is `Glm4MoE` (glm4_moe_lite.py:86-87, +// instantiated at :161-165), NOT `DeepseekV2MoE`, and the two resolve their +// router logit dtype differently: +// +// * `Glm4MoE`'s gate is `nn.Linear(hidden_size, n_routed_experts, bias=False, +// dtype=torch.float32)` (glm4_moe.py:141-146), fed +// `hidden_states.to(dtype=torch.float32)` (:218), declared +// `router_logits_dtype=torch.float32` (:205). No config key participates: +// fp32 is a property of the CLASS. +// * `DeepseekV2MoE` reads it from the config — +// `GateLinear(..., out_dtype=_get_moe_router_dtype(config))` +// (deepseek_v2.py:308-314, :123-133) — which is fp32 only for +// `model_type == "glm_moe_dsa"` or an explicit `moe_router_dtype: +// "float32"`. +// +// The published `zai-org/GLM-4.7-Flash` config.json declares no +// `moe_router_dtype`, so composing DeepSeek-V2's parser for GLM resolved the +// wrong dtype and rounded the router logits to bf16 in front of a top-4-of-64 +// `noaux_tc` selection. `ParseDeepseekV2Params` is NOT the place to fix that: it +// serves `DeepseekV2ForCausalLM` and deliberately does not read `model_type` +// (deepseek_v2_weights.cpp:321-330). GLM-5.3 already carries its own answer the +// same way (glm_moe_dsa.cpp:353). +#ifndef VLLM_MODEL_EXECUTOR_MODELS_GLM4_MOE_LITE_H_ +#define VLLM_MODEL_EXECUTOR_MODELS_GLM4_MOE_LITE_H_ + +#include "vllm/model_executor/models/deepseek_v2.h" +#include "vllm/transformers_utils/hf_config.h" + +namespace vllm { + +// Resolve `Glm4MoeLiteForCausalLM` params. Composes `ParseDeepseekV2Params` with +// `allow_mtp_tail = true` (GLM-4.7-Flash ships `num_nextn_predict_layers: 1` and +// the loader never requests the tail — glm4_moe_lite.py:358-360, :633-643) and +// then applies what `Glm4MoeLite` being `Glm4MoE` rather than `DeepseekV2MoE` +// means. Every GLM registry hook — `parse_config`, `load_weights`, +// `make_kv_cache` — resolves through this one function, so the params the +// forward reads and the params the config hook validates cannot drift. +// +// Pure/host; unit-testable without a checkpoint +// (tests/vllm/models/test_glm4_moe_lite_router_dtype.cpp). +DeepseekV2Params ParseGlm4MoeLiteParams(const HfConfig& config); + +} // namespace vllm + +#endif // VLLM_MODEL_EXECUTOR_MODELS_GLM4_MOE_LITE_H_ diff --git a/src/vllm/model_executor/models/glm4_moe_lite_registry.cpp b/src/vllm/model_executor/models/glm4_moe_lite_registry.cpp index 9b14b16f2..6460b2fde 100644 --- a/src/vllm/model_executor/models/glm4_moe_lite_registry.cpp +++ b/src/vllm/model_executor/models/glm4_moe_lite_registry.cpp @@ -47,6 +47,7 @@ #include #include "vllm/model_executor/models/deepseek_v2.h" +#include "vllm/model_executor/models/glm4_moe_lite.h" #include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier #include "vllm/model_executor/models/qwen3_5_common.h" // HostLogits #include "vllm/model_executor/models/qwen3_5_internal.h" // detail::DeviceTokenIdsScope @@ -56,6 +57,25 @@ #include "vt/dtype.h" namespace vllm { + +// The ONE place this port says what GLM's MoE block is. Declared in +// glm4_moe_lite.h, which carries the upstream anchors and the reason +// `ParseDeepseekV2Params` is not the place for this. +DeepseekV2Params ParseGlm4MoeLiteParams(const HfConfig& config) { + DeepseekV2Params p = ParseDeepseekV2Params(config, /*allow_mtp_tail=*/true); + // `Glm4MoeLite` IS `Glm4MoE` (glm4_moe_lite.py:86-87), whose gate is + // `nn.Linear(..., dtype=torch.float32)` fed `hidden_states.to(torch.float32)` + // and declared `router_logits_dtype=torch.float32` (glm4_moe.py:141-146, :218, + // :205). fp32 is a property of the class, so no config key can withhold it — + // and the published GLM-4.7-Flash config.json ships no `moe_router_dtype`, + // which is exactly the value `ParseDeepseekV2Params` correctly reads for the + // architecture IT serves. Rounding the logits to bf16 in front of a + // top-4-of-64 `noaux_tc` selection changes which experts run + // (tests/vllm/models/test_glm4_moe_lite_router_dtype.cpp measures it). + p.router_dtype_is_f32 = true; + return p; +} + namespace { // registry.py _ModelInfo for GLM-4.7-Flash: text generation, NOT hybrid (MLA is @@ -98,9 +118,14 @@ std::unique_ptr LoadGlm4MoeLiteForCausalLM( } // allow_mtp_tail = true: GLM-4.7-Flash ships `num_nextn_predict_layers: 1`; the // loader pulls only the main layers, so the MTP tail is skipped, not refused. - return std::make_unique( - registration, LoadDeepseekV2ForCausalLMWeights(*source.safetensors, config, - /*allow_mtp_tail=*/true)); + DeepseekV2Weights weights = LoadDeepseekV2ForCausalLMWeights( + *source.safetensors, config, /*allow_mtp_tail=*/true); + // The loader resolves `w.params` through `ParseDeepseekV2Params`, which answers + // for `DeepseekV2MoE`. GLM-4.7-Flash's MoE block is `Glm4MoE`, so the params + // the FORWARD reads must be GLM's. Same bytes, same function, one extra step. + weights.params = ParseGlm4MoeLiteParams(config); + return std::make_unique(registration, + std::move(weights)); } void PrepareGlm4MoeLiteForCausalLM(LoadedModel& model, const HfConfig& config, @@ -158,7 +183,7 @@ ForwardLogits ForwardGlm4MoeLiteForCausalLM(LoadedModel& model, // GLM-4.7-Flash's `num_nextn_predict_layers: 1` is accepted (tail skipped); every // other unsupported field still throws with a precise message. void ParseGlm4MoeLiteConfig(const HfConfig& config) { - (void)ParseDeepseekV2Params(config, /*allow_mtp_tail=*/true); + (void)ParseGlm4MoeLiteParams(config); } // MLA KV-cache spec: exactly ONE `MLAAttentionSpec` group, 1 head, @@ -166,7 +191,7 @@ void ParseGlm4MoeLiteConfig(const HfConfig& config) { // DeepSeek-V2) wide, NO factor-2 and NO separate V. v1::KVCacheConfig MakeGlm4MoeLiteKVCache(const HfConfig& config, int block_size, int num_blocks) { - const DeepseekV2Params p = ParseDeepseekV2Params(config, /*allow_mtp_tail=*/true); + const DeepseekV2Params p = ParseGlm4MoeLiteParams(config); const int head_size = static_cast(p.mla.head_size()); v1::KVCacheConfig kv; diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d2df8cf4d..89e33734f 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1445,6 +1445,10 @@ target_include_directories(test_deepseek_v2_paged_engine PRIVATE # query branch AND the noaux_tc grouped router. Loader gate (host, MTP-tail skip) # + the SACRED paged-engine greedy gate (checkpoint-gated, dgx-only). vllm_cpp_add_test(test_glm4_moe_lite_load vllm/models/test_glm4_moe_lite_load.cpp) +# The MoE ROUTER DTYPE gate (#2928): GLM's fp32 router is a property of +# `Glm4MoE`, not a config key, and the difference is measured through the CPU +# forward at GLM's top-4-of-64 `noaux_tc` shape. Host-only, no checkpoint. +vllm_cpp_add_test(test_glm4_moe_lite_router_dtype vllm/models/test_glm4_moe_lite_router_dtype.cpp) vllm_cpp_add_test(test_glm4_moe_lite_paged_engine vllm/models/test_glm4_moe_lite_paged_engine.cpp) target_compile_definitions(test_glm4_moe_lite_paged_engine PRIVATE PARITY_GOLDENS_DIR="${CMAKE_SOURCE_DIR}/tests/parity/goldens") diff --git a/tests/vllm/models/test_glm4_moe_lite_router_dtype.cpp b/tests/vllm/models/test_glm4_moe_lite_router_dtype.cpp new file mode 100644 index 000000000..7e0b0a535 --- /dev/null +++ b/tests/vllm/models/test_glm4_moe_lite_router_dtype.cpp @@ -0,0 +1,696 @@ +// GLM-4.7-Flash (`Glm4MoeLiteForCausalLM`) MoE ROUTER DTYPE gate — issue #2928, +// row MODEL-TEXT-GLM4-MOE-LITE-ROUTER-F32. +// +// Upstream `Glm4MoeLiteForCausalLM` does NOT use `DeepseekV2MoE`. Its MoE block +// is `Glm4MoeLite`, a bare subclass of `Glm4MoE` (glm4_moe_lite.py:86-87, +// instantiated at :161-165), whose gate is +// `nn.Linear(hidden_size, n_routed_experts, bias=False, dtype=torch.float32)` +// (glm4_moe.py:141-146) fed `hidden_states.to(dtype=torch.float32)` (:218), with +// `router_logits_dtype=torch.float32` (:205). No config key participates: the +// fp32 router is a property of the CLASS. Read at 5559679229, the pin the +// `tests/parity/goldens/glm4_moe_lite_greedy/` artifacts were captured on. +// +// `DeepseekV2MoE` resolves the same dtype from the config instead +// (deepseek_v2.py:308-314 through `_get_moe_router_dtype`, :123-133 — fp32 only +// for `model_type == "glm_moe_dsa"` or an explicit `moe_router_dtype: +// "float32"`). Our port composes the DeepSeek-V2 block for GLM, and the +// published `zai-org/GLM-4.7-Flash` config.json declares NO `moe_router_dtype`, +// so before this row our GLM router logits were rounded to bf16 in front of the +// top-k. +// +// WHY THAT IS NOT THE "merely too wide" HAZARD deepseek_v2.h:164-168 describes. +// GLM routes top-4 of 64 with `topk_method: noaux_tc` — sigmoid scores plus +// `e_score_correction_bias` decide the SELECTION — and `norm_topk_prob: true`. +// bf16 carries an 8-bit mantissa, so the rounding lands in front of a DISCRETE +// rank-4 boundary and the resulting error is bimodal, not a tolerance. The +// vehicle the block was gated on (DeepSeek-V2-Lite) is top-2 of 4, softmax, +// greedy, no bias, which is why `test_deepseek_v2_forward.cpp`'s tripwire case +// measures the two arms as bit-identical there. This file measures them at GLM's +// shape instead of arguing about it. +// +// Host-only: a CPU forward over a tiny synthetic model. No checkpoint, no GPU. +// The SACRED engine gate for this architecture is +// tests/vllm/models/test_glm4_moe_lite_paged_engine.cpp and needs a 58.2 GiB +// snapshot that is on neither this host nor the NAS. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/deepseek_v2.h" +#include "vllm/model_executor/models/glm4_moe_lite.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/ops.h" +#include "vt/tensor.h" + +namespace fs = std::filesystem; + +namespace { + +using vllm::DeepseekV2DenseMlp; +using vllm::DeepseekV2LayerWeights; +using vllm::DeepseekV2Model; +using vllm::DeepseekV2Params; +using vllm::DeepseekV2Weights; +using vllm::HfConfig; +using vllm::OwnedTensor; +using vllm::PagedKvCache; +using vllm::v1::CommonAttentionMetadata; +using vt::DType; + +vt::Queue Q() { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } + +std::string ScratchDir() { + const char* env = std::getenv("TMPDIR"); + const fs::path base = env != nullptr ? fs::path(env) : fs::temp_directory_path(); + const fs::path dir = base / "vllm_cpp_glm4_moe_lite_router_dtype_test"; + std::error_code ec; + fs::create_directories(dir, ec); + return dir.string(); +} + +// A tiny model carrying GLM-4.7-Flash's ROUTER shape verbatim — 64 routed +// experts, top-4, one shared expert, `topk_method: noaux_tc` with NO +// `scoring_func` key (GLM omits it and its class hardcodes sigmoid), +// `norm_topk_prob: true`, `routed_scaling_factor: 1.8`, `n_group`/`topk_group` 1, +// `first_k_dense_replace: 1`, `num_nextn_predict_layers: 1` — and NO +// `moe_router_dtype` key, exactly as the published config.json does not declare +// one. Everything else is shrunk so the CPU forward is fast; the router shape is +// what this file is about. +std::string WriteGlmTinyConfig(const std::string& name, int num_hidden_layers) { + const std::string path = ScratchDir() + "/" + name + ".json"; + std::ofstream f(path); + f << R"({ + "architectures": ["Glm4MoeLiteForCausalLM"], + "model_type": "glm4_moe_lite", + "hidden_size": 64, + "num_hidden_layers": )" + << num_hidden_layers << R"(, + "num_attention_heads": 4, + "num_key_value_heads": 4, + "vocab_size": 100, + "intermediate_size": 16, + "moe_intermediate_size": 16, + "n_routed_experts": 64, + "num_experts_per_tok": 4, + "n_group": 1, + "topk_group": 1, + "norm_topk_prob": true, + "topk_method": "noaux_tc", + "routed_scaling_factor": 1.8, + "moe_layer_freq": 1, + "n_shared_experts": 1, + "first_k_dense_replace": 1, + "num_nextn_predict_layers": 1, + "q_lora_rank": 768, + "qk_nope_head_dim": 16, + "qk_rope_head_dim": 8, + "v_head_dim": 16, + "kv_lora_rank": 24, + "rms_norm_eps": 1e-05, + "rope_theta": 1000000, + "max_position_embeddings": 128, + "tie_word_embeddings": false, + "torch_dtype": "bfloat16" +})"; + f.close(); + return path; +} + +OwnedTensor MakeBf16(const std::vector& shape, bool nk, uint32_t seed, + float scale = 0.08f) { + OwnedTensor o; + o.dtype = DType::kBF16; + o.nk = nk; + o.rank = static_cast(shape.size()); + int64_t numel = 1; + for (int i = 0; i < o.rank; ++i) { + o.shape[i] = shape[static_cast(i)]; + numel *= shape[static_cast(i)]; + } + o.bytes.resize(static_cast(numel) * sizeof(uint16_t)); + auto* p = reinterpret_cast(o.bytes.data()); + std::mt19937 rng(seed); + std::uniform_real_distribution dist(-scale, scale); + for (int64_t i = 0; i < numel; ++i) p[i] = vt::F32ToBF16(dist(rng)); + return o; +} + +OwnedTensor MakeF32(const std::vector& shape, uint32_t seed, float scale) { + OwnedTensor o; + o.dtype = DType::kF32; + o.nk = false; + o.rank = static_cast(shape.size()); + int64_t numel = 1; + for (int i = 0; i < o.rank; ++i) { + o.shape[i] = shape[static_cast(i)]; + numel *= shape[static_cast(i)]; + } + o.bytes.resize(static_cast(numel) * sizeof(float)); + auto* p = reinterpret_cast(o.bytes.data()); + std::mt19937 rng(seed); + std::uniform_real_distribution dist(-scale, scale); + for (int64_t i = 0; i < numel; ++i) p[i] = dist(rng); + return o; +} + +DeepseekV2DenseMlp MakeMlp(int64_t H, int64_t I, uint32_t seed) { + DeepseekV2DenseMlp m; + m.gate_up_proj = MakeBf16({2 * I, H}, /*nk=*/true, seed); + m.down_proj = MakeBf16({H, I}, /*nk=*/true, seed + 1); + return m; +} + +// GLM's q_lora branch (`q_lora_rank: 768`, shrunk here): `fused_qkv_a_proj` + +// `q_a_layernorm` + `q_b_proj`, the branch DeepSeek-V2-Lite never takes. +vllm::DeepseekV2MlaWeights MakeMla(const DeepseekV2Params& p, uint32_t seed) { + const vllm::mla::MlaBlockDims& d = p.mla; + const int64_t H = d.hidden_size, N = d.num_heads, L = d.kv_lora_rank; + const int64_t P = d.qk_nope_head_dim, R = d.qk_rope_head_dim, V = d.v_head_dim; + const int64_t Dqk = d.qk_head_dim(); + const int64_t QL = d.q_lora_rank; + vllm::DeepseekV2MlaWeights w; + w.fused_qkv_a_proj = MakeBf16({QL + L + R, H}, /*nk=*/true, seed); + w.q_a_layernorm = MakeBf16({QL}, false, seed + 1, 0.5f); + w.q_b_proj = MakeBf16({N * Dqk, QL}, /*nk=*/true, seed + 2); + w.kv_a_layernorm = MakeBf16({L}, false, seed + 3, 0.5f); + w.kv_b_proj = MakeBf16({N * (P + V), L}, /*nk=*/true, seed + 4); + w.o_proj = MakeBf16({H, N * V}, /*nk=*/true, seed + 5); + const vllm::mla::AbsorbedKvBProj a = vllm::mla::AbsorbKvBProjBf16( + reinterpret_cast(w.kv_b_proj.bytes.data()), d); + w.w_uk_t = MakeBf16({N, P, L}, false, 1, 0.0f); + std::memcpy(w.w_uk_t.bytes.data(), a.w_uk_t.data(), + a.w_uk_t.size() * sizeof(uint16_t)); + w.w_uv = MakeBf16({N, L, V}, false, 1, 0.0f); + std::memcpy(w.w_uv.bytes.data(), a.w_uv.data(), a.w_uv.size() * sizeof(uint16_t)); + return w; +} + +DeepseekV2Weights GlmTinyWeights(const DeepseekV2Params& p, uint32_t seed0) { + const int64_t H = p.hidden_size, V = p.vocab_size; + const int64_t E = p.n_routed_experts, I = p.moe_intermediate_size; + DeepseekV2Weights w; + w.params = p; + w.embed_tokens = MakeBf16({V, H}, false, seed0 + 1); + w.final_norm = MakeBf16({H}, false, seed0 + 2, 0.5f); + w.lm_head = MakeBf16({H, V}, false, seed0 + 3); + { + const int64_t rows = p.max_position_embeddings, rot = p.mla.qk_rope_head_dim; + const std::vector cache = + vllm::mla::BuildDeepseekRopeCosSinCache(p.rope, rows); + w.rope_cos_sin_cache = MakeBf16({rows, rot}, false, 1, 0.0f); + auto* dst = reinterpret_cast(w.rope_cos_sin_cache.bytes.data()); + for (size_t i = 0; i < cache.size(); ++i) dst[i] = vt::F32ToBF16(cache[i]); + } + uint32_t seed = seed0 + 100; + for (int64_t l = 0; l < p.num_hidden_layers; ++l) { + DeepseekV2LayerWeights lw; + lw.input_layernorm = MakeBf16({H}, false, seed++, 0.5f); + lw.post_attention_layernorm = MakeBf16({H}, false, seed++, 0.5f); + lw.attn = MakeMla(p, seed); + seed += 10; + lw.is_moe = p.is_moe_layer(l); + if (lw.is_moe) { + lw.moe.router_gate = MakeBf16({H, E}, false, seed++); + // `topk_method: noaux_tc` — the f32 learned selection bias + // (deepseek_v2.py:316-318). GLM always carries one. + lw.moe.e_score_correction_bias = MakeF32({E}, seed++, 0.05f); + for (int64_t e = 0; e < E; ++e) { + lw.moe.expert_gate.push_back(MakeBf16({H, I}, false, seed)); + lw.moe.expert_up.push_back(MakeBf16({H, I}, false, seed + 1)); + lw.moe.expert_down.push_back(MakeBf16({I, H}, false, seed + 2)); + seed += 3; + } + lw.moe.shared = MakeMlp(H, p.shared_intermediate_size(), seed); + seed += 2; + } else { + lw.dense = MakeMlp(H, p.intermediate_size, seed); + seed += 2; + } + w.layers.push_back(std::move(lw)); + } + return w; +} + +struct MlaCachePool { + std::vector> buf; + std::vector attn_kv; + MlaCachePool(const DeepseekV2Params& p, int64_t num_blocks, int64_t block_size) { + const int64_t head_size = p.mla.head_size(); + for (int64_t l = 0; l < p.num_hidden_layers; ++l) + buf.emplace_back(static_cast(num_blocks * block_size * head_size), 0); + for (auto& b : buf) { + PagedKvCache kv; + kv.data = b.data(); + kv.dtype = DType::kBF16; + kv.num_blocks = num_blocks; + kv.block_size = block_size; + kv.num_kv_heads = 1; + kv.head_size = head_size; + attn_kv.push_back(kv); + } + } +}; + +CommonAttentionMetadata PrefillMeta(int64_t T, int64_t block_size) { + CommonAttentionMetadata m; + m.num_reqs = 1; + m.num_actual_tokens = static_cast(T); + m.query_start_loc = {0, static_cast(T)}; + m.query_start_loc_cpu = m.query_start_loc; + m.seq_lens = {static_cast(T)}; + m.seq_lens_cpu = m.seq_lens; + m.max_query_len = static_cast(T); + m.max_seq_len = static_cast(T); + m.block_table_num_cols = 1; + m.block_table_tensor = {0}; + for (int64_t t = 0; t < T; ++t) m.slot_mapping.push_back(t % block_size); + m.causal = true; + return m; +} + +std::vector RunTiny(const DeepseekV2Weights& w) { + const int64_t T = 5, bs = 8; + MlaCachePool pool(w.params, /*num_blocks=*/2, bs); + const CommonAttentionMetadata am = PrefillMeta(T, bs); + const std::vector tokens = {3, 17, 42, 8, 61}; + const std::vector positions = {0, 1, 2, 3, 4}; + vt::Queue q = Q(); + return DeepseekV2Model::Forward(tokens, positions, am, pool.attn_kv, w, q); +} + +} // namespace + +// ════════════════════════════════════════════════════════════════════════════ +// T1 — the PARSE. GLM's router dtype is a class property upstream, so it cannot +// come from a config key the checkpoint does not ship. +// ════════════════════════════════════════════════════════════════════════════ +TEST_CASE("glm4-moe-lite router: the GLM parse resolves f32 where the DeepSeek parse does not") { + const HfConfig cfg = vllm::LoadHfConfig(WriteGlmTinyConfig("router_parse", 2)); + + // `Glm4MoeLite` IS `Glm4MoE` (glm4_moe_lite.py:86-87), whose gate is fp32 with + // no config key in the path (glm4_moe.py:141-146,205,218). + const DeepseekV2Params glm = vllm::ParseGlm4MoeLiteParams(cfg); + CHECK(glm.router_dtype_is_f32); + + // The SAME bytes through the DeepSeek-V2 parser resolve upstream's `None`, + // because `_get_moe_router_dtype` (deepseek_v2.py:123-133) sees no + // `moe_router_dtype` key and `model_type` is not `glm_moe_dsa`. That parser is + // correct as it stands and this row does not change it. + const DeepseekV2Params dsv2 = vllm::ParseDeepseekV2Params(cfg, /*allow_mtp_tail=*/true); + CHECK_FALSE(dsv2.router_dtype_is_f32); + + // Everything else the two resolve must be identical: this row moves ONE field. + CHECK(glm.n_routed_experts == dsv2.n_routed_experts); + CHECK(glm.num_experts_per_tok == dsv2.num_experts_per_tok); + CHECK(glm.norm_topk_prob == dsv2.norm_topk_prob); + CHECK(glm.has_e_score_correction_bias == dsv2.has_e_score_correction_bias); + CHECK(glm.routed_scaling_factor == doctest::Approx(dsv2.routed_scaling_factor)); + CHECK(glm.scoring_func == dsv2.scoring_func); + + // The GLM router shape this file exists for. + CHECK(glm.n_routed_experts == 64); + CHECK(glm.num_experts_per_tok == 4); + CHECK(glm.norm_topk_prob); + CHECK(glm.has_e_score_correction_bias); + CHECK(glm.scoring_func == vt::MoeScoringFunc::kSigmoid); + CHECK(glm.routed_scaling_factor == doctest::Approx(1.8f)); +} + +// ════════════════════════════════════════════════════════════════════════════ +// T2 — the FORWARD. At GLM's router shape the dtype is OBSERVABLE, so the too- +// narrow store is not the invisible-width hazard; and the params GLM's own load +// path resolves select the arm upstream computes. +// ════════════════════════════════════════════════════════════════════════════ +TEST_CASE("glm4-moe-lite router: bf16 logits CHANGE the forward at GLM's top-4-of-64 noaux_tc shape") { + const HfConfig cfg = vllm::LoadHfConfig(WriteGlmTinyConfig("router_fwd", 4)); + const DeepseekV2Params glm = vllm::ParseGlm4MoeLiteParams(cfg); + REQUIRE(glm.router_dtype_is_f32); + REQUIRE(glm.is_moe_layer(1)); + REQUIRE_FALSE(glm.is_moe_layer(0)); // first_k_dense_replace: 1 + + // Fixed seeds, so the count below is a property of the build and not of a + // random draw. Each seed is a whole synthetic GLM: different attention, + // different gate, different experts, different bias. + const std::vector seeds = {1, 17, 53, 101, 199, 271, 353, 431, + 509, 601, 691, 787, 877, 971, 1063, 1153}; + int seeds_differing = 0; + double worst = 0.0; + for (uint32_t s : seeds) { + DeepseekV2Weights w = GlmTinyWeights(glm, s); + REQUIRE(w.params.router_dtype_is_f32); + const std::vector f32 = RunTiny(w); + REQUIRE(!f32.empty()); + for (float x : f32) REQUIRE(std::isfinite(x)); + + // The arm this port took before #2928: the DeepSeek-V2 answer for a config + // that ships no `moe_router_dtype`. + w.params.router_dtype_is_f32 = false; + const std::vector bf16 = RunTiny(w); + REQUIRE(bf16.size() == f32.size()); + for (float x : bf16) REQUIRE(std::isfinite(x)); + + bool differs = false; + for (size_t i = 0; i < f32.size(); ++i) { + if (f32[i] != bf16[i]) differs = true; + worst = std::max(worst, std::abs(static_cast(f32[i]) - + static_cast(bf16[i]))); + } + if (differs) ++seeds_differing; + } + + MESSAGE("router dtype observable in the GLM forward: ", seeds_differing, "/", + seeds.size(), " seeds differ, worst |f32 - bf16| logit = ", worst); + // The claim is bimodal, not a tolerance: rounding the logits in front of a + // top-4-of-64 `noaux_tc` selection changes WHICH experts run. One seed is + // enough to falsify "a token gate cannot see this either way"; the count and + // the margin are printed so a later reader sees how far from the edge it is. + CHECK(seeds_differing >= 1); + CHECK(worst > 0.0); +} + +// ════════════════════════════════════════════════════════════════════════════ +// T2b — determinism, so T2's comparison is a dtype statement and not a +// run-to-run one. +// ════════════════════════════════════════════════════════════════════════════ +TEST_CASE("glm4-moe-lite router: each arm is deterministic run to run") { + const HfConfig cfg = vllm::LoadHfConfig(WriteGlmTinyConfig("router_det", 4)); + const DeepseekV2Params glm = vllm::ParseGlm4MoeLiteParams(cfg); + DeepseekV2Weights w = GlmTinyWeights(glm, 17); + + const std::vector a = RunTiny(w); + const std::vector b = RunTiny(w); + REQUIRE(a.size() == b.size()); + CHECK(std::memcmp(a.data(), b.data(), a.size() * sizeof(float)) == 0); + + w.params.router_dtype_is_f32 = false; + const std::vector c = RunTiny(w); + const std::vector d = RunTiny(w); + REQUIRE(c.size() == d.size()); + CHECK(std::memcmp(c.data(), d.data(), c.size() * sizeof(float)) == 0); +} + +// ════════════════════════════════════════════════════════════════════════════ +// T3 — the MECHANISM, measured rather than argued. The error a too-narrow +// router store makes is a SELECTION error, so it is bimodal and not a +// tolerance: assert that the selected SET changes, and print the margin. +// ════════════════════════════════════════════════════════════════════════════ +namespace { + +vt::Device CpuDev() { return vt::Device{vt::DeviceType::kCPU, 0}; } + +vt::Tensor Contig(void* data, DType dt, const std::vector& shape) { + vt::Tensor t; + t.data = data; + t.dtype = dt; + t.device = CpuDev(); + t.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = t.rank - 1; i >= 0; --i) { + t.shape[i] = shape[static_cast(i)]; + t.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return t; +} + +// One `vt::MoeRouterTopK` call at GLM-4.7-Flash's REAL router configuration. +std::vector RouteGlm(const std::vector& logits, int64_t T, int64_t E, + std::vector* bias) { + vt::MoeRouterTopKArgs args{}; + args.top_k = 4; // num_experts_per_tok + args.renormalize = true; // norm_topk_prob + args.scoring_func = vt::MoeScoringFunc::kSigmoid; // noaux_tc hardcodes sigmoid + args.num_expert_group = 1; // n_group + args.topk_group = 1; // topk_group + args.routed_scaling_factor = 1.8f; // routed_scaling_factor + std::vector weights(static_cast(T * args.top_k), 0.0f); + std::vector ids(static_cast(T * args.top_k), -1); + vt::Tensor tl = Contig(const_cast(logits.data()), DType::kF32, {T, E}); + vt::Tensor tw = Contig(weights.data(), DType::kF32, {T, args.top_k}); + vt::Tensor ti = Contig(ids.data(), DType::kI32, {T, args.top_k}); + vt::Tensor tb = Contig(bias->data(), DType::kF32, {E}); + vt::Queue q = Q(); + vt::MoeRouterTopK(q, tw, ti, tl, args, &tb); + return ids; +} + +} // namespace + +TEST_CASE("glm4-moe-lite router: the bf16 store CHANGES WHICH EXPERTS RUN") { + // GLM-4.7-Flash's real router: 64 routed experts, top-4, `noaux_tc` sigmoid + // with a learned selection bias, `norm_topk_prob`, `routed_scaling_factor` + // 1.8, one expert group. + const int64_t E = 64, K = 4, T = 4096; + std::mt19937 rng(20260904); + std::normal_distribution nd(0.0f, 1.0f); + std::vector logits_f32(static_cast(T * E)); + for (float& x : logits_f32) x = nd(rng); + std::vector bias(static_cast(E)); + for (float& x : bias) x = 0.05f * nd(rng); + + // The ONLY difference: `deepseek_v2.cpp:363` stores the gate GEMM's f32 result + // into a bf16 buffer before the top-k reads it. + std::vector logits_bf16 = logits_f32; + for (float& x : logits_bf16) x = vt::BF16ToF32(vt::F32ToBF16(x)); + + std::vector bias_a = bias, bias_b = bias; + const std::vector ids_f32 = RouteGlm(logits_f32, T, E, &bias_a); + const std::vector ids_bf16 = RouteGlm(logits_bf16, T, E, &bias_b); + REQUIRE(ids_f32.size() == ids_bf16.size()); + + int tokens_with_a_different_SET = 0; + for (int64_t t = 0; t < T; ++t) { + std::vector a(ids_f32.begin() + static_cast(t * K), + ids_f32.begin() + static_cast((t + 1) * K)); + std::vector b(ids_bf16.begin() + static_cast(t * K), + ids_bf16.begin() + static_cast((t + 1) * K)); + std::sort(a.begin(), a.end()); + std::sort(b.begin(), b.end()); + if (a != b) ++tokens_with_a_different_SET; + } + + const double rate = static_cast(tokens_with_a_different_SET) / + static_cast(T); + MESSAGE("bf16 router store changed the top-4 SET for ", + tokens_with_a_different_SET, "/", T, " tokens (", 100.0 * rate, "%)"); + // A SET difference means a different expert's weights ran. It cannot be + // absorbed by a tolerance, and it is what `deepseek_v2.h:164-168`'s "a token + // gate cannot see this either way" does not cover: that sentence is about a + // store that is too WIDE. + CHECK(tokens_with_a_different_SET > 0); +} + +// ════════════════════════════════════════════════════════════════════════════ +// T4 — REACHABILITY. The two cases above prove the value; this one proves that +// the PRODUCTION load-and-forward path carries it. It goes in through +// `ModelRegistry::Load` and `ModelRegistry::Forward` over a synthetic +// GLM-4.7-Flash checkpoint on disk — no hand-built `DeepseekV2Weights`, no +// internal type — and compares the registry's own logits to the two arms. +// +// Delete `weights.params = ParseGlm4MoeLiteParams(config);` from +// `LoadGlm4MoeLiteForCausalLM` and this case reds while T1 and T2 stay green, +// which is exactly what the loader call site owes. +// ════════════════════════════════════════════════════════════════════════════ +namespace { + +struct StEntry { + std::string name; + std::string dtype; // "BF16" or "F32" + std::vector shape; + uint32_t seed = 0; + float scale = 0.08f; +}; + +void AppendEntry(std::vector* out, const std::string& name, + const std::vector& shape, uint32_t seed, + float scale = 0.08f, const char* dtype = "BF16") { + out->push_back(StEntry{name, dtype, shape, seed, scale}); +} + +// Exactly the tensors `LoadDeepseekV2ForCausalLMWeights` resolves for this +// config, in checkpoint layout: the q_lora attention branch +// (`q_a_proj` + `kv_a_proj_with_mqa` merged into `fused_qkv_a_proj`, +// deepseek_v2_weights.cpp:160-173), a dense layer 0 (`first_k_dense_replace: 1`) +// and a `noaux_tc` MoE layer 1 with its f32 `e_score_correction_bias`, 64 routed +// experts and one shared expert. +std::vector GlmTinyCheckpointEntries(const DeepseekV2Params& p, + uint32_t seed0) { + const int64_t H = p.hidden_size, V = p.vocab_size; + const int64_t N = p.mla.num_heads, L = p.mla.kv_lora_rank; + const int64_t R = p.mla.qk_rope_head_dim, QL = p.mla.q_lora_rank; + const int64_t Dqk = p.mla.qk_head_dim(), VD = p.mla.v_head_dim; + const int64_t P = p.mla.qk_nope_head_dim; + const int64_t E = p.n_routed_experts, I = p.moe_intermediate_size; + std::vector e; + uint32_t s = seed0; + AppendEntry(&e, "model.embed_tokens.weight", {V, H}, s++); + AppendEntry(&e, "model.norm.weight", {H}, s++, 0.5f); + AppendEntry(&e, "lm_head.weight", {V, H}, s++); + for (int64_t l = 0; l < p.num_hidden_layers; ++l) { + const std::string b = "model.layers." + std::to_string(l) + "."; + AppendEntry(&e, b + "input_layernorm.weight", {H}, s++, 0.5f); + AppendEntry(&e, b + "post_attention_layernorm.weight", {H}, s++, 0.5f); + AppendEntry(&e, b + "self_attn.q_a_proj.weight", {QL, H}, s++); + AppendEntry(&e, b + "self_attn.kv_a_proj_with_mqa.weight", {L + R, H}, s++); + AppendEntry(&e, b + "self_attn.q_a_layernorm.weight", {QL}, s++, 0.5f); + AppendEntry(&e, b + "self_attn.q_b_proj.weight", {N * Dqk, QL}, s++); + AppendEntry(&e, b + "self_attn.kv_a_layernorm.weight", {L}, s++, 0.5f); + AppendEntry(&e, b + "self_attn.kv_b_proj.weight", {N * (P + VD), L}, s++); + AppendEntry(&e, b + "self_attn.o_proj.weight", {H, N * VD}, s++); + if (p.is_moe_layer(l)) { + AppendEntry(&e, b + "mlp.gate.weight", {E, H}, s++); + AppendEntry(&e, b + "mlp.gate.e_score_correction_bias", {E}, s++, 0.05f, + "F32"); + for (int64_t x = 0; x < E; ++x) { + const std::string ex = b + "mlp.experts." + std::to_string(x) + "."; + AppendEntry(&e, ex + "gate_proj.weight", {I, H}, s++); + AppendEntry(&e, ex + "up_proj.weight", {I, H}, s++); + AppendEntry(&e, ex + "down_proj.weight", {H, I}, s++); + } + const int64_t SI = p.shared_intermediate_size(); + AppendEntry(&e, b + "mlp.shared_experts.gate_proj.weight", {SI, H}, s++); + AppendEntry(&e, b + "mlp.shared_experts.up_proj.weight", {SI, H}, s++); + AppendEntry(&e, b + "mlp.shared_experts.down_proj.weight", {H, SI}, s++); + } else { + AppendEntry(&e, b + "mlp.gate_proj.weight", {p.intermediate_size, H}, s++); + AppendEntry(&e, b + "mlp.up_proj.weight", {p.intermediate_size, H}, s++); + AppendEntry(&e, b + "mlp.down_proj.weight", {H, p.intermediate_size}, s++); + } + } + return e; +} + +void WriteGlmTinyCheckpoint(const std::vector& entries, + const std::string& path) { + nlohmann::json header = nlohmann::json::object(); + size_t off = 0; + for (const StEntry& e : entries) { + size_t n = 1; + for (int64_t d : e.shape) n *= static_cast(d); + const size_t w = e.dtype == "F32" ? 4u : 2u; + header[e.name] = {{"dtype", e.dtype}, + {"shape", e.shape}, + {"data_offsets", {off, off + n * w}}}; + off += n * w; + } + const std::string hs = header.dump(); + std::ofstream out(path, std::ios::binary); + const uint64_t hlen = hs.size(); + out.write(reinterpret_cast(&hlen), 8); + out.write(hs.data(), static_cast(hs.size())); + for (const StEntry& e : entries) { + size_t n = 1; + for (int64_t d : e.shape) n *= static_cast(d); + std::mt19937 rng(e.seed); + std::uniform_real_distribution dist(-e.scale, e.scale); + for (size_t i = 0; i < n; ++i) { + const float v = dist(rng); + if (e.dtype == "F32") { + out.write(reinterpret_cast(&v), 4); + } else { + const uint16_t b = vt::F32ToBF16(v); + out.write(reinterpret_cast(&b), 2); + } + } + } +} + +} // namespace + +TEST_CASE("glm4-moe-lite router: the PRODUCTION load+forward carries the f32 router") { + const std::string cfg_path = WriteGlmTinyConfig("router_reach", 4); + const HfConfig cfg = vllm::LoadHfConfig(cfg_path); + const DeepseekV2Params p = vllm::ParseGlm4MoeLiteParams(cfg); + REQUIRE(p.is_moe_layer(1)); + REQUIRE(p.router_dtype_is_f32); + + // A fixture that cannot SEPARATE the two arms witnesses nothing, so the case + // searches a fixed seed list for one that does and then uses it. Same + // population T2 samples; here it also has to survive the safetensors round + // trip and the real loader. Deterministic: the seeds are written down. + const std::vector seeds = {7, 61, 137, 233, 331, 431, 541, 653, 769, 883}; + const std::string ckpt = ScratchDir() + "/router_reach.safetensors"; + std::vector want_f32, want_bf16; + uint32_t witness = 0; + for (uint32_t seed : seeds) { + WriteGlmTinyCheckpoint(GlmTinyCheckpointEntries(p, seed), ckpt); + std::vector probe; + probe.push_back(vllm::SafetensorsFile::Open(ckpt)); + DeepseekV2Weights w = vllm::LoadDeepseekV2ForCausalLMWeights( + probe, cfg, /*allow_mtp_tail=*/true); + w.params.router_dtype_is_f32 = true; + const std::vector a = RunTiny(w); + w.params.router_dtype_is_f32 = false; + const std::vector b = RunTiny(w); + REQUIRE(a.size() == b.size()); + if (a != b) { + want_f32 = a; + want_bf16 = b; + witness = seed; + break; + } + } + MESSAGE("separating checkpoint seed = ", witness); + REQUIRE(witness != 0); + REQUIRE(!want_f32.empty()); + REQUIRE(want_f32 != want_bf16); + // `ckpt` on disk is the witness fixture: the loop wrote it last before break. + + // ── the production entry points ────────────────────────────────────────── + std::vector shards; + shards.push_back(vllm::SafetensorsFile::Open(ckpt)); + const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(shards); + std::unique_ptr model = vllm::ModelRegistry::Load(cfg, source); + REQUIRE(model != nullptr); + CHECK(model->registration().architecture == "Glm4MoeLiteForCausalLM"); + + const int64_t T = 5, bs = 8; + MlaCachePool pool(p, /*num_blocks=*/2, bs); + const CommonAttentionMetadata am = PrefillMeta(T, bs); + const std::vector tokens = {3, 17, 42, 8, 61}; + const std::vector positions = {0, 1, 2, 3, 4}; + const std::vector logits_indices; + std::vector gdn_state; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + vt::Queue q = Q(); + const vllm::ModelForwardInput input{tokens, positions, am, gdn_meta, + pool.attn_kv, gdn_state, cfg, q, + logits_indices}; + const vllm::ForwardLogits got = vllm::ModelRegistry::Forward(*model, input); + // `gather_logits` defaults TRUE, so the registry's DEFAULT configuration takes + // the device-resident arm (`ForwardGlm4MoeLiteForCausalLM` -> + // `DeepseekV2Model::ForwardDevice`). That is the arm to measure; read it back. + REQUIRE(got.rows * got.vocab == static_cast(want_f32.size())); + std::vector logits(want_f32.size(), 0.0f); + if (got.on_device()) { + vt::Backend& cpu = vt::GetBackend(vt::DeviceType::kCPU); + cpu.Copy(q, logits.data(), got.device_tensor.data, + logits.size() * sizeof(float)); + cpu.Synchronize(q); + } else { + REQUIRE(got.host.size() == want_f32.size()); + logits = got.host; + } + + // The registry's own output IS the f32 arm, and is NOT the bf16 one. + CHECK(std::memcmp(logits.data(), want_f32.data(), + want_f32.size() * sizeof(float)) == 0); + CHECK(std::memcmp(logits.data(), want_bf16.data(), + want_bf16.size() * sizeof(float)) != 0); +}