Karmic beta to canonical (Oct 5): rows of #868, through #982 by voipmonitor · Pull Request #894 · local-inference-lab/vllm

voipmonitor · 2026-09-24T17:41:34Z

Snapshot of integration/karmic-kraken-beta at f1c2508f10 (Oct 5, 18:10 Prague), fast-forwarded from 980d84ef after #961, #963, #965, #970, #971, #974, #976, #977, #978, #980, #983 and #982. Canonical dev/karmic-kraken @ ab86b70734 is an ancestor of this branch, so the diff of this PR is exactly the beta-only changes. The full table with evidence, twins and audit is in #868; the b12x side is local-inference-lab/b12x#448.

New since the Oct 1 snapshot:

[beta] Qwen3.8-Flash-Next: share one host-RAM PLE table between same-host processes #961: Qwen3.8-Flash-Next processes share one host-RAM PLE table (opt-in; needs b12x#454). Twin [KK] Qwen3.8-Flash-Next: share one host-RAM PLE table between same-host processes #962.
[beta] Integrate CSF loading and online scale compression with the Kimi runtime #963: FP4-CSF checkpoint loading (stored, losslessly compressed expert scales) plus the Kimi-K3 runtime units [KK] Bound InstantTensor staging and retain deferred checkpoint ownership #841, [KK] Reuse identical native payloads and recover incomplete autotune caches #846, [KK] Preserve parallel-draft precision and mixed DCP cache geometry #858, [KK] Prepare B12X Kimi KDA prefill and expose MLA partial precision #937, [KK] Load DS4.1 and Kimi-K3 lossless X4T checkpoints #949 and [KK] Release dense MLA priming buffers after preparation #953. Twins: those PRs, with Own FP4-CSF loading and preserve native DS4.1 expert precision #956 on [KK] Load DS4.1 and Kimi-K3 lossless X4T checkpoints #949.
Quantize exact GLM Spark projections and MTP experts on load #965: GLM Spark weights converted on load (opt-in; no profile uses it). Twin Load exact GLM Spark projections and NVFP4 MTP from the QAD source #968.
b12x MoE: opt-in NVFP4-activation prefill for W4A16 experts, decode rows stay W4A16 #970: opt-in NVFP4-activation prefill for W4A16 MoE, decode rows stay W4A16 (needs b12x#472). Twin [KK] b12x MoE: opt-in NVFP4-activation prefill for W4A16 experts, decode rows stay W4A16 #972.
Serve lossless MXFP4-CSF DeepSeek-V4-Flash and Vision-Exp checkpoints #971: MXFP4-CSF loading for DeepSeek-V4-Flash and Vision-Exp. Twin [KK] Load lossless MXFP4-CSF DeepSeek-V4-Flash and Vision-Exp checkpoints #973.
qwen4_exp: resolve the PLE storage dtype from an FP4-CSF checkpoint root #974: Qwen PLE storage dtype read from an FP4-CSF checkpoint root. Twin [KK] qwen4_exp: resolve the PLE storage dtype from an FP4-CSF checkpoint root #975.
modelopt: zero-initialize NVFP4 MoE input scales so W4A16 layers never enable A4 prefill from garbage #976 and b12x MoE: choose the A4 prefill by row type, not by call size #977: NVFP4 MoE input scales zero-initialized, so layers without stored scales never enable A4 prefill; A4 prefill chosen by row type (prefill rows A4, decode rows W4A16). Twin [KK] b12x MoE: opt-in NVFP4-activation prefill for W4A16 experts, decode rows stay W4A16 #972 (same commits as b12x MoE: opt-in NVFP4-activation prefill for W4A16 experts, decode rows stay W4A16 #970).
NVFP4-CSF: expand the next layer's scales during its attention #978: NVFP4-CSF layers expand the next layer's scales on a side stream during its attention. Twin [KK] NVFP4-CSF: expand the next layer's scales during its attention #979.
[Worker] Load weights without the split limit under expandable segments #980: weights load without max_split_size_mb:20 when expandable segments are on. Twin [Worker] Load weights without the split limit under expandable segments (canonical twin of #980) #981.
[b12x] Clarify A4 prefill logs for FP4-CSF and draft MoE layers #983: the startup log names the prefill choice of every NVFP4-CSF layer (log only).
MXFP4-CSF: load DeepSeek-V4.1-Flash checkpoints at TP3 #982 (Defilan): the MXFP4-CSF loader admits TP3 for DeepSeek-V4.1-Flash.

The online CSF scale encoder was taken out of beta on Oct 3 (history rewritten); nothing of it is in this branch. Squash-merge it to make dev/karmic-kraken equal to the beta.

🤖 Generated with Claude Code

…at TP > 1 Two per-step costs of DFlash under tensor parallelism: - The context projection fc (target aux hidden states -> draft hidden, e.g. 20480 x 4096) was a ReplicatedLinear, so every rank streamed the whole weight each step. It is now a ColumnParallelLinear with gather_output=True at TP > 1: each output keeps its full dot product (bit-identical), each rank reads 1/tp of the weight. MiMo-V2.6-Flash TP4 + DFlash K7: C1 step time -60 us, +20K tokens of KV capacity. - Probabilistic drafting all-gathered the drafter's [rows, vocab] logits (224 x 152576 BF16 at C32 K7, ~1.2 ms of a ~33 ms PCIe TP4 step). With VLLM_DFLASH_VOCAB_PARALLEL_DRAFT=1 each rank Gumbel-samples its vocab shard with the same absolute-token-id noise, and the per-rank winners are merged; verification uses the drawn token's logit and a global logsumexp combined from per-rank statistics, and rejection resampling runs per shard the same way. The all-gathers shrink to [rows, 5] and [reqs, 2]. It applies to plain vocab-parallel heads, standard rejection sampling and no watermarking; anything else keeps the full-vocab path. Default off. Tests: tests/v1/spec_decode/test_vocab_parallel_drafting.py simulates TP4 shards against the full-vocab kernels: identical draft tokens and cache shards, logsumexp within 1e-5 of FP64, identical rejection sampling (accept lengths, resampled and bonus tokens) across greedy and 0.6-1.3 temperatures, CUDA-graph pad rows and verification depths 0-7. MiMo-V2.6-Flash TP4 C32: 2182 -> 2263 tok/s. Signed-off-by: MadeBy561 <madeby561@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Online `fp8_per_block` quantization (speculative_config.quantization) quantizes the drafter's linears while they load, so the fused context K/V projection concatenated raw FP8 rows without their block scales and F.linear failed on a BF16 x FP8 matmul. Dequantize block-FP8 K/V rows with their scales for the fused projection; each layer keeps its own FP8 GEMMs, and serialized MXFP8 rows stay packed for their fused linear method. Signed-off-by: MadeBy561 <madeby561@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…recoverable The online acceptance estimator learns only from drafts that adaptive verification admits. A draft predicted near-certain carries almost no IRLS weight, so a data-poor refit that saw one confidently wrong draft divided its residual by ~the ridge and moved an intercept by about -20. Predicted acceptance then fell to ~1e-8, the budget admitted no drafts, and nothing was ever graded again: speculation stopped for the rest of a long request (reported as DFlash "falling off" during long thinking). - Bound each refit's coefficient change (trust region) and keep the slope and intercepts within fixed limits, off logistic saturation. - After EXPLORE_AFTER_IDLE_STEPS consecutive steps that schedule drafts but verify none, verify one draft per request for a step so the estimator keeps receiving evidence. Every rank makes the same choice. Only the number of verified drafts changes; rejection sampling keeps the output distribution exact. Signed-off-by: MadeBy561 <madeby561@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

max_num_scheduled_tokens bounds a step so that decode streams keep moving while long prompts prefill. When a step has no runnable decode (a single long request, or every running request still prefilling), the cap only shrinks the prefill chunk: more steps and more per-step overhead for the same work. With VLLM_SCHEDULER_UNCAP_PREFILL_ONLY_STEPS=1, a step whose decode-aware path finds no eligible decode uses max_num_batched_tokens as its token cap and budget. Steps with runnable decode are unchanged, and the default (0) keeps current scheduling for every deployment. MiMo-V2.6-Flash TP4, batch 4096 / scheduled 2048: standalone prefill at 32K goes from 11.8K to 12.7K tok/s (+8-12% across 8K-128K). Tests: tests/v1/core/test_prefill_compute_share_scheduler.py (both settings). Signed-off-by: MadeBy561 <madeby561@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

MiMo-V2 audio needs two torchaudio transforms: the media parser resamples input audio to 24 kHz (default method "torchaudio"), and MiMoVLProcessor computes a 128-bin mel spectrogram with torchaudio.transforms.MelSpectrogram. Images built on a vendor PyTorch (for example NGC 2.14.0a0) have no matching torchaudio wheel, so every audio request failed ("torchaudio is required for audio"). vllm.multimodal.audio gains plain-PyTorch ports of the default torchaudio.transforms.Resample (Hann-windowed sinc, width 6, rolloff 0.99, kernel built in float64 and stored as float32) and torchaudio.transforms.MelSpectrogram (periodic Hann window, reflect-padded STFT, HTK mel scale, no filterbank normalization). They follow torchaudio's arithmetic step for step, and are used only when torchaudio is not installed: - _get_torchaudio_resampler returns the port (cached per rate pair), so AudioResampler(method="torchaudio") works everywhere with the same output. - MiMoVLProcessor uses the ports for its mel spectrogram and its own resampler. Deployments with torchaudio installed take exactly the same code paths as before. Tests (tests/multimodal/test_audio.py, tests/models/multimodal/ test_mimo_v2_omni.py): - with torchaudio: torch.equal against torchaudio for resampling from 8-48 kHz (mono and stereo) and for MiMo's and torchaudio's default mel settings; - without torchaudio: pinned reference values, the "torchaudio" resampler path, and MiMo's audio preprocessing end to end (frames and token count). On a torchaudio-less image: 75 passed, 13 skipped (the torchaudio comparisons). Against torchaudio 2.11 / torch 2.14: 38/38 cases bitwise identical. Signed-off-by: MadeBy561 <madeby561@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…lags serve-mimo26-flash.sh now starts the configuration the MiMo b12x work was qualified with: - DFlash drafting (7 tokens) from ${MODEL_PATH}/dflash, drafted across the target's TP ranks, with adaptive verification, probabilistic drafts and standard rejection sampling; the drafter keeps an FP8 KV cache (DFLASH_KV_CACHE_DTYPE). - The MiMo b12x switches, each overridable: paged-decode KV layout (VLLM_KV_CACHE_LAYOUT=BLHNC), BF16 activations for the FP4 MoE, FP32 router weights in the W4A16 top-k combine, exact dense split-K (B12X_DENSE_SPLITK_TURBO=0), BF16 GEMV plans, L2 weight prefetch, vocab-parallel drafting, uncapped prefill-only steps, and the PCIe all-reduce size thresholds. - Scheduling: 32 sequences, 4096 batched / 2048 scheduled tokens, prefill compute share 0.8, async scheduling, FULL_AND_PIECEWISE graphs up to 256. - Vision and audio through MiMoV2OmniForCausalLM. - KV cache FP8 by default: twice the BF16 capacity, and what fits the model at TP2. KV_CACHE_DTYPE=bfloat16 selects the exact BF16 cache. - MODEL_REVISION 5711b268: the first Hugging Face revision whose dflash/config.json is valid JSON (fixed in b2674c72); weights unchanged. Signed-off-by: MadeBy561 <madeby561@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Signed-off-by: MadeBy561 <madeby561@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

…d-store-tracking [KV Connector] Track allocated blocks and keep CoW group store ranges aligned (from #867)

…ternal-prefix-hits [KV Connector] Serve external prefix hits for align-mode Mamba hybrids (from #867)

#864 requires an atomic request-boundary checkpoint connector for every Qwen KV transfer. That also rejected CACHE_MODE=native: vLLM maps native offload to SimpleCPUOffloadConnector with --recurrent-checkpoint-policy aligned, so every beta since #864 refused to start Qwen with the native CPU cache. Aligned retention does not use request-boundary checkpoints. Accept it when the connector declares supports_aligned_hybrid_transfer (SimpleCPU, and OffloadingConnector with the #869/#870 fixes) and DCP is 1. Other connectors, DCP and the auto/request_boundaries policies keep the atomic requirement. Based on commit 4 of #867, narrowed to connectors that move aligned hybrid state. Co-authored-by: Carlos Augusto <mb@lab.how> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

…load-gate fix(config): let aligned Qwen offload through the atomic-checkpoint gate

The request timeline started at the renderer's arrival stamp, so a request held after the server received it but before rendering, for example while its body was still arriving, looked fast. An outermost ASGI middleware now stamps the HTTP receipt and the end of the request body, the timeline reports both spans, and a request that ends without any response after the stall threshold is logged even though it never reached the engine. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

…-timeline feat(serve): time requests from HTTP receipt, not only from rendering

…hrough stale block-table rows (vllm-project#56734) Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit d2983f2) Signed-off-by: Martin Vit <martin@voipmonitor.org>

Qwen3.8-Flash-Next checkpoints are published under two model types that load the same Qwen4Exp implementation: the official ones declare qwen4_exp / qwen4_exp_text, renamed ones declare qwen3_8_flash_next / qwen3_8_flash_next_text. Only Qwen3_8FlashNextTextConfig opted in to supports_full_tp_dcp_with_kv_gather, so ModelConfig rejected TP4/DCP4 (two KV heads) for qwen4_exp checkpoints while accepting it for renamed ones. Declare the capability on Qwen4ExpTextConfig so both spellings inherit it. Only the B12X QSA backend implements DCP; the non-B12X NVIDIA and AMD QSA backends still refuse DCP > 1 when they are constructed.

…refill b12x MoE: opt-in NVFP4-activation prefill for W4A16 experts, decode rows stay W4A16

Add the deepseek_v4_flash MXFP4-CSF family (43 target layers of 256 routed experts, hidden 4096, MoE intermediate 2048, DeepSeek-native tensor names, TP1/2/4/8) and dispatch it to DeepseekV4Mxfp4CsfConfig. The config derives from the class CUDA registers as deepseek_v4_fp8 and parses the retained source block-FP8 fields, so attention, dense, shared-expert, vision and MTP/DSpark draft weights keep their native methods. Target routed experts read compressed scales with MXFP8 activations unless VLLM_B12X_MOE_FP4_FORCE_A16=1 requests BF16. Expert parallelism is rejected: MegaMoE experts bypass quantization methods and would never read the compressed experts. The loader family table records each family's compressed layer range and rejects reads outside it. The FP4-CSF guide documents the DeepSeek MXFP8 activation default and the DeepSeek-V4-Flash serving contract. Validation: tests/quantization/test_mxfp4_csf.py and test_nvfp4_csf.py pass on CPU (74 tests, 13 new); ruff, mypy hook, typos pass. No GPU serving run yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>