feat(GFX1100-TG200): T25 keep ssm_out as Q5_K with runtime input permutation by ghazni101 · Pull Request #2807 · mudler/vllm.cpp

ghazni101 · 2026-09-03T17:48:05Z

Closes #2806.

Row: GFX1100-TG200

T25 keeps the SSM output projection as Q5_K and adds a runtime V-head
permutation kernel (PermuteVHeadsKernelRocm). The permutation reorders
V-heads from grouped (krpk+r) to tiled (rnum_k+k) order before the K-quant
GEMV, enabling the keep-quant path to avoid a dequantize-requantize round
trip on the output projection.

Adds kPermuteVHeads op, PermuteVHeads dispatch function, and the ROCm
kernel. Restores kCastF16/CastF16/CastF16KernelRocm that were lost
during cherry-pick auto-merge resolution.

Validated by test_rocm_quant_dot (12/12 cases, 797 assertions).
Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy
decode, seed 0.

Depends on #2782 (F1), #2790 (T4a), #2792 (lever-C), #2796 (T27), #2800
(T24), and #2804 (T21).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GLM-5-2 [OMP]

… earns ae56547 lowered scripts/record-anchor-baseline.json from main's 28 stale to 0 and said in its body that it had updated "all 28 relocated symbols" across four matrix files. It edited no matrix file. The budget moved, the citations did not, and check-agent-record then reds on the merged tree at 30 > 0. Restores the baseline to main's earned 28 stale / 5 broken, and repairs the two citations this stack really does invalidate: T21 (mudler#2804) and T25 (mudler#2807) add lines to qwen3_5_gguf_weights.cpp, moving LoadGgufSharedEmbedAndHeadBf16 from 1034 to 1051 and LoadQwen3_5MTPFromGguf from 1447 to 1526. With those two repaired the tree measures exactly 28, which is the baseline it now carries. This leaves ae56547's two real fixes -- the MoeSiluMul bf16 intermediate rounding and the wvSplitK m + y < M store bound -- untouched. Closes mudler#2934. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

localai-org-maint-bot · 2026-09-05T02:32:05Z

Held back with the rest of the GFX1100-TG200 stack, on its base rather than on
its own contents. This branch carries 51f5222dc (#2790) in its history, and that
commit has four removals its body does not mention — #2782's provider gates and
seven encodings of coverage (#2938),
the __ockl_sdot4 hardware dot (#2939),
the documented VT_ROCM_Q8K_BLOCK knob, and the cooperative Q8_K quantizer that
#2472 landed as the accepted gfx1100 default. The full write-up is on
#2790.

I reviewed this change on its own and have no objection to it. Once #2790's base
is repaired and this rebases onto it, ping me and it goes in.

Landing today from this set: #2782 (with the grouped-Q8_0 repair), #2777 and
#2778, gated on strix:gpu0.

ghazni101 · 2026-09-05T13:07:18Z

Rebased onto the repaired stack (base b9f2ef4 = the external-contributor landing branch; chain tips: lever-C 0ac919d, T27 48f9390, T24 5d8a95d, T21 6b7deb7, T25 2ff6af4). Verification on gfx1100 (RX 7900 XTX, container rocm-dev:10.0.0, -DCMAKE_HIP_ARCHITECTURES=gfx1100): record gates green (check-env-doc, check-agent-record, check-rocm-dp4a-intrinsic); vllm-cli + test_rocm_quant_dot compile and link at -Werror; the DEFAULT path (all arms off) produces byte-identical completions to the staging baseline on Qwen3.5-4B Q4_K_M (canonical TG200 prompt, 256 tokens, greedy, seed 0) — leg-by-leg bisect: lever-C/T27/T24/T21/T25 all IDENTICAL. One default-path defect was found and fixed during the rebase: T21's VT_GDN_ROWPERM_KEEP_QUANT was default-ON and silently flipped trunk numerics; it is now opt-in by value like its COLPERM sibling, and the token-exact gate passes again.

This branch's head is now 2ff6af4; content otherwise unchanged from the reviewed version (plus the rebase repairs each commit body discloses). Ping for re-review.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:OMEN-ALPHA [OMP]

… kROCM The GGUF loader routes a block-typed weight to MatmulBTQuant whenever the running device has the provider, so registering these two ops lights up keep-quant compute on every ROCm board with no model-path change: the dense and grouped MoE towers stage once through ResidentWeight and dispatch to the new device GEMM. Coverage mirrors the CUDA sibling exactly — the ten Q8_K-family encodings plus a native Q8_0 arm. The integer dots are the portable scalar forms of the CPU reference bodies in the CPU accumulation order, because gfx1100 exposes no signed byte dot (v_dot4_i32_iu8 is unsigned-only; sdot4 needs a feature this target does not offer), and the gate is bit-exactness against the CPU tier at NMSE 1e-6 with the f64 dequant band at 5e-4. Unsupported dtypes throw naming the dtype instead of silently falling back to a host kernel that cannot follow device pointers; VT_GGUF_KEEP_QUANT=0 restores load-time expansion. Gates on gfx1100 / ROCm 7.14.0: test_rocm_quant_dot 132,094 assertions green across all ten encodings (decode through prefill shapes, broadcast and per-row grouped arms over a poisoned output buffer), focused ctest 'rocm|cross_device|quant' 20/21 with only the pre-existing MoeSiluMul bf16 exactness failure (mudler#1588) remaining, and an end-to-end Qwen3.5-0.8B Q4_K_M decode that is deterministic on device. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:ox-alpha [omp]

The new kROCM provider takes over kMatmulBTQuantGrouped from the kernel in rocm_grouped_gemm.hip and delegates Q4_K/Q5_K/Q6_K back to it, but not Q8_0. Q8_0 has no arm in rocm_quant_dot.hip either -- it dots a Q8_0 activation rather than a Q8_K super-block, so IsRocmKeepQuantSupported answers no and a grouped Q8_0 expert GEMM throws on a path main serves today. Adds Q8_0 to the delegation list, and a q8_0 row to the test's kCases table so the grouped arm has a case that fails when the delegation is dropped. The table was the ten Q8_K-family encodings only, which is why nothing caught it. Closes mudler#2927. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

…CM provider" This reverts commit 82d99ea. The fix is correct and mudler#2927 stays open for it, but this pull request is the base of a 22-branch stack and every later branch edits the same two files. Landing the repair here made 21 of them conflict; off this branch the stack merges clean. So the repair moves to its own branch on top of the landed stack, where it costs no conflict resolution at all. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

The new kROCM provider takes over kMatmulBTQuantGrouped from the kernel in rocm_grouped_gemm.hip and delegates Q4_K/Q5_K/Q6_K back to it, but not Q8_0. Q8_0 has no arm in rocm_quant_dot.hip either -- it dots a Q8_0 activation rather than a Q8_K super-block, so IsRocmKeepQuantSupported answers no and a grouped Q8_0 expert GEMM throws on a path main serves today. Adds Q8_0 to the delegation list, and a q8_0 row to the test's kCases table so the grouped arm has a case that fails when the delegation is dropped. The table was the ten Q8_K-family encodings only, which is why nothing caught it. Closes mudler#2927. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

Issue mudler#1588 still lacks active cache-state evidence and a three-mode ROCm correctness gate. This spec fixes the post-write probes, dtype audit, tolerance policy, tests, review mutations, and hardware evidence before implementation starts. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-5.6-sol [codex]

The rejected plan treated a red local gate as usable, invented a numerical envelope, and described oracle and provider paths that could not run. Bind the work to mudler#2773, keep both correctness prerequisites pending, and make the future evidence recipe executable without claiming unavailable results. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6 [codex]

The mudler#2773 plan must describe the production caller and active oracle layout before instrumentation starts. Correct BF16 selector normalization, name the existing Qwen3.5 path, and record its shared-seam debt in mudler#2923. Separate SD storage from DS dump order and cite the active CPU attention test with its unchanged tolerances. Runtime acceptance remains pending. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [codex]

Adds the VT_GEMV_MMVQ=1 opt-in K-quant decode GEMV arm for MatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue (deleting the standalone QuantizeQ8KK launch) and widens the gate to engine dtypes (bf16/f16 activations, bf16/f32 outputs). Sub-levers: - lever-B1: VT_GEMV_MMVQ_FOLD_MAX makes the fold crossover tunable at runtime - lever-B2: VT_SKINNY_BF16=1 f32-out decode-skinny arm for GDN BA projections - repair: m-gates the whole dispatch and makes the GEMV bit-equal to baseline - repair-2: host-side dispatch-route counters + F1/F2 routing-witness gates - lever-B2 test: red-first f32-out decode-skinny gate, true-unset routing window Architecture: F1 moved the live MatmulBTQuantKernelRocm to rocm_quant_dot.hip (anonymous namespace, internal linkage). T4a's MMVQ arm lives in rocm_grouped_gemm.hip's version (external linkage, renamed to *Gdn). This PR adds delegation: rocm_quant_dot.hip forwards Q4_K/Q5_K/Q6_K calls to the Gdn version, preserving F1's IQ-type providers while activating T4a's MMVQ arm. The default path (VT_GEMV_MMVQ unset) is byte-unchanged from F1. The arm is opt-in and validated by test_rocm_quant_dot (6/6 cases, 719 assertions) and test_rocm_skinny_f32 (2/2 cases, 51 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Rebased onto the external-contributor landing branch (staging tip b9f2ef4), which already carries F1 (mudler#2782) and its grouped-Q8_0 repair (mudler#2927). This version carries NONE of the removals the previous base commit 51f5222 made: the ten-row kCases table, kMaxNmseErr/nmse_ref_max and the three F1 provider cases are restored beside this arm's kKQuantCases (mudler#2938); Dp4a keeps the __ockl_sdot4 hardware dot (mudler#2939); the documented VT_ROCM_Q8K_BLOCK selector (SelectQ8KQuantArm/LaunchQ8KQuantizer), the mudler#2472 cooperative gfx1100 default (QuantizeQ8KCooperativeK) and the VT_ROCM_Q6K_SMALL_PRIVATE A/B arm are restored with their witness helpers; the shared bench-evidence file keeps lever B1's section 14 record, whose truncation this branch had carried. Depends on mudler#2782 (F1 keep-quant GEMM infra). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP] Assisted-by: AGENT:OMEN-ALPHA [OMP]

…Norm epilogue Lever-C adds an opt-in fused norm-quant epilogue (VT_NORM_QUANT_FUSED=1): RmsNormRowKernel emits the row's Q8_K blocks alongside its normal output, and MatmulBTQuant's K-quant branch skips the standalone QuantizeQ8KK when the consuming activation matches the producer token. Byte-identical to the standalone path by construction (shared QuantQ8KSBlock body). New files: - src/vt/rocm/rocm_act_quant.h: shared Q8_K quant-block body - src/vt/rocm/rocm_norm_quant_bridge.h: producer-consumer token contract Also fixes T4a routing counter placement (moved outside anonymous namespace for external linkage) and restores VT_GEMV_MMVQ_FOLD_MAX env var reading that was lost during cherry-pick conflict resolution. The default path (VT_NORM_QUANT_FUSED unset) is byte-unchanged. Validated by test_rocm_quant_dot (12/12 cases, 797 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1) and mudler#2790 (T4a). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]

The standalone QuantizeQ8KK kernel used 1 thread per 256-element superblock, each doing a serial scan of 256 elements (~800 instructions). For decode (m=1, nsb=10) only 10 of 128 threads were active, and on wave32 each thread is its own wave, so the kernel took ~13.4 us/call = 540 us/tok (6.0% of wall time). The new QuantizeQ8KKWarpCoop kernel uses 8 threads per superblock (32 elements each). The amax scan is done per-chunk (ascending, ax > amax first-occurrence), then reduced across 8 threads via __shfl_xor_sync with lower-chunk-index tie-break — equivalent to a sequential scan of all 256 elements. The quantization (iscale = -127/mx, DNearestInt, clamp 127) and bsums are order-independent. Output is BYTE-IDENTICAL to the original QuantQ8KSBlock, asserted by the gate test (16/16, 839 assertions) under VT_QUANT_Q8K_WARP=1. For m=1, nsb=10: 1 block, 80/128 threads active (vs 10/128), 3 waves of ~100 instructions (vs 10 waves of ~800) = ~8x fewer wave-cycles. A/B on acceptance workload (Qwen3.5-4B Q4_K_M, 256 tokens, temp 0, seed 0): OFF median: 91.532 tok/s ON median: 93.417 tok/s +2.06%, 5/5 pairs ON>OFF, all 5 byte-identical (1039 bytes) Gated by VT_QUANT_Q8K_WARP (default OFF, read per-call). Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP] FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM [omp]

…pKernel The fused Q8_K quant epilogue in RmsNormRowCoopKernel re-reads the normalized output from global memory (DLoadAct on orow) after Pass 3 stores it. On gfx1100 the 5 KB bf16 row (h=2560) competes with the weight and input in the 16 KB L1, so the re-read can miss to L2. T24 stores the normalized row to dynamic shared memory during Pass 3 (when the value is already in registers) and reads from LDS in the quant epilogue, eliminating the global re-read. The LDS buffer is h * sizeof(Tout) bytes (5 KB for bf16 h=2560), well within the 64 KB per-CU limit. Env gate VT_RMSNORM_LDS_QUANT (default ON) controls the optimization: set to 0 to revert to the global re-read path for A/B isolation. The gate is read per-call so captured graphs and in-process tests pick it up at dispatch time. Byte-identity: the LDS store uses the same conversion as Store (bf16 RNE for bf16 output, exact copy for f32), and DLoadAct reads the same bytes from LDS as from global. Gate test: 16/16 cases, 839 assertions, all passed. A/B measurement pending: the co-tenant 27B model holds the GPU VRAM, blocking the acceptance workload. The A/B script is staged at agent-artifacts/tg200-t24/ab-t24.sh for when the GPU is available. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP] FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM [omp]

…ctions The GDN layers attn_qkv (Q5_K, 24 tensors [2560,8192]) and attn_gate (Q4_K, 24 tensors [4096,2560]) were expanded to bf16 at load time because the V-head row reorder classified them as kTransformedWeight. The reorder is a ROW permutation — quantization blocks are along the K (column) dimension and are self-contained per row — so it is block-safe. T21 routes these tensors as kMatmulWeight to allow keep-quant, copies the blocks via OwnGgufQuantBlocks(mmap_src=nullptr), and applies ReorderVRows to the block bytes at load time. The forward pass already dispatches quantized nk=true weights through vt::MatmulBT, so no forward-pass change was needed. A/B: +3.9% (87.4 to 90.8 tok/s median, 5/5 pairs). Gate 16/16, 839 assertions. Output coherent but not byte-identical (Q5_K integer dot product vs bf16 float MAC). VT_GDN_ROWPERM_KEEP_QUANT=0 reverts to the old bf16 expansion path for A/B isolation. The improvement is less than the projected 14% because the Q5_K GEMV kernel has lower effective bandwidth on small grids (n=2560) than assumed, and wvSplitKSml is more efficient on these grids than projected. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:glm-5-2 [omp] FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM [omp]

…t amax at 0 RmsNormRowCoopKernel's vec predicate constrained activation/residual widths but not the gamma type, while the vector body decodes the gamma as 8 bf16 halves per uint4. DispatchRmsNormWeight admits kF32/kF16 gammas independent of Tin (mudler#2492 decoupling), so with bf16 activation and an f32 or f16 gamma the vector path multiplied by garbage read from the wrong byte range. The predicate now requires Tw == __hip_bfloat16 exactly (sizeof==2 would still misread f16); the scalar path's Load() overloads decode per dtype and are unchanged. Also seeds the epilogue quantizer's mx/amax at 0 like the standalone QuantizeQ8KK: seeding from the thread's own element let a NaN poison d instead of producing the scalar contract's zero block.

ssm_out (out_proj) is Q5_K in the GGUF checkpoint but was expanded to bf16 at load time because the V-head column reorder (ReorderVCols) cuts across Q5_K 256-element block boundaries. T25 keeps the weight in tiled Q5_K order (no ReorderVCols) and permutes the 4096-element GEMV input from grouped to tiled order at runtime instead, cutting weight bandwidth ~4x (Q5_K ~5 MB vs bf16 20 MB per call). The permutation is a simple gather of 128-element groups within each of the 4096-element rows, gated by VT_GDN_COLPERM_KEEP_QUANT=1 (default OFF). A new out_proj_tiled flag on GdnLayerWeights distinguishes the tiled Q5_K path (needs input permutation) from the gdn_expand_nk bf16 path (already column-reordered, no permutation needed) — the nk flag alone conflates both. A/B (5 interleaved pairs, --max-tokens 256 --temperature 0 --seed 0): OFF median=90.930 tok/s, ON median=91.703 tok/s, +0.85%, 5/5 ON>OFF. Output coherent but NOT byte-identical (Q5_K vs bf16 weight precision). Gate test: 16/16, 839 assertions. The improvement is modest because the permutation kernel launch overhead (~13.4 us x 24 calls = ~322 us/tok) offsets most of the weight bandwidth savings (~368 us/tok). The net gain is ~46 us/tok. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP] FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM [omp]

ghazni101 · 2026-09-06T09:03:36Z

Rebased onto upstream/main (6f5e9dc). The base is now origin/main, not stage/ext-prs-2026-09-04. Ping for re-review.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GLM-5-2 [OMP]

ghazni101 force-pushed the row/GFX1100-TG200-T25 branch from e0d4314 to 8ea44eb Compare September 3, 2026 19:30

ghazni101 force-pushed the row/GFX1100-TG200-T25 branch from 8ea44eb to fdfb339 Compare September 4, 2026 07:45

localai-org-maint-bot mentioned this pull request Sep 5, 2026

The record-anchor baseline was lowered to zero stale without the repair that earns it #2934

Open

This was referenced Sep 5, 2026

perf(GFX1100-TG200): T16 YTILE=4 default for wvSplitK decode-skinny GEMV #2787

Open

perf(GFX1100-TG200): T4a MMVQ K-quant decode GEMV arm #2790

Open

ghazni101 force-pushed the row/GFX1100-TG200-T25 branch from fdfb339 to 2ff6af4 Compare September 5, 2026 13:05

ghazni101 force-pushed the row/GFX1100-TG200-T25 branch from 2ff6af4 to 4aba1b9 Compare September 5, 2026 16:43

ghazni101 and others added 9 commits September 6, 2026 08:49

ghazni101 added 5 commits September 6, 2026 08:50

ghazni101 force-pushed the row/GFX1100-TG200-T25 branch from 4aba1b9 to e740b77 Compare September 6, 2026 08:51

Navigation Menu

feat(GFX1100-TG200): T25 keep ssm_out as Q5_K with runtime input permutation - #2807

feat(GFX1100-TG200): T25 keep ssm_out as Q5_K with runtime input permutation#2807
ghazni101 wants to merge 14 commits into
mudler:mainfrom
ghazni101:row/GFX1100-TG200-T25

ghazni101 commented Sep 3, 2026

Uh oh!

localai-org-maint-bot commented Sep 5, 2026

Uh oh!

ghazni101 commented Sep 5, 2026

Uh oh!

ghazni101 commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ghazni101 commented Sep 3, 2026

Uh oh!

localai-org-maint-bot commented Sep 5, 2026

Uh oh!

ghazni101 commented Sep 5, 2026

Uh oh!

ghazni101 commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants