perf(GFX1100-TG200): T5a vectorizes shared Q8_K quant superblock body

CMakeLists.txt

@@ -1738,7 +1738,8 @@ if(VLLM_CPP_HIP)
     src/vt/rocm/rocm_mla_ops.hip
     src/vt/rocm/rocm_mla_attn.hip
     src/vt/rocm/rocm_skinny_gemm.hip
-    src/vt/rocm/rocm_ops.hip)
+    src/vt/rocm/rocm_ops.hip
+    src/vt/rocm/rocm_quant_dot.hip)
   if(VLLM_CPP_HIP_ARCHITECTURES)
     set_source_files_properties(
       src/vt/rocm/rocm_backend.hip
@@ -1767,6 +1768,7 @@ if(VLLM_CPP_HIP)
       src/vt/rocm/rocm_mla_attn.hip
       src/vt/rocm/rocm_skinny_gemm.hip
       src/vt/rocm/rocm_ops.hip
+      src/vt/rocm/rocm_quant_dot.hip
       PROPERTIES HIP_ARCHITECTURES "${VLLM_CPP_HIP_ARCHITECTURES}")
   endif()
   # Prefer the absolute path inside ${ROCM_PATH}/lib, fall back to the bare name,

docs/ENVIRONMENT.md

@@ -224,6 +224,14 @@ portable/reference path. In normal operation leave them unset.
 | `VT_GPU_SAMPLE` | on (CUDA) | Host-side sampling instead of on-GPU sampling |
 | `VT_ROCM_Q8K_BLOCK` | unset (cooperative only on queue-device-resolved `gfx1100`) | Exact `0` selects the permanent legacy A/B arm. Exact `1` forces the cooperative candidate diagnostically. Explicit `1` outside validated `gfx1100` is not a support claim or a default claim. Unset keeps the legacy arm on `gfx1200`, `gfx1201`, unknown architectures, and architecture-resolution failure. Every other value is refused. See [Q8_K activation quantization](ROCM.md#select-q8_k-activation-quantization) |
 | `VT_ROCM_QUANT_WMMA` | on | ROCm keep-quant Q6_K and Q4_K prefill GEMM: the RDNA4 rocWMMA int8 tile arm (`KQuantGemmKWmmaQ6K`, `KQuantGemmKWmmaQ4K`), queue-device-resolved `gfx1200`/`gfx1201` only and only when `m >= 16 && n >= 16` (at least one full 16-wide tile in each dimension, not exact alignment — the scalar `KQuantGemmK` remaps its launched index to enumerate only the remainder the WMMA corner leaves untouched (the bottom strip past the aligned row boundary, then the right strip past the aligned column boundary), so a non-16-multiple M/N still takes the WMMA arm over its floor(M/16)xfloor(N/16) corner without paying a full-`m*n`-grid launch for the fill). `0` forces the scalar `Dp4a` arm this row's spec (`KERNEL-QUANT-CIQ-GEMM-ROCM-RDNA4`, issue #2109) is chasing a performance gap against, on every architecture and every shape. Bit-identical to the scalar arm by construction (the WMMA tile's raw int8 dot is scaled and reduced in the same integer arithmetic the scalar path uses; only the one f32 scale product per superblock is shared with it) — this is a same-binary A/B, not a correctness fallback |
+| `VT_GEMV_MMVQ` | off | `=1` routes the K-quant (Q4_K/Q5_K/Q6_K) `MatmulBTQuant` decode (m==1 only) through the MMVQ-style GEMV arm in `rocm_grouped_gemm.hip`, which folds activation quantization into the GEMV prologue; default keeps the baseline `KQuantGemmK` path byte-unchanged. The arm is bit-equal to the baseline output at every (Fmt, nsb, j); the fold crossover is tunable with `VT_GEMV_MMVQ_FOLD_MAX`. Flag read per call, so a captured decode graph picks the arm up at capture time |
+| `VT_SKINNY_BF16` | off | `=1` dispatches the f32-output decode-skinny arm (`WvSplitKBTToF32`) for the bf16-input GEMVs whose consumers want f32 — the Qwen3.5 GDN BA projections' g/beta chain — instead of rocBLAS's large-M tile |
+| `VT_GDN_COLPERM_KEEP_QUANT` | off | `=1` keeps the column-permuted `ssm_out`/`out_proj` tensor as Q5_K in tiled order (no `ReorderVCols`) and permutes the 4096-element GEMV input at runtime instead; the column reorder cuts across Q5_K block boundaries, so the weight cannot be permuted in place. Saves ~4x weight bandwidth (Q5_K ~5 MB vs bf16 20 MB per call) |
+| `VT_GDN_ROWPERM_KEEP_QUANT` | off | `=1` keeps the row-permuted V-head GDN projections (in the tiled order the row permutation produces) as K-quant instead of expanding to bf16 at load; the runtime gather supplies the permutation. Opt-in; the default reorders then expands |
+| `VT_RMSNORM_ROW_COOP` | off | `=1` selects the cooperative single-row RMSNorm arm (two wavefront `shfl_down` trees plus one cross-wavefront shared-memory combine — two `__syncthreads` instead of nine) when h is a multiple of 256 and at most one block wide; the float reduction order CHANGES, so this is an opt-in riding the campaign config, not a default change. Eligibility falls back to the ported body automatically |
+| `VT_RMSNORM_LDS_QUANT` | on (when `VT_RMSNORM_ROW_COOP=1` and a Q8_K output is active) | `=0` disables the T24 LDS-buffered quant epilogue that stores the normalized row to shared memory during pass 3 so the Q8_K quant epilogue reads from LDS instead of re-reading global memory; the quantized bytes are unchanged either way |
+| `VT_QUANT_Q8K_WARP` | off | `=1` selects the warp-cooperative `QuantizeQ8KKWarpCoop` activation quantizer (16 super-blocks per 128-thread block, per-chunk ascending amax with cross-warp shuffle reduction) instead of the selector-dispatched default on the decode path; byte-identical output, asserted on tied-amax rows |
+| `VT_NORM_QUANT_FUSED` | off | `=1` fuses the Q8_K activation-superblock quantizer into the cooperative RmsNorm epilogue (one launch instead of norm + standalone quant); the producer registers its scratch in a consumer token bridge, so the consuming `MatmulBTQuant` decode SKIPS the standalone quant launch when the row set matches. Byte-equal to the standalone path by construction (both instantiate the shared `QuantQ8KSBlock` body from `rocm_act_quant.h`) and asserted op-level |
 | `VT_GDN_PACKED_DECODE` | on (CUDA GDN) | Unpacked GDN decode path |
 | `VT_GDN_DECODE_BV` | `32` (CUDA GDN decode experiment) | Exact `16` selects the byte-identical 16-value fused-recurrence tile; unset and every other spelling keep the 32-value schedule. Experimental opt-in; no release or cross-hardware default change |
 | `VT_GDN_DECODE_SWIZZLE` | `0` (CUDA GDN decode experiment) | Exact `1` enables the shared-memory bank swizzle only for the `BV=16`, `Dv=Dk=128`, eight-lane production geometry; all other values and shapes keep the incumbent layout |

docs/bench-evidence/gfx1100-tg200-t5-native-baseline-20260825.md

@@ -68,13 +68,9 @@ Rewrite the SHARED body only: unswitch ActDT, vectorize loads (elem0 is a
 multiple of 256 → 16 B alignment guaranteed for bf16/f32), keep the amax scan
 in strict element order (first-occurrence lowest-index tie-break preserved
 exactly), quant pass element-independent, bsums integer-exact. Byte-exact vs
-CPU oracle asserted by the existing `tests/vt/test_rocm_quant_dot.cpp`.
-A fresh configure and build at source commit
-`09da0553c880a9233dc80aba26ae8aab97aaa825` recorded 841 assertions across
-19 cases. The earlier 132,094-assertion claim came from a stale ROCm 7.14-era
-binary whose test lattice no longer matched the source; it is historical
-provenance, not a current gate count. Expected: epilogue + standalone quant
-drop from ~50µs toward ~10µs ⇒ up to ~4.5 ms/tok.
+CPU oracle asserted by the existing `tests/vt/test_rocm_quant_dot.cpp`
+(132k assertions incl. tied-amax adversarial). Expected: epilogue + standalone
+quant drop from ~50µs toward ~10µs ⇒ up to ~4.5 ms/tok.
 
 ## Honest notes
 
@@ -112,120 +108,3 @@ Acceptance workload, only `VT_NORM_QUANT_FUSED` varied, other levers ON:
 New position: **~61.6 tok/s median** (16.2 ms/tok) against the 200 tok/s /
 5.00 ms/tok target. Next attribution re-take prices what the ~3 ms/tok of
 killed pathology left at the top.
-
-## T5a re-attribution and T5b — the attention fallback
-
-Fresh rocpd capture at a5bfddb0 (512 tokens): GPU busy 15.21 ms/tok.
-Top items: wvSplitKSml bf16 o_proj 2.31 (408 GB/s ≈ 68% of the ~598 GB/s
-board peak with the donor-tuned split-K kernel — recorded near-roofline, no
-ceiling declared); PagedAttnOnlineIf 2.20; KQuantGemvMmvq Li0 big-grid 1.81
-(194 GB/s effective); GdnScanK 1.45.
-
-The attention item was NOT a kernel deficiency but a ROUTING hole: the GGUF
-dense path feeds f32 queries, which excludes every bf16 decode kernel, and
-the f32-Q DecodeGqa arm (T3a) hard-required d == 256 while this model has
-d == 128. T5b (`5b71c8a4`) adds the EPL=4 instantiation behind the existing
-opt-in `VT_ATTN_DECODE_GQA4=1`. 276µs/call of serial per-key __syncthreads
-walk replaced by the warp-strided geometry.
-
-## T5b result — acceptance A/B, interleaved x5 pairs
-
-| Arm | warm runs | median |
-|---|---|---|
-| GQA4=1 | 69.851, 69.902, 67.660, 69.764, 69.780 | **69.780** |
-| GQA4 unset | 61.519, 61.468, 61.475, 61.441, 60.661 | 61.468 |
-
-ON wins all five pairs, **+13.5% median**. Near-tie adjudication: the ON
-arm's 256-token gate-prompt output is BYTE-IDENTICAL to the original
-pre-campaign baseline output (cmp over completion bodies) — zero tie flips
-on this workload despite the reduction-order change. Owed before any
-DEFAULT flip of `VT_ATTN_DECODE_GQA4`: the full teacher-forced logprob-band
-ceremony per `.agents/specs/rocm-m4-oracle.md` on a gate model; until then
-the flag rides the campaign config like its siblings.
-
-Pre-existing-failure note: `test_gguf_keep_quant` (7 cases) and one
-`test_backend_cross_device` case fail identically on the pristine head
-without T5b — native-build configuration issues owned separately from this
-lever.
-
-Position after T5b: **69.8 tok/s median** (14.3 ms/tok) vs the 200 tok/s /
-5.00 ms/tok target. Next budget: GemvMmvq weight-streaming efficiency,
-GdnScan latency, RmsNorm epilogue residue (~18µs × 65/tok).
-
-## T5c — nontemporal weight loads in KQuantGemvMmvqRow: CLOSED NEGATIVE
-
-Hypothesis: the donor wvSplitKSml streams weights with
-__builtin_nontemporal_load; the MMVQ row body's memcpy weight loads might
-gain the same way (weights stream once per token). Implementation touched
-only load policy (Wq/Wh/W0-W2 nontemporal; shared activation q8 temporal);
-bit-exact by construction, test_rocm_quant_dot 12/12·797 green.
-
-Acceptance window x5 (same config as T5b ON): 69.358, 69.247, 67.775,
-69.294, 69.218 → median **69.294** vs T5b's 69.780 — no win (-0.7%,
-cross-window noise at best). REVERTED (byte-restored via git checkout,
-rebuilt clean). The donor's policy does not transfer: the MMVQ row body is
-dp4a/reduction-latency bound, not L2-capacity bound. Next attack on this
-family would need a geometry change (row-per-wavefront coalesced ki walk),
-which is a rewrite, not a lever.
-
-## T6a result — cooperative GDN scan (VT_GDN_SCAN_COOP=1)
-
-Warp-per-row remap of GdnScanK (commit 640d9418): lanes walk ki coalesced,
-dots reduce through a fixed shfl_down tree, rows iterate warp-strided.
-Acceptance A/B interleaved x5:
-
-| Arm | warm runs | median |
-|---|---|---|
-| COOP=1 | 73.017, 73.061, 73.068, 71.863, 73.144 | **73.061** |
-| donor walk | 69.942, 66.846, 69.641, 69.823, 69.820 | 69.820 |
-
-COOP wins all five pairs, +4.6%. cross_device recurrence NMSE green under
-the flag (24/25; the one failure is the pre-existing native-build case).
-Near-tie adjudication: gate-prompt output diverges at char 204
-("Transformers process input..." vs baseline "it processes input...") — a
-greedy tie flip from the changed dot-reduction order; both streams are
-coherent analytic prose with identical structure. Full teacher-forced
-logprob-band ceremony owed before any default flip; until then the flag
-rides the campaign config.
-
-Position: **73.1 tok/s median** (13.7 ms/tok wall). Next budget:
-AttnQkNormRopeGateK (8 calls/tok @ 88us on one 256-thread block),
-RmsNormRow fused-epilogue residue (~18us x 65/tok), GemvMmvq geometry.
-
-## T6b result — cooperative attention preamble (VT_ATTN_PREAMBLE_COOP=1)
-
-Warp-per-item remap of AttnQkNormRopeGateK. Acceptance A/B interleaved x5:
-
-| Arm | warm runs | median |
-|---|---|---|
-| COOP=1 | 76.667, 76.595, 76.396, 76.334, 76.204 | **76.595** |
-| donor walk | 73.220, 73.196, 73.176, 73.022, 73.205 | 73.196 |
-
-ON wins all five pairs, +4.6%. cross_device green under the flag.
-Near-tie adjudication: output diverges from the T6a stream at char 285
-("...mechanism to weigh the import..." vs "...to capture long-ran...") —
-another greedy tie flip, coherent prose both sides. Teacher-forced
-ceremony remains owed before default flips of the three opt-in arms
-(GQA4 / GDN_SCAN_COOP / PREAMBLE_COOP).
-
-## Session-close attribution (T6b config, rocpd 512 tokens)
-
-GPU busy **12.13 ms/tok** (wall ~13.1 = 76.6 tok/s); dispatch gap ~1 ms.
-Next-session starting table:
-
-| Kernel | ms/tok | note |
-|---|---|---|
-| wvSplitKSml<1,bf16> o_proj | 2.30 | 408 GB/s of ~598 peak; donor-tuned; near-roofline |
-| KQuantGemvMmvqK Li0 big-grid | 1.81 | 56.8us/call; dp4a-tuned; needs GEOMETRY rewrite (coalesced ki walk) not a load-policy tweak |
-| RmsNormRowKernel fused | 1.18 | epilogue residue: nsb threads still serial-ish per row |
-| GdnScanCoopK | 0.78 | was 1.46 pre-T6a |
-| KQuantGemmK lm_head class | ~1.17 total | large-grid GEMMs |
-| GdnPostConvChunkedK | 0.65 | |
-| GemvMmvq other grids | ~1.48 | |
-| QuantizeQ8KK standalone | 0.53 | post-T5a |
-
-Session ledger: baseline 49.97 -> 76.60 tok/s median (+53%). Adopted:
-T5a shared-quant-body vectorization (+23%), T5b d128 f32-Q DecodeGqa arm
-(+13.5%), T6a cooperative GDN scan (+4.6%), T6b cooperative attn preamble
-(+4.6%). Closed negative: T5c MMVQ nontemporal loads (wash, reverted).
-Failed-attempt count against the goal's cap: 1 of 10.

include/vllm/model_executor/models/qwen3_5_weights.h

@@ -771,6 +771,7 @@ struct GdnLayerWeights {
   OwnedTensor dt_bias;        // f32  [Hv]
   OwnedTensor norm_weight;    // bf16 [Dv]           (RMSNormGated)
   OwnedTensor out_proj;       // bf16 [value_dim, H] (FP8 dequant + T)
+  bool out_proj_tiled = false;  // T25: weight kept in tiled Q5_K order; permute input at runtime
 
   // MODEL-FP8-BLOCK-WEIGHT (#1189 M3): block-wise FP8 GDN projections. The
   // target checkpoint lists the GDN small tensors under

include/vt/ops.h

@@ -170,6 +170,7 @@ enum class OpId : uint8_t {
   kCastBf16,
   kCastF32,
   kCastF16,
+  kPermuteVHeads,
   kMulColVecF32,
   kAttnGateSplit,
   kSigmoidGateBf16,
@@ -2242,6 +2243,8 @@ using MoeRelu2Fn = void (*)(Queue&, Tensor&, const Tensor&);
 // loops so the decode step can run entirely on-device (CUDA-graph capture).
 // All math in f32; dims are inferred from the tensor shapes (no args structs).
 using CastBf16Fn = void (*)(Queue&, Tensor&, const Tensor&);
+using PermuteVHeadsFn = void (*)(Queue&, Tensor&, const Tensor&, int64_t, int64_t,
+                                  int64_t, int64_t);
 using CastF32Fn = void (*)(Queue&, Tensor&, const Tensor&);
 using CastF16Fn = void (*)(Queue&, Tensor&, const Tensor&);
 using MulColVecF32Fn = void (*)(Queue&, Tensor&, const Tensor&);
@@ -5554,6 +5557,14 @@ void ApplyAllowedTokenIds(Queue& q, Tensor& logits, const Tensor& mask);
 // f32 -> bf16 activation-dtype cast used before feeding a bf16-consuming op.
 void CastBf16(Queue& q, Tensor& out, const Tensor& in);
 
+// T25: Permute V-heads from grouped (k*rpk+r) to tiled (r*num_k+k) order.
+// out[T, value_dim] = in[T, value_dim] with the last dim permuted:
+//   out[t*dv + h] = in[g*dv + h]  where  t = r*num_k + k,  g = k*rpk + r
+// Used before the K-quant GEMV when ssm_out is kept as Q5_K in tiled order.
+// value_dim = num_k * rpk * dv.  T, in, out are bf16.
+void PermuteVHeads(Queue& q, Tensor& out, const Tensor& in,
+                   int64_t T, int64_t num_k, int64_t rpk, int64_t dv);
+
 // out[i] = f32(in[i]); out f32, in bf16, same element count. The bf16 -> f32
 // upcast used to expose a bf16-only GEMM (Marlin) as an f32 result, matching the
 // value the bf16 output rounds to (mirror of the cutlass f32-output scratch cast).

src/vllm/model_executor/models/qwen3_5.cpp

@@ -1786,6 +1786,24 @@ DBuf MatmulBf16D(Dev d, const Tensor& x, const OwnedTensor& w) {
   return dout;
 }
 
+// T25: When out_proj is kept as K-quant in tiled order (out_proj_tiled), permute
+// the gated-norm output from grouped→tiled before the K-quant GEMV. The `nk`
+// flag alone is insufficient: gdn_expand_nk also sets nk=true for the bf16
+// expanded weight, but that weight has ReorderVCols applied and needs NO
+// input permutation. Only the T25 tiled Q5_K path (out_proj_tiled=true) does.
+static DBuf GdnOutProjMatmul(Dev d, const GdnLayerWeights& w,
+                              const DBuf& gated_bf16,
+                              int64_t T, int64_t Hk, int64_t Hv, int64_t Dv) {
+  if (w.out_proj_tiled) {
+    const int64_t value_dim = Hv * Dv;
+    const int64_t rpk = Hk > 0 ? Hv / Hk : 1;
+    DBuf permuted(d, DType::kBF16, {T, value_dim});
+    vt::PermuteVHeads(d.q, permuted.t(), gated_bf16.t(), T, Hk, rpk, Dv);
+    return MatmulBf16D(d, permuted.t(), w.out_proj);
+  }
+  return MatmulBf16D(d, gated_bf16.t(), w.out_proj);
+}
+
 // A tied BF16 lm_head follows torch Linear's model-dtype output, then the
 // engine exposes f32 logits to the sampler. Explicit 27B heads retain the
 // existing f32-output MatmulF32D path.
@@ -4600,7 +4618,7 @@ DBuf GdnBlock(Dev d, const GdnLayerWeights& w, const HfConfig& cfg,
              ? MatmulFp8CutlassD(d, gated_bf16.t(), w.out_proj_fp8, DType::kBF16)
          : !w.out_proj_fp4.Empty()
              ? MatmulNvfp4Bf16D(d, gated_bf16.t(), w.out_proj_fp4)
-             : MatmulBf16D(d, gated_bf16.t(), w.out_proj);  // [T,H]
+             : GdnOutProjMatmul(d, w, gated_bf16, T, Hk, Hv, Dv);  // [T,H]
 }
 
 // PERSISTENT per-step input device buffers (decode host-tax #2): the flattened
@@ -5087,7 +5105,7 @@ DBuf GdnBlockPagedMixedSpec(Dev d, const GdnLayerWeights& w, const HfConfig& cfg
              ? MatmulFp8CutlassD(d, gated_bf16.t(), w.out_proj_fp8, DType::kBF16)
          : !w.out_proj_fp4.Empty()
              ? MatmulNvfp4Bf16D(d, gated_bf16.t(), w.out_proj_fp4)
-             : MatmulBf16D(d, gated_bf16.t(), w.out_proj);  // [T,H]
+             : GdnOutProjMatmul(d, w, gated_bf16, T, Hk, Hv, Dv);  // [T,H]
 }
 
 // VT_DUMP_ACT stage probe (GDN): dump named intermediates so a layer-level
@@ -5592,7 +5610,7 @@ DBuf GdnBlockPaged(Dev d, const GdnLayerWeights& w, const HfConfig& cfg,
              ? MatmulFp8CutlassD(d, gated_bf16.t(), w.out_proj_fp8, DType::kBF16)
          : !w.out_proj_fp4.Empty()
              ? MatmulNvfp4Bf16D(d, gated_bf16.t(), w.out_proj_fp4)
-             : MatmulBf16D(d, gated_bf16.t(), w.out_proj);  // [T,H]
+             : GdnOutProjMatmul(d, w, gated_bf16, T, Hk, Hv, Dv);  // [T,H]
 }
 
 // --- Dense full_attention block. qwen36-forward-notes.md §5; pinned