record(GFX1100-TG200): small-N GEMV geometry levers closed negative (VT_GEMV_WARPS, VT_GEMV_SPLITK)

.agents/specs/gfx1100-tg200.md

@@ -206,6 +206,25 @@ gap: HIP graph capture (T2), kernel fusion, or persistent kernels. A fresh
 rocprofv3 attribution capture with dispatch counts per token is the next
 step to price the overhead precisely.
 
+
+T37 outcome (2026-08-29, attempt #5 of the small-N GEMV budget,
+[evidence](../../docs/bench-evidence/gfx1100-tg200-t37-gemv-warps-20260829.md)):
+closed negative with numbers. Two new angles implemented default-OFF:
+VT_GEMV_WARPS (1..8 warps/block — the untested direction; T19/T32 tested
+MORE warps) is bit-identical and wins every microbench shape at 4
+warps/block (1.09-1.64x, gate_up 96.9->64.1 us/kernel by rocprof), but
+the engine A/B is a WASH (base 86.073 vs ON 85.641 median, byte-identical
+bodies): the microbench re-reads one warm L2-resident weight tensor while
+the engine cold-streams a different layer's weights every launch, where
+block distribution does not change the controller-limited rate.
+VT_GEMV_SPLITK (deterministic split-K + ascending-order reduce) wins only
+ssm_out in the bench and pays a reduce launch — parked default-OFF,
+adjudication owed if ever enabled. Five-of-five small-N GEMV geometry
+attempts now closed (T19, T32 x2, T37 warps, T37 splitk): geometry
+micro-tuning cannot move these launches; cold-DRAM streaming + in-graph
+gaps co-dominate. Remaining traceable paths: launch-count reduction or
+accepting the position.
+
 Owed before ANY default flip of the opt-in arms (GQA4 / GDN_SCAN_COOP /
 GDN_SCAN_SPLIT / PREAMBLE_COOP / RMSNORM_ROW_COOP / GDN_NORMGATED_COOP /
 GDN_POSTCONV_COOP): teacher-forced logprob-band ceremony per

CMakeLists.txt

@@ -1735,7 +1735,8 @@ if(VLLM_CPP_HIP)
     src/vt/rocm/rocm_mla_fused_norm_rope.hip
     src/vt/rocm/rocm_mla_ops.hip
     src/vt/rocm/rocm_skinny_gemm.hip
-    src/vt/rocm/rocm_ops.hip)
+    src/vt/rocm/rocm_ops.hip
+    src/vt/rocm/rocm_quant_dot.hip)
   if(VLLM_CPP_HIP_ARCHITECTURES)
     set_source_files_properties(
       src/vt/rocm/rocm_backend.hip
@@ -1762,6 +1763,7 @@ if(VLLM_CPP_HIP)
       src/vt/rocm/rocm_mla_ops.hip
       src/vt/rocm/rocm_skinny_gemm.hip
       src/vt/rocm/rocm_ops.hip
+      src/vt/rocm/rocm_quant_dot.hip
       PROPERTIES HIP_ARCHITECTURES "${VLLM_CPP_HIP_ARCHITECTURES}")
   endif()
   # Prefer the absolute path inside ${ROCM_PATH}/lib, fall back to the bare name,

docs/bench-evidence/gfx1100-tg200-t37-gemv-warps-20260829.md

@@ -0,0 +1,89 @@
+# GFX1100-TG200 T37 — small-N GEMV bandwidth, attempt #5 (VT_GEMV_WARPS / VT_GEMV_SPLITK): microbench wins, engine wash, closed (2026-08-29)
+
+## Question
+
+Fifth attack on the small-N decode GEMV bandwidth budget (T34: gate_up 57%,
+attn_output-Q6K 26%, ssm_out 54%, attn_gate 44%, lm_head 99% do-not-touch).
+Prior washes: T19 kGemvWarps 8→16 (block-limited), T32 uint4 wider loads
+(compiler already coalesces), T32 more warps. New angles this round:
+block-geometry in the UNTESTED direction (fewer warps per block = more
+blocks), and a deterministic split-K decomposition.
+
+## Wave/occupancy math (the CPU-first work)
+
+256-thread blocks (8 warps); n=2560 → 320 blocks vs ~768 concurrent-block
+capacity — the MMVQ shapes issue only ~42-67% of the device's warp slots
+(ssm_out 2560 warps vs 3840 wave32 slots). attn_output-Q6K (coop arm) issues
+2560 blocks = 5.3 fills — NOT underfilled; its 26% BW is per-block
+latency/ramp, not occupancy. gate_up issues 2304 blocks = 3 exact fills.
+
+## Levers implemented (default OFF, read per call)
+
+- `VT_GEMV_WARPS=1..8` (default 8): warps per block in the MMVQ launch —
+  pure launch geometry, per-row math unchanged, outputs BIT-IDENTICAL.
+- `VT_GEMV_SPLITK=2..32` (default 1 = off): each row's super-blocks split
+  across S warps writing S float partials to the graph-safe quant scratch;
+  a reduce kernel folds them in ascending split order (deterministic, no
+  data atomics; KQuantGemvMmvqSplitKK + KQuantGemvSplitKReduceK; the row
+  body gained range/partial parameters with defaults reproducing the old
+  body exactly). Reduction-order change — owes near-tie adjudication before
+  any engine A/B.
+
+## Microbench (tests/vt/t37_gemv_bench.cpp, executable-only; free)
+
+Through the REAL dispatch, 10 warm + 50 timed (mean us), bit-compare vs
+baseline:
+
+| shape | base | warps2 | warps4 | splitk2 | splitk4 | splitk8 |
+|---|---|---|---|---|---|---|
+| ssm_out_q5 2560×4096 | 48.1 | 0.99x | 1.09x | 1.53x | **1.62x** | 1.54x |
+| attn_gate_q4 4096×2560 | 28.4 | 1.14x | **1.24x** | 0.95x | 0.91x | 0.71x |
+| ffn_down_q4 2560×9216 | 71.8 | 1.01x | 1.18x | 1.05x | **1.24x** | 1.15x |
+| ffn_down_q6 2560×9216 | 62.6 | 1.05x | **1.27x** | 1.13x | 1.23x | 1.19x |
+| gate_up_q4 18432×2560 | 117.6 | 1.37x | **1.64x** | 1.43x | 1.22x | 0.84x |
+
+warps4: faster everywhere, mism=0 (bit-exact, no adjudication needed).
+splitk4: wins only ssm_out, adds a reduce launch (~1.9 us + a gap).
+
+rocprofv3 kernel-trace attribution of the bench (kernel means):
+gate_up baseline 96.9 us → warps4 64.1 us; ssm_out 42.9 → 33.4 (warps4) /
+29.9+1.9 reduce (splitk4); attn_gate 17.0 → 14.5. Raw:
+agent-artifacts/tg200-t36/results/t37attribution_kernel_trace.csv.
+
+## Engine A/B (idle window: loadavg 0.42 at start, 1.20 at end; 1 warm + 5 reps; gpu-ctl held)
+
+| arm | tok/s (reps 1..5) | median | body md5 |
+|---|---|---|---|
+| base (knob unset) | 86.177 86.139 86.073 83.527 85.871 | **86.073** | a0fa1c4a… |
+| VT_GEMV_WARPS=4 | 85.378 85.641 85.757 85.801 85.634 | **85.641** (−0.5%) | a0fa1c4a… |
+
+Bit-identical bodies as designed (no adjudication required — and none
+would have helped: the verdict is a wash, not a coherence question).
+
+## Why the 1.2-1.6x microbench win died in the engine
+
+The bench launches the SAME weight tensor 50× back-to-back: after the first
+iteration the weights are L2/Infinity-Cache resident, so the measured
+baseline is an L2-latency regime where block distribution matters. The
+engine reads a DIFFERENT layer's weights on every launch (cold DRAM
+streaming, already controller-limited) and replays through the graph where
+~4.1 us inter-kernel gaps co-dominate. The trace's own numbers say the same
+thing: the engine's real gate_up launch is ~52 us (vs 96.9 in the warm
+bench) — the warm-bench baseline was inflated, and with it the headroom.
+
+## Disposition
+
+**T37 attempt #5 closed negative with numbers.** Engine wash (−0.5%,
+overlapping distributions). Both knobs ship default-OFF and inert
+(`VT_GEMV_WARPS` unset → 8; `VT_GEMV_SPLITK` unset → 1; both byte-identical
+to the pre-T37 dispatch). Five-of-five small-N GEMV geometry attempts are
+now closed (T19, T32×2, T37 warps, T37 splitk-parked): the measured
+conclusion is that kernel-geometry micro-tuning cannot move these launches
+on this workload; the cold-DRAM streaming rate and the in-graph launch gaps
+co-dominate. Remaining traceable paths: launch-count reduction (fusing
+whole GEMVs — T35-r3 class) or accepting the position.
+
+Gates on the committed tree: test_rocm_quant_dot assertions 841/841 (both
+with and without VT_GEMV_WARPS=4), test_rocm_prefill_tile 720/720,
+ctest -R 'rocm|quant' at the HEAD baseline result set (2 pre-existing reds
+unchanged).