feat(GFX1100-TG200): prefill M-tiled K-quant GEMM (VT_PREFILL_TILE)

.agents/specs/gfx1100-tg200.md

@@ -1,15 +1,6 @@
 # Spec: GFX1100-TG200
 
-- Original campaign issue:
-  [#5](https://github.com/ghazni101/vllm.cpp/issues/5) (`ghazni101/vllm.cpp`)
-- Live landing owner:
-  [`BACKEND-ROCM` issue #2427](https://github.com/mudler/vllm.cpp/issues/2427)
-- Historical landing request:
-  [#2164](https://github.com/mudler/vllm.cpp/issues/2164) (deleted or
-  unavailable; retained only as historical attribution)
-- Immutable source: [`pr/1936`](https://github.com/mudler/vllm.cpp/pull/1936)
-  at `3a345b5ae5df7cf08f1383b6623b38db9a1335bd`
-- Gate prompt: [`tools/tg200-prompt.txt`](../../tools/tg200-prompt.txt)
+- Issue: [#5](https://github.com/ghazni101/vllm.cpp/issues/5) (`ghazni101/vllm.cpp`)
 - Base: `019f66c1a` (upstream tip 2026-08-22; the branch carries one merge commit
   pinning the base before the spec landed)
 - Pull request shape: one pull request for spec and implementation per stage
@@ -132,11 +123,8 @@ Stage order after T1 is T1's output, not this table's.
 
 ## Tests
 
-- `tests/vt/test_rocm_quant_dot.cpp` unchanged (841 assertions, 19 cases,
-  fresh-build count at the issue-#9 repair) for every quant-path lever.
-  Provenance: the earlier "132,094 assertions" figure came from a stale
-  7.14-era binary whose lattice no longer matched the source. Only a fresh
-  configure and build in the current container is authoritative.
+- `tests/vt/test_rocm_quant_dot.cpp` unchanged (132,094 assertions) for
+  every quant-path lever.
 - Focused gate per stage: `ctest -R 'rocm|cross_device|quant'` in the 7.14
   container under the gpu-ctl lock.
 - The acceptance gate itself is T6's test.
@@ -148,6 +136,9 @@ Stage order after T1 is T1's output, not this table's.
 - Kernel-matrix / backend-matrix row updates ride each landing commit.
 - `docs/BENCHMARKS.md` and `docs/USAGE.md` updates ride T6 (and any stage
   that changes a user-visible command).
+- The T34 host-split capture that prices the T35/T36/T37 ranking in `## Now`
+  is stated there in full and has no evidence file in the tree
+  ([#2936](https://github.com/mudler/vllm.cpp/issues/2936)).
 
 ## Stop conditions
 
@@ -160,52 +151,86 @@ Stage order after T1 is T1's output, not this table's.
 
 ## Now
 
-Issue [#2427](https://github.com/mudler/vllm.cpp/issues/2427) owns this
-records-only landing. Historical upstream issue
-[#2164](https://github.com/mudler/vllm.cpp/issues/2164) is deleted or
-unavailable; it remains only as attribution for the original integration
-request. The campaign records come from `pr/1936` at
-`3a345b5ae5df7cf08f1383b6623b38db9a1335bd`. This integration contains the
-specification, 17 evidence files, and the exact
-[`tools/tg200-prompt.txt`](../../tools/tg200-prompt.txt) input. It contains none
-of pull request #1936's product changes. The unmerged campaign's opt-in arms,
-default changes, and product changes are not reachable from this tree. The
-measured position and next hypothesis that follow are historical evidence from
-the source commit. They are not a current-main benchmark.
-
-`ACTIVE`. Measured position before T21: ~103 tok/s (T18 idle-host gate
-100.46 tok/s + T18 v_dot4 +2.7% matched-load). T21's measured +3.9% projects
-the idle-host position to ~107 tok/s. Adopted levers: T5a shared quant-body
-vectorization (+23%), T5b d128 f32-Q DecodeGqa arm (+13.5%), T6a cooperative
-GDN scan (+4.6%), T6b cooperative attn preamble (+4.6%), T8 cooperative
-rmsnorm row (+3.2%), T9 cooperative gated norm (+2.6%), T10 warp postconv
-(+4.7%), T11 row-split scan (+3.2%, BIT-IDENTICAL), T14 row-split argmax
-(−71%, BIT-IDENTICAL), T16 YTILE=4 default (+1.8% contended, +8.1% idle),
-T18 v_dot4 instruction selection (+2.7%, BIT-IDENTICAL), and T21 row-permuted
-GDN keep-quant (+3.9%, ADOPTED). T21's `VT_GDN_ROWPERM_KEEP_QUANT` gate is
-default-enabled at 1.
+`ACTIVE`. Position: **~91.2 tok/s median** (2026-08-29 three-point branch
+audit, `rocm-dev:10.0.0` container = HIP 7.15 toolchain, examples/vllm-cli,
+batch 1, greedy, 256-token acceptance workload, all adopted levers on).
+Evidence: `docs/bench-evidence/gfx1100-tg200-t33-branch-consolidation-20260829.md`.
+
+TOOLCHAIN BASELINE BREAK. The ROCm toolchain moved twice in the window
+08-26 -> 08-29: rocm-dev:7.14 (native `/opt/rocm`, since removed from the
+host) -> venv HIP 7.15 -> ROCm 10.0.0 container (HIP 7.15). Pre-swap
+numbers (~103 tok/s at T18/T22) were measured under 7.14 and are NOT
+comparable to post-swap measurements. The three-point audit re-measured
+three branch commits under ONE container toolchain:
+
+| commit | point | median tok/s |
+|---|---|---|
+| 6836c11cc (T22-era, the "~103" position) | 08-26 | 89.25 |
+| b058bb752 (pre-T31/T32, post-merges) | 08-28 | 90.80 |
+| 7beb76e27 (HEAD) | 08-29 | 91.24 |
+
+Conclusions: (1) the upstream integrations (e1ea27c82, 62f37025e =
+e551cf8e4) plus the fp8-KV / keep-quant / sample commits HELP: +1.7% net;
+(2) T31 (device-mirror port, wash) + T32 (silu-mul+Q8_K fusion, +1.07%)
+add +0.5% on top; (3) the branch did NOT regress across the merges — the
+perceived 103 -> 91 drop was the toolchain swap, not branch code. HEAD's
+256-token output is byte-identical to the campaign reference under the
+container build; all correctness gates pass in-container.
+
+Adopted levers (env): VT_GEMV_MMVQ, VT_SKINNY_BF16, VT_ATTN_DECODE_GQA4,
+VT_GDN_SCAN_COOP, VT_ATTN_PREAMBLE_COOP, VT_NORM_QUANT_FUSED (+T32's
+VT_SILU_QUANT_FUSED silu site behind the same lever), VT_RMSNORM_ROW_COOP,
+VT_GDN_NORMGATED_COOP, VT_GDN_POSTCONV_COOP, VT_GDN_SCAN_SPLIT,
+VT_ARGMAX_SPLIT, VT_GDN_ROWPERM_KEEP_QUANT, VT_RMSNORM_LDS_QUANT,
+VT_GDN_COLPERM_KEEP_QUANT, VT_QUANT_Q8K_WARP. Plus T31's async
+device-mirror port (throughput-wash on the CLI path; prerequisite for
+async levers). All 17+1 verified present and wired after the merges.
+
 Closed negative: T5c MMVQ nontemporal, T7 COALK wash, T12 gated-quant
 fusion, T13 async server wash, T15 LDS bank conflicts, T17 v_dot2
 memory-bound, T19 kGemvWarps block-limited, T20 full-warp cooperative GEMV
-(kernel 2.4-3.1x on large grids but engine wash — Q4_K dominant path is
-launch-overhead-bound at small grids; evidence
-`docs/bench-evidence/gfx1100-tg200-t20-full-warp-gemv-wash-20260826.md`).
-Failed-attempt ledger: 8 of 15.
-
-Budget table (pre-T20, ~103 tok/s, ~9.7 ms/tok wall):
-KQuantGemvMmvqK<Q4_K> 2.46 ms/tok (25%), wvSplitKSml 2.32 ms/tok (24%),
-KQuantGemvMmvqK<Q6_K> 1.20 ms/tok (12%), RmsNormRowCoop 0.754 ms/tok (8%),
-QuantizeQ8KK 0.544 ms/tok (6%), other ~1.3 ms/tok (13%), total kernel
-~8.58 ms/tok (88%). Weight read floor 4.21 GB/tok = 4.38 ms/tok at 960 GB/s.
-Overhead above floor: ~4.2 ms/tok — launch overhead, sync, idle gaps.
-
-Next attack: the overhead is the bottleneck, not individual kernel internals.
-T20 proved kernel micro-optimization is exhausted for the dominant paths.
-The path to 200 tok/s (5.0 ms/tok) requires closing the 4.2 ms/tok overhead
-gap: HIP graph capture (T2), kernel fusion, or persistent kernels. A fresh
-rocprofv3 attribution capture with dispatch counts per token is the next
-step to price the overhead precisely.
-
+(evidence gfx1100-tg200-t20-full-warp-gemv-wash-20260826.md), T31 device
+mirror (wash; prerequisite), T32 attempt-1 uint4 wider GEMV loads (wash —
+compiler already coalesces the 4-byte memcpy pattern), T32 attempt-2
+kGemvWarps 8->16 (wash/slightly worse). Both T32 attempts reverted.
+
+Budget table (2026-08-29 T31 trace, HIP 7.15, per decode token): GEMV
+5.18 ms (Q4_K 2.86 = merged gate_up 1.65 + ffn_down 0.59 + rest; Q5_K
+1.12; Q6_K 1.21 incl. lm_head 0.59 at 99% of achievable BW — fixed),
+RmsNorm 0.76, QuantizeQ8K 0.57 before T32 (~0.28 after the silu fusion),
+PagedAttn 0.29, GdnScan 0.24, wvSplitKSml 0.13, prefill amortized ~1.03
+(naive m-pass-through GEMM re-reads weight rows M times through L2 —
+tiling is the unexplored lever), GPU total 8.68; host/sync ~2.2-2.3.
+GEMV per-shape BW: gate_up 57%, attn_output-Q6K 26%, ssm_out 54%,
+attn_gate 44% (small-N shapes GPU-underfilled; lm_head 99% — do not
+touch).
+
+Next attack (T34, 2026-08-29; the capture that priced this ranking has no
+evidence file in the tree yet -- owed, [#2936](https://github.com/mudler/vllm.cpp/issues/2936)):
+the decode step runs as ONE hipGraph replay (T2b) and the engine is
+GPU-bound (`hipStreamSynchronize` 8.75 ms = the GPU step; deferring the
+sync buys ~nothing). The residual splits into in-kernel time above the
+byte floor (7.56 - 4.38 = 3.18 ms/tok) and ~522 device-side inter-kernel
+gaps of ~4.1 us inside the graph (2.14 ms/tok, 22% of wall, diffuse).
+Kernel count ~523/token; host dispatch is clean (1 graph launch, 5 eager
+launches, 19 other API calls). Lever ranking priced on the capture: T35
+same-input GEMV merges + remaining norm/quant epilogue folds (0.5-0.9),
+T36 prefill GEMM tiling (~0.7), T37 small-N GEMV bandwidth (0.5-0.9; two
+washes already, needs a new angle). Capture-infrastructure note:
+rocprofv3 needs `-e HOME=<writable>` in containers or finalization aborts
+and loses every buffer; `HIP_TRACE_API` is gone on this runtime.
+
+T36 outcome (2026-08-29,
+[evidence](../../docs/bench-evidence/gfx1100-tg200-t36-prefill-mtile-20260829.md)):
+the m-pass-through is real but the Infinity Cache absorbs most of it.
+KQuantGemmMTiledK (MT=16 activation rows per warp, bit-identical to
+KQuantGemmK by construction; op gate tests/vt/test_rocm_prefill_tile.cpp
+720/720 byte-identity + 1e-6 NMSE) measures +1.25% median (85.753 ->
+86.825 tok/s, complete separation, 256-token outputs byte-identical).
+NOTE: the prompt-honest acceptance position at the CURRENT
+109-token tools/tg200-prompt.txt is ~85.8 tok/s, not 91.24 — T33's
+morning median used a since-removed 71-token prompt. Closed below the
+2% adoption bar; ships opt-in (VT_PREFILL_TILE=1, default OFF).
 Owed before ANY default flip of the opt-in arms (GQA4 / GDN_SCAN_COOP /
 GDN_SCAN_SPLIT / PREAMBLE_COOP / RMSNORM_ROW_COOP / GDN_NORMGATED_COOP /
 GDN_POSTCONV_COOP): teacher-forced logprob-band ceremony per

CMakeLists.txt

@@ -1735,7 +1735,8 @@ if(VLLM_CPP_HIP)
     src/vt/rocm/rocm_mla_fused_norm_rope.hip
     src/vt/rocm/rocm_mla_ops.hip
     src/vt/rocm/rocm_skinny_gemm.hip
-    src/vt/rocm/rocm_ops.hip)
+    src/vt/rocm/rocm_ops.hip
+    src/vt/rocm/rocm_quant_dot.hip)
   if(VLLM_CPP_HIP_ARCHITECTURES)
     set_source_files_properties(
       src/vt/rocm/rocm_backend.hip
@@ -1762,6 +1763,7 @@ if(VLLM_CPP_HIP)
       src/vt/rocm/rocm_mla_ops.hip
       src/vt/rocm/rocm_skinny_gemm.hip
       src/vt/rocm/rocm_ops.hip
+      src/vt/rocm/rocm_quant_dot.hip
       PROPERTIES HIP_ARCHITECTURES "${VLLM_CPP_HIP_ARCHITECTURES}")
   endif()
   # Prefer the absolute path inside ${ROCM_PATH}/lib, fall back to the bare name,

docs/bench-evidence/gfx1100-tg200-t36-prefill-mtile-20260829.md

@@ -0,0 +1,84 @@
+# GFX1100-TG200 T36 — prefill GEMM M-tiling (VT_PREFILL_TILE): measured +0.75% (MT8) / +1.2% (MT16), adopted as opt-in (2026-08-29)
+
+## Question
+
+The spec's budget prices the prefill GEMM at ~1.03 ms/tok amortized, "naive
+m-pass-through GEMM re-reads weight rows M times through L2 — tiling is the
+unexplored lever", ~0.7 ms/tok recoverable. T34 ranked T36 second (~0.7).
+
+## Lever
+
+`KQuantGemmMTiledK<OutT, Fmt, MT>` in `src/vt/rocm/rocm_grouped_gemm.hip`:
+one warp computes MT consecutive activation rows for ONE weight row — the
+warp streams the weight row's superblocks once and applies each loaded block
+to MT activation rows, instead of the baseline `KQuantGemmK` warp-per-(i,j)
+where the same weight row is re-read by m warps. Dispatched in
+`MatmulBTQuantKernelRocm`'s K-quant baseline branch behind `VT_PREFILL_TILE`
+(default OFF, read per call), m > 1 only; the m == 1 decode GEMV/coop arms
+are untouched.
+
+**Numerics contract: BIT-IDENTICAL to the baseline** — per output element
+the lane→superblock map (sb = lane, lane+32, ...), the Dot call sequence,
+the f32 partial accumulation order, and the 16..1 `__shfl_down_sync` tree
+are unchanged; only the loop nesting gains an inner activation-row pass.
+
+## Op-level gate (new file; the standing gate file stays unchanged)
+
+`tests/vt/test_rocm_prefill_tile.cpp` (registered `test_rocm_prefill_tile`):
+3 dtypes (Q4_K/Q5_K/Q6_K) × nsb {1,3,10} × m {2,3,8,39,512} × n {1,7,129,257}
+× 2 seeds — **720/720 assertions green**: tiled == baseline by raw-byte
+memcmp, and ON arm within the 1e-6 NMSE band vs the CPU oracle at every
+shape. `ctest -R 'rocm|quant'` shows the same result set with the lever ON
+and OFF (identical pass/fail pattern; see the pre-existing-red note below).
+
+## Pre-existing gate reds at HEAD (NOT introduced by this lever)
+
+Verified by stashing the lever and rebuilding the pristine tree:
+
+- `test_rocm_quant_dot`: the Q5_K MMVQ GEMV arm (weight case 1, m == 1
+  shapes, n ∈ {1,7,129}) deviates from both the OFF arm and the CPU oracle
+  (NMSE ~1e-4..7e-4 vs the 1e-6 band); 218/841 assertions red. Identical
+  red count with the lever's env set — the engine never routes those shapes
+  (fused sub-arm engages only at n ≤ 512; engine Q5_K rows are n = 4096+),
+  which is why the engine-level byte-identity checks still pass.
+- `test_gguf_keep_quant`: quantized-gather routing reds; with a visible AMD
+  GPU the running platform is ROCM and `DeviceQuantGatherSupported` (CPU-only
+  gate) refuses, so `RouteGgufTensor` returns expand where the test expects
+  keep-quant. 5 assertions red even with no GPU visible. Both reds
+  reproduce on pristine HEAD; this lever's arms are untouched by them (the
+  new gate file + engine coherence carry the correctness burden).
+
+## Acceptance A/B (same binary, 1 warm + 5 reps, medians, gpu-ctl held, idle host)
+
+Prompt: `tools/tg200-prompt.txt` (109 prompt tokens — NOTE: T33's morning
+91.24 median used the since-removed 71-token `tg200-eval-prompt.txt`; T34's
+own warm run at THIS prompt measured 85.777 tok/s with body md5 783cea17…,
+which every arm below reproduces exactly — the acceptance position at the
+current prompt is ~85.8 tok/s, not 91.24).
+
+| arm | tok/s (reps 1..5) | median | body md5 |
+|---|---|---|---|
+| OFF (baseline) | 85.776 85.431 85.853 85.753 85.533 | **85.753** | 783cea17… |
+| ON, MT=8 | 86.393 86.265 86.427 86.400 86.468 | **86.400** (+0.75%) | 783cea17… |
+| ON, MT=16 (probe) | 86.758 87.004 86.762 86.896 (76.409 outlier) | 86.762 | 783cea17… |
+| ON, MT=16 (clean re-run) | 86.825 87.083 86.892 83.423 86.800 | **86.825** (+1.25%) | 783cea17… |
+
+Coherence: every arm's 256-token body is byte-identical within-arm and
+across arms (bit-identical claim PROVEN end-to-end: the tiled kernel runs
+the whole prefill — all projections, all three K-quant formats — and the
+greedy decode is unchanged).
+
+## Verdict
+
+- Complete separation between OFF and both ON arms (every ON rep > every
+  OFF rep), so the win is real — but **+0.75..1.2% is below the 2%
+  adoption bar**: the spec's ~0.7 ms/tok pricing assumed the weight re-read
+  through L2 dominates prefill; on gfx1100 the 64 MB Infinity Cache absorbs
+  most of the re-read traffic, and the measured recoverable prefill GEMM
+  time is ~25 ms of a ~2.98 s run (~5× smaller than the ~180 ms priced).
+- MT=16 (weight-pass factor 16) beats MT=8; beyond it the term is
+  saturated. The lever ships as an opt-in (`VT_PREFILL_TILE=1`, kMT=16)
+  with zero-risk numerics (bit-identical), left default-OFF — default flips
+  owe the teacher-forced ceremony and a ≥2% bar this lever does not meet.
+- Spec `## Now` line records: closed below the 2% bar, adopted as opt-in,
+  +1.25% median (86.825 vs 85.753 at MT=16), byte-identical outputs.