Skip to content

perf(KERNEL-QUANT-CIQ-GEMM-ROCM): widen the WMMA block to 8 warps, measure, reject - #3033

Open
joral wants to merge 1 commit into
mudler:mainfrom
joral:row/KERNEL-QUANT-CIQ-GEMM-ROCM-WIDEBLOCK
Open

perf(KERNEL-QUANT-CIQ-GEMM-ROCM): widen the WMMA block to 8 warps, measure, reject#3033
joral wants to merge 1 commit into
mudler:mainfrom
joral:row/KERNEL-QUANT-CIQ-GEMM-ROCM-WIDEBLOCK

Conversation

@joral

@joral joral commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Issue #3032's same-tool rocprofv3 trace found KQuantGemmKWmmaQ4K/Q6K
4.9x-10.6x slower per-kernel than llama.cpp's mul_mat_q on the identical
tensor-core mechanism, and llama.cpp's launch uses double the warps per
block and 16-48x fewer, bigger blocks. Row: KERNEL-QUANT-CIQ-GEMM-ROCM
(already ACTIVE); spec update in the same commit.

Each warp in this kernel already owns a fully independent output tile with
its own shared-memory slice, so widening from 4 to 8 warps/block was the
obvious first lever: does packing more of that already-independent work
into fewer, bigger launches close the gap. Templated both kernels on
WarpsPerBlock, added an 8-warp instantiation behind
VT_ROCM_QUANT_WMMA_WIDE=1 (default off).

The answer is no. Op-level A/B (quant-gemm-bench, RX 9060 XT,
best-of-4, idle host), all six Q4_K/Q6_K prefill shapes:

Shape 4-warp (current default) 8-warp ratio
Q4_K N=3072 K=2048 1948.5 GFLOP/s 1827.9 GFLOP/s -6.2%
Q4_K N=12288 K=2048 2242.7 GFLOP/s 2027.9 GFLOP/s -9.6%
Q4_K N=2048 K=6144 1941.3 GFLOP/s 1966.1 GFLOP/s +1.3%
Q6_K N=3072 K=2048 1874.0 GFLOP/s 1775.9 GFLOP/s -5.2%
Q6_K N=12288 K=2048 2159.1 GFLOP/s 1981.4 GFLOP/s -8.2%
Q6_K N=2048 K=6144 1885.0 GFLOP/s 1933.5 GFLOP/s +2.6%

Geomean -4.3%, a net regression. Each warp shares no loaded/dequantized
data with any other warp in its block, so widening only spreads the same
LDS-capped occupancy budget over fewer, bigger blocks rather than doing
more work per warp — the axis this measurement rules out, not the one
explaining llama.cpp's actual advantage. The sharper, still-open
hypothesis — cross-warp data reuse in llama.cpp's own kernel — is recorded
in the spec's ## Owed, unread and unconfirmed, as the next traceable
step.

Gated on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3): ctest -R rocm|cross_device, both configs, 46/46 cases, 84066/84066 assertions,
zero regression (the two remaining preflight failures,
test_tower_skip_rss_arm and the tools suite's test_drop_file_cache,
are pre-existing environment artifacts — a tmpfs /tmp and an unrelated
benchmark-instrumentation self-test — unrelated to this change and
reproduced identically on an unmodified checkout).

The toggle stays in the tree, default off, as a ready-made A/B for
re-checking this specific axis on different hardware (gfx1201) or a future
toolchain revision, matching this file's own VT_ROCM_Q6K_SMALL_PRIVATE
precedent for a measured-and-rejected diagnostic arm, rather than making
the next person re-derive it from scratch.

Closes #3032 with this evidence: the experiment it proposed ran, and the
result is a documented negative, not an open question.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-sonnet-5 [Claude Code]

…asure, reject (mudler#3032)

Issue mudler#3032's same-tool rocprofv3 trace found KQuantGemmKWmmaQ4K/Q6K
4.9x-10.6x slower per-kernel than llama.cpp's mul_mat_q on the identical
tensor-core mechanism, with llama.cpp launching double the warps per
block and 16-48x fewer, bigger blocks. Each warp here already owns a
fully independent output tile with its own shared-memory slice, so
widening the block from 4 to 8 warps was the obvious first lever to
test: does packing more of that already-independent work into fewer,
bigger launches close any of the gap.

Templated both kernels on WarpsPerBlock and added an 8-warp
instantiation behind VT_ROCM_QUANT_WMMA_WIDE=1 (default off, same
same-binary-A/B posture VT_ROCM_QUANT_WMMA and VT_ROCM_Q6K_SMALL_PRIVATE
already ship with). Hardware-verified correct on both configs: ctest -R
rocm|cross_device, 46/46 cases, 84066/84066 assertions, zero regression.

The answer is no. Op-level A/B (quant-gemm-bench, RX 9060 XT, best-of-4,
idle host) across the six Q4_K/Q6_K prefill shapes: geomean -4.3%, a net
regression. Each warp shares no loaded or dequantized data with any
other warp in its block, so widening only spreads the same LDS-capped
occupancy budget over fewer, bigger blocks rather than doing more work
per warp -- the axis this measurement rules out, not the one that
explains llama.cpp's actual advantage. The sharper, still-open
hypothesis this points at -- cross-warp data reuse in llama.cpp's own
kernel -- is recorded in the spec's Owed section, unread and
unconfirmed, as the next traceable step.

The toggle stays in the tree, default off, as a ready-made A/B for
re-checking this specific axis on different hardware or a future
toolchain revision, rather than making the next person re-derive it
from scratch.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-sonnet-5 [Claude Code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ROCm KQuantGemmKWmmaQ4K/Q6K prefill launches 16-48x more, smaller blocks than llama.cpp's mul_mat_q and is 4.9-10.6x slower

1 participant