perf(KERNEL-QUANT-CIQ-GEMM-ROCM): widen the WMMA block to 8 warps, measure, reject - #3033
Open
joral wants to merge 1 commit into
Open
perf(KERNEL-QUANT-CIQ-GEMM-ROCM): widen the WMMA block to 8 warps, measure, reject#3033joral wants to merge 1 commit into
joral wants to merge 1 commit into
Conversation
…asure, reject (mudler#3032) Issue mudler#3032's same-tool rocprofv3 trace found KQuantGemmKWmmaQ4K/Q6K 4.9x-10.6x slower per-kernel than llama.cpp's mul_mat_q on the identical tensor-core mechanism, with llama.cpp launching double the warps per block and 16-48x fewer, bigger blocks. Each warp here already owns a fully independent output tile with its own shared-memory slice, so widening the block from 4 to 8 warps was the obvious first lever to test: does packing more of that already-independent work into fewer, bigger launches close any of the gap. Templated both kernels on WarpsPerBlock and added an 8-warp instantiation behind VT_ROCM_QUANT_WMMA_WIDE=1 (default off, same same-binary-A/B posture VT_ROCM_QUANT_WMMA and VT_ROCM_Q6K_SMALL_PRIVATE already ship with). Hardware-verified correct on both configs: ctest -R rocm|cross_device, 46/46 cases, 84066/84066 assertions, zero regression. The answer is no. Op-level A/B (quant-gemm-bench, RX 9060 XT, best-of-4, idle host) across the six Q4_K/Q6_K prefill shapes: geomean -4.3%, a net regression. Each warp shares no loaded or dequantized data with any other warp in its block, so widening only spreads the same LDS-capped occupancy budget over fewer, bigger blocks rather than doing more work per warp -- the axis this measurement rules out, not the one that explains llama.cpp's actual advantage. The sharper, still-open hypothesis this points at -- cross-warp data reuse in llama.cpp's own kernel -- is recorded in the spec's Owed section, unread and unconfirmed, as the next traceable step. The toggle stays in the tree, default off, as a ready-made A/B for re-checking this specific axis on different hardware or a future toolchain revision, rather than making the next person re-derive it from scratch. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-sonnet-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #3032's same-tool
rocprofv3trace foundKQuantGemmKWmmaQ4K/Q6K4.9x-10.6x slower per-kernel than llama.cpp's
mul_mat_qon the identicaltensor-core mechanism, and llama.cpp's launch uses double the warps per
block and 16-48x fewer, bigger blocks. Row:
KERNEL-QUANT-CIQ-GEMM-ROCM(already
ACTIVE); spec update in the same commit.Each warp in this kernel already owns a fully independent output tile with
its own shared-memory slice, so widening from 4 to 8 warps/block was the
obvious first lever: does packing more of that already-independent work
into fewer, bigger launches close the gap. Templated both kernels on
WarpsPerBlock, added an 8-warp instantiation behindVT_ROCM_QUANT_WMMA_WIDE=1(default off).The answer is no. Op-level A/B (
quant-gemm-bench, RX 9060 XT,best-of-4, idle host), all six Q4_K/Q6_K prefill shapes:
Geomean -4.3%, a net regression. Each warp shares no loaded/dequantized
data with any other warp in its block, so widening only spreads the same
LDS-capped occupancy budget over fewer, bigger blocks rather than doing
more work per warp — the axis this measurement rules out, not the one
explaining llama.cpp's actual advantage. The sharper, still-open
hypothesis — cross-warp data reuse in llama.cpp's own kernel — is recorded
in the spec's
## Owed, unread and unconfirmed, as the next traceablestep.
Gated on
isravale(RX 9060 XT, gfx1200, ROCm 7.2.3):ctest -R rocm|cross_device, both configs, 46/46 cases, 84066/84066 assertions,zero regression (the two remaining preflight failures,
test_tower_skip_rss_armand thetoolssuite'stest_drop_file_cache,are pre-existing environment artifacts — a tmpfs
/tmpand an unrelatedbenchmark-instrumentation self-test — unrelated to this change and
reproduced identically on an unmodified checkout).
The toggle stays in the tree, default off, as a ready-made A/B for
re-checking this specific axis on different hardware (gfx1201) or a future
toolchain revision, matching this file's own
VT_ROCM_Q6K_SMALL_PRIVATEprecedent for a measured-and-rejected diagnostic arm, rather than making
the next person re-derive it from scratch.
Closes #3032 with this evidence: the experiment it proposed ran, and the
result is a documented negative, not an open question.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-sonnet-5 [Claude Code]