Conversation
mmb_enabled() was true for every gfx1151 device, and mmb_quant_type() claims almost every quant type, so the BF16 WMMA path took MUL_MAT/MUL_MAT_ID away from MMQ for every model. Its tiles and fusions are tuned for the qwen4exp shapes (the hardcoded 320/10240 gate-mix shapes among them); elsewhere it loses. Measured on gfx1151, ROCm 7.2.1, llama-bench -ngl 99 -fa on, prefill t/s: model ub mmb on this PR pre-#63 (cfe6bb1) Signal-3.8-27B Q4_K_XL 512 119.6 467.7 461.2 Signal-3.8-27B Q4_K_XL 16384 123.3 421.8 416.1 Qwen3.8-27B GSQ-RCO IQ3_S 512 116.3 438.8 434.9 gemma-4-26B-A4B Q4_0 2048 1502.2 2087.6 2083.0 gpt-oss-20b MXFP4 2048 2066.2 2406.6 2411.3 Dense qwen35 paid 3.4-3.9x, MoE 14-28%. Token generation is unchanged in every case (the path only engages from 512 tokens up, so it never runs during decode). The llama layer now opts a model in by architecture through the backend's get_proc_address, so qwen4exp keeps the path it was tuned for and nothing else pays for it. A process that loads several models shares the flag, last load wins; that is noted at the definition. Not measured: qwen4exp itself, for want of a checkpoint on this box. The gain there is pwilkin's (#63) and this does not change that path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Yes, do it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
mmb_enabled()is true for every gfx1151 device andmmb_quant_type()claims almost every quanttype, so #63's BF16 WMMA path takes
MUL_MAT/MUL_MAT_IDaway from MMQ for every model. Itstiles and fusions are tuned for the qwen4exp shapes — the hardcoded
320/10240gate-mix shapesin
ggml-cuda.cuamong them — and on other architectures it loses badly.This makes the path opt-in per model, by architecture, through the backend's
get_proc_address(the same mechanism
ggml_backend_cpu_numa_initand friends use). qwen4exp keeps exactly the pathit was tuned for; nothing else pays for it.
This is not a revert and it is not a claim that the path is bad — on its target it is @pwilkin's
win and this PR does not touch it. It is a claim that it should not be on by default for
architectures nobody measured it on.
Measurements
All runs
llama-bench -ngl 99 -fa on -r 2,-band-ubset equal to the prompt length.Prefill t/s:
cfe6bb14eDense qwen35 pays 3.4-3.9x, MoE 14-28%. This PR restores pre-#63 throughput in every case.
Token generation is unchanged everywhere, as expected — the path only engages from 512 tokens up,
so it never runs during decode:
Correctness:
test-backend-ops -b ROCm029695/29695 on master and with this PR — the path wasnumerically correct, just slower than the MMQ path it displaced.
test-llama-archspasses with nofailures, qwen4exp included (OK 9.97e-14 on the Radeon 8060S).
Additional information
Note for @pwilkin: the gate is per process and set at model load, so a process serving several
models at once shares it and the last load wins. That is noted at the definition. If you would
rather gate by tensor shape or by quant type — the fast decoders only cover IQ4_NL, Q8_0 and BF16,
while
mmb_quant_type()claims about twenty types — that would be finer-grained than this, andyour numbers should decide it.
Related: #74 fixes a SIGSEGV in the qwen4exp PLE prefetch that hits the same non-qwen4exp models at
ubatch >= 4096. The two are independent.
qwen4exp verification (added after review of the first draft)
ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUFQ2_0 (66.4 GB,qwen4exp), same box and flags.This is the architecture the path was built for, so the gate must leave it alone — and it does:
0636c9aee)8c1c282ecmmb stays enabled for qwen4exp: had the gate missed it, prefill would have dropped to ~120 t/s the
way the dense qwen35 models do.
--lazy-mode on-direct(thellama_ple_diskreader) also runsunchanged: pp4096 549.72 / tg32 29.31 on master versus 548.48 / 29.82 with this PR.
Requirements
belong in halo-box/llama.cpp instead
and this description, and ran every measurement; the owner reviews.
mmb_enabled()already required RDNA3.5. Speculative decoding throughput was not measured (
llama-benchhas noMTP/draft support), only plain decode.