Skip to content

cuda: enable the BF16 WMMA matmul path only for qwen4exp - #75

Open
dzannotti wants to merge 1 commit into
masterfrom
perf/mmb-qwen4exp-only
Open

dzannotti wants to merge 1 commit into
masterfrom
perf/mmb-qwen4exp-only

Conversation

@dzannotti

@dzannotti dzannotti commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Overview

mmb_enabled() is true for every gfx1151 device and mmb_quant_type() claims almost every quant
type, so #63's BF16 WMMA path takes MUL_MAT/MUL_MAT_ID away from MMQ for every model. Its
tiles and fusions are tuned for the qwen4exp shapes — the hardcoded 320/10240 gate-mix shapes
in ggml-cuda.cu among them — and on other architectures it loses badly.

This makes the path opt-in per model, by architecture, through the backend's get_proc_address
(the same mechanism ggml_backend_cpu_numa_init and friends use). qwen4exp keeps exactly the path
it was tuned for; nothing else pays for it.

This is not a revert and it is not a claim that the path is bad — on its target it is @pwilkin's
win and this PR does not touch it. It is a claim that it should not be on by default for
architectures nobody measured it on.

Measurements

Device:     Ryzen AI Max+ 395 (Radeon 8060S, gfx1151)
Memory:     128 GB LPDDR5X-8000
Kernel:     Linux 7.0.0-31-generic, amdgpu params: none
Backend:    ROCm 7.2.1 (rocm/dev-ubuntu-24.04:7.2.1-complete container), HIP_LAUNCH_BLOCKING=0
Build:      -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
Baseline:   8c1c282ec (master)  /  cfe6bb14e (last commit before #63)
Change:     42f7e8948
Models:     agentionai/Signal-3.8-27B-GGUF AP-Q4_K_XL          (qwen35, dense)
            ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF IQ3_S-mtp     (qwen35, dense)
            google/gemma-4-26B-A4B-it-qat-q4_0-gguf            (gemma4, MoE)
            gpt-oss-20b-MXFP4                                  (MoE)

All runs llama-bench -ngl 99 -fa on -r 2, -b and -ub set equal to the prompt length.
Prefill t/s:

model ubatch master (mmb on) this PR pre-#63 cfe6bb14e
Signal-3.8-27B Q4_K_XL 512 119.59 ± 0.51 467.70 461.21 ± 4.74
Signal-3.8-27B Q4_K_XL 4096 127.86 469.62
Signal-3.8-27B Q4_K_XL 16384 123.31 421.81 416.13
Qwen3.8-27B IQ3_S 512 116.39 ± 0.42 438.83 434.89
Qwen3.8-27B IQ3_S 16384 115.65 403.45
gemma-4-26B-A4B Q4_0 2048 1502.24 2087.63 2083.04
gpt-oss-20b MXFP4 2048 2066.16 2406.58 2411.30

Dense qwen35 pays 3.4-3.9x, MoE 14-28%. This PR restores pre-#63 throughput in every case.

Token generation is unchanged everywhere, as expected — the path only engages from 512 tokens up,
so it never runs during decode:

  Signal-3.8-27B Q4_K_XL   tg64  12.09 -> 12.09
  Qwen3.8-27B IQ3_S        tg64  15.72 -> 15.72
  gemma-4-26B-A4B          tg64  65.39 -> 65.39
  gpt-oss-20b MXFP4        tg64  73.58 -> 73.58

Correctness: test-backend-ops -b ROCm0 29695/29695 on master and with this PR — the path was
numerically correct, just slower than the MMQ path it displaced. test-llama-archs passes with no
failures, qwen4exp included (OK 9.97e-14 on the Radeon 8060S).

Additional information

Note for @pwilkin: the gate is per process and set at model load, so a process serving several
models at once shares it and the last load wins. That is noted at the definition. If you would
rather gate by tensor shape or by quant type — the fast decoders only cover IQ4_NL, Q8_0 and BF16,
while mmb_quant_type() claims about twenty types — that would be finer-grained than this, and
your numbers should decide it.

Related: #74 fixes a SIGSEGV in the qwen4exp PLE prefetch that hits the same non-qwen4exp models at
ubatch >= 4096. The two are independent.

qwen4exp verification (added after review of the first draft)

ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF Q2_0 (66.4 GB, qwen4exp), same box and flags.
This is the architecture the path was built for, so the gate must leave it alone — and it does:

build pp512 pp2048 tg64 pp4096 (ub 4096) pp16384 (ub 16384)
#63 alone (0636c9aee) 447.87 443.22 28.81 545.80 573.49
master 8c1c282ec 452.54 445.97 30.31 547.60 576.60
this PR 451.92 445.49 30.32 545.17 575.59

mmb stays enabled for qwen4exp: had the gate missed it, prefill would have dropped to ~120 t/s the
way the dense qwen35 models do. --lazy-mode on-direct (the llama_ple_disk reader) also runs
unchanged: pp4096 549.72 / tg32 29.31 on master versus 548.48 / 29.82 with this PR.

Requirements

  • I have read and agree with the contributing guidelines
  • This change is Strix Halo specific, or justified by measurements on Strix Halo. General llama.cpp improvements
    belong in halo-box/llama.cpp instead
  • AI usage disclosure: AGENT-AUTHORED. Claude (Claude Code) bisected the regression, wrote the patch
    and this description, and ran every measurement; the owner reviews.
  • What was NOT verified: CUDA (NVIDIA) was not built; the gate is a no-op there since mmb_enabled()
    already required RDNA3.5. Speculative decoding throughput was not measured (llama-bench has no
    MTP/draft support), only plain decode.

mmb_enabled() was true for every gfx1151 device, and mmb_quant_type() claims
almost every quant type, so the BF16 WMMA path took MUL_MAT/MUL_MAT_ID away
from MMQ for every model. Its tiles and fusions are tuned for the qwen4exp
shapes (the hardcoded 320/10240 gate-mix shapes among them); elsewhere it
loses.

Measured on gfx1151, ROCm 7.2.1, llama-bench -ngl 99 -fa on, prefill t/s:

  model                        ub     mmb on   this PR   pre-#63 (cfe6bb1)
  Signal-3.8-27B Q4_K_XL       512     119.6     467.7     461.2
  Signal-3.8-27B Q4_K_XL     16384     123.3     421.8     416.1
  Qwen3.8-27B GSQ-RCO IQ3_S    512     116.3     438.8     434.9
  gemma-4-26B-A4B Q4_0        2048    1502.2    2087.6    2083.0
  gpt-oss-20b MXFP4           2048    2066.2    2406.6    2411.3

Dense qwen35 paid 3.4-3.9x, MoE 14-28%. Token generation is unchanged in every
case (the path only engages from 512 tokens up, so it never runs during decode).

The llama layer now opts a model in by architecture through the backend's
get_proc_address, so qwen4exp keeps the path it was tuned for and nothing else
pays for it. A process that loads several models shares the flag, last load
wins; that is noted at the definition.

Not measured: qwen4exp itself, for want of a checkpoint on this box. The gain
there is pwilkin's (#63) and this does not change that path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pwilkin

pwilkin commented Sep 18, 2026

Copy link
Copy Markdown

Yes, do it.
That's a problem with quick performance fixes - they tend to introduce regressions elsewhere :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants