Row: BACKEND-ROCM
#1876 measured QuantizeQ8KK, the thread-per-superblock activation quantizer,
at 95.5 us/call against llama.cpp's 1.6 us, and put it at 41% of the ROCm
decode gap on gfx1200. 70d8d317e and 17f4568d1 landed the fix, a
block-per-superblock cooperative quantizer with a shared-memory argmax
reduction (QuantizeQ8KCooperativeK,
c796fea41:src/vt/rocm/rocm_grouped_gemm.hip:153-208).
gfx1151 does not get it. The selector is gated on a strict gfx1100 prefix
test (:641-643, :673-694):
constexpr bool kQ8KGfx1100DefaultAccepted = true;
...
return gfx1100_default_accepted && Q8KArchIsGfx1100(arch)
? Q8KQuantArm::kCandidate
: Q8KQuantArm::kLegacy;
So on Strix Halo every MatmulBTQuant dispatch quantizes its activation with
QuantizeQ8KK at 128 threads (:714-717). VT_ROCM_Q8K_BLOCK=1 overrides it.
The gate is scoped that narrowly on purpose -- the comment at :641-643 says
the real-checkpoint and profiler evidence covered gfx1100 only, which is the
right discipline. This issue is the missing measurement, not a complaint about
the gate.
Why it is worth measuring rather than assuming
At K = 5120 the activation is one row of nsb = 20 superblocks, so the
legacy kernel launches one block with 20 of 128 threads doing work, each
serially reducing 256 elements. The kernel is tiny in bytes and large in
latency, and it runs once per quantized matmul: on Qwen3.8-27B that is
roughly five to seven dispatches per layer across 64 layers, so a few hundred
launches per token.
That is a fixed per-dispatch cost, which is why it was 41% of a 9B model's gap
and is expected to be a much smaller share of a 27B one -- 27B decode is
185.3 ms/token (#3002) against the 9B's 57.3 ms, over similar dispatch counts.
No figure is claimed here. The point is that the cost is unmeasured on this
board and the cheap arm is already written and gated off.
What is asked
VT_ROCM_Q8K_BLOCK=1 versus default, same binary, same lease, on
Qwen3.8-27B-Q4_K_M on strix:gpu0. If it wins, widen Q8KArchIsGfx1100 to
carry gfx1151 with the evidence beside it; if it loses, record that on this
row so the next reader does not re-derive it. Either result can ride #3015's
profiling lease.
Related: #1876, #3015, #3016, #2497, #2921, #41.
Row:
BACKEND-ROCM#1876 measured
QuantizeQ8KK, the thread-per-superblock activation quantizer,at 95.5 us/call against llama.cpp's 1.6 us, and put it at 41% of the ROCm
decode gap on gfx1200.
70d8d317eand17f4568d1landed the fix, ablock-per-superblock cooperative quantizer with a shared-memory argmax
reduction (
QuantizeQ8KCooperativeK,c796fea41:src/vt/rocm/rocm_grouped_gemm.hip:153-208).gfx1151 does not get it. The selector is gated on a strict
gfx1100prefixtest (
:641-643,:673-694):So on Strix Halo every
MatmulBTQuantdispatch quantizes its activation withQuantizeQ8KKat 128 threads (:714-717).VT_ROCM_Q8K_BLOCK=1overrides it.The gate is scoped that narrowly on purpose -- the comment at
:641-643saysthe real-checkpoint and profiler evidence covered gfx1100 only, which is the
right discipline. This issue is the missing measurement, not a complaint about
the gate.
Why it is worth measuring rather than assuming
At
K = 5120the activation is one row ofnsb = 20superblocks, so thelegacy kernel launches one block with 20 of 128 threads doing work, each
serially reducing 256 elements. The kernel is tiny in bytes and large in
latency, and it runs once per quantized matmul: on
Qwen3.8-27Bthat isroughly five to seven dispatches per layer across 64 layers, so a few hundred
launches per token.
That is a fixed per-dispatch cost, which is why it was 41% of a 9B model's gap
and is expected to be a much smaller share of a 27B one -- 27B decode is
185.3 ms/token (#3002) against the 9B's 57.3 ms, over similar dispatch counts.
No figure is claimed here. The point is that the cost is unmeasured on this
board and the cheap arm is already written and gated off.
What is asked
VT_ROCM_Q8K_BLOCK=1versus default, same binary, same lease, onQwen3.8-27B-Q4_K_Monstrix:gpu0. If it wins, widenQ8KArchIsGfx1100tocarry gfx1151 with the evidence beside it; if it loses, record that on this
row so the next reader does not re-derive it. Either result can ride #3015's
profiling lease.
Related: #1876, #3015, #3016, #2497, #2921, #41.