Skip to content

gfx1151 falls back to the legacy QuantizeQ8KK: the cooperative Q8_K activation quantizer is gated on a strict gfx1100 prefix #3018

Description

@localai-org-maint-bot

Row: BACKEND-ROCM

#1876 measured QuantizeQ8KK, the thread-per-superblock activation quantizer,
at 95.5 us/call against llama.cpp's 1.6 us, and put it at 41% of the ROCm
decode gap on gfx1200. 70d8d317e and 17f4568d1 landed the fix, a
block-per-superblock cooperative quantizer with a shared-memory argmax
reduction (QuantizeQ8KCooperativeK,
c796fea41:src/vt/rocm/rocm_grouped_gemm.hip:153-208).

gfx1151 does not get it. The selector is gated on a strict gfx1100 prefix
test (:641-643, :673-694):

constexpr bool kQ8KGfx1100DefaultAccepted = true;
...
return gfx1100_default_accepted && Q8KArchIsGfx1100(arch)
           ? Q8KQuantArm::kCandidate
           : Q8KQuantArm::kLegacy;

So on Strix Halo every MatmulBTQuant dispatch quantizes its activation with
QuantizeQ8KK at 128 threads (:714-717). VT_ROCM_Q8K_BLOCK=1 overrides it.

The gate is scoped that narrowly on purpose -- the comment at :641-643 says
the real-checkpoint and profiler evidence covered gfx1100 only, which is the
right discipline. This issue is the missing measurement, not a complaint about
the gate.

Why it is worth measuring rather than assuming

At K = 5120 the activation is one row of nsb = 20 superblocks, so the
legacy kernel launches one block with 20 of 128 threads doing work, each
serially reducing 256 elements. The kernel is tiny in bytes and large in
latency, and it runs once per quantized matmul: on Qwen3.8-27B that is
roughly five to seven dispatches per layer across 64 layers, so a few hundred
launches per token.

That is a fixed per-dispatch cost, which is why it was 41% of a 9B model's gap
and is expected to be a much smaller share of a 27B one -- 27B decode is
185.3 ms/token (#3002) against the 9B's 57.3 ms, over similar dispatch counts.
No figure is claimed here. The point is that the cost is unmeasured on this
board and the cheap arm is already written and gated off.

What is asked

VT_ROCM_Q8K_BLOCK=1 versus default, same binary, same lease, on
Qwen3.8-27B-Q4_K_M on strix:gpu0. If it wins, widen Q8KArchIsGfx1100 to
carry gfx1151 with the evidence beside it; if it loses, record that on this
row so the next reader does not re-derive it. Either result can ride #3015's
profiling lease.

Related: #1876, #3015, #3016, #2497, #2921, #41.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions