Skip to content

ROCm grouped keep-quant GEMM drops Q8_0 when the kROCM provider takes over #2927

Description

@localai-org-maint-bot

Row: GFX1100-TG200

#2782 moves OpId::kMatmulBTQuantGrouped on kROCM from
MatmulBTQuantGroupedKernelRocm (src/vt/rocm/rocm_grouped_gemm.hip, renamed
*Gdn there) to the new provider in src/vt/rocm/rocm_quant_dot.hip.

The new provider delegates Q4_K/Q5_K/Q6_K back to the *Gdn kernel but not
Q8_0. Q8_0 also has no arm in rocm_quant_dot.hip: it dots a Q8_0
activation, not a Q8_K super-block, so IsRocmKeepQuantSupported(kQ8_0)
answers no. A grouped Q8_0 expert GEMM therefore throws
"no keep-quant kernel for dtype q8_0" on a path that main serves today.

The scalar arm is unaffected: MatmulBTQuantKernelRocm routes Q8_0 to its own
MatmulQ8_0Rocm.

tests/vt/test_rocm_quant_dot.cpp cannot see it because its kCases table
carries the ten Q8_K-family encodings and no Q8_0.

Found while reviewing #2782. Fixed in the same flow, in that pull request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions