feat(KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT): port IQ4_XS/IQ3_XXS to ROCm - #3029
Open
joral wants to merge 2 commits into
Open
feat(KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT): port IQ4_XS/IQ3_XXS to ROCm#3029joral wants to merge 2 commits into
joral wants to merge 2 commits into
Conversation
Issue mudler#1940 says src/vt/rocm/ ports zero I-quant formats, so IQ4_XS and IQ3_XXS still expand to bf16 on a ROCm keep-quant load instead of staying compressed. This lands DotIQ4XS and DotIQ3XXS in rocm_grouped_gemm.hip, adapted from the existing CUDA bodies rather than freshly ported, on both the plain and grouped/MoE arms, and admits both dtypes in DeviceKeepQuantSupported's ROCm case. The spec's FMA-contraction risk is resolved by measurement, not assumption. CUDA's DotIQ4XS needed non-fused float intrinsics to stay bit-exact against the oracle, because nvcc's project-wide flag never reaches .cu files. HIP already compiles with -ffp-contract=off project-wide, and a new bit-exact gate over the same real checkpoint bytes and the same oracle bits CUDA's gate uses confirms plain arithmetic is sufficient on this toolchain, so the port carries no workaround CUDA needed. Three existing tests asserted the prior ROCm behavior by name and go red under the new routing, each for the reason it should: the cross-device table gains both dtypes and a new bit-exact case, the keep-quant loader's exhaustive per-device count moves from 8 to 10, and a residency pin that named issue mudler#1940 splits per-tensor now that one of its two towers keeps. Run on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3) under the GPU file mutex: test_backend_cross_device 41/42 (the one failure, MoeSiluMul, is confirmed pre-existing and unrelated via an independent binary built from the sibling KERNEL-QUANT-CIQ-GEMM-ROCM-RDNA4-w1 worktree reproducing the identical mismatch); test_gguf_keep_quant 52/52; test_gguf_device_fit 24/24; every other ROCm suite green with zero regression. The row's actual acceptance criterion also landed: an end-to-end reload of the real motivating checkpoint, Nail-Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf (19.39 GB, on isravale). Its own header histogram is IQ4_XS/Q5_K/Q6_K only, so it exercises exactly the dtypes this change adds plus the two already supported. It loads, generates coherent tokens, and its resident footprint (~19.33 GiB) matches the on-disk size instead of the ~70 GiB a bf16 expansion would produce -- the fix for the host-RAM SIGSEGV this checkpoint hit before this row, on the artifact that motivated it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-sonnet-5 [Claude Code]
…OCm keep-quant line The keep-quant enumeration went stale when this row's implementation added IQ4_XS/IQ3_XXS to ROCm's DeviceKeepQuantSupported but touched no docs file, a gap a fresh review of PR mudler#3029 found. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-sonnet-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #1940 says
src/vt/rocm/ports zero I-quant formats, so an IQ4_XS orIQ3_XXS tensor still expands to bf16 on a ROCm keep-quant load instead of
staying compressed. This lands
DotIQ4XSandDotIQ3XXSinrocm_grouped_gemm.hip, adapted from the existing CUDA bodies rather thanfreshly ported, on both the plain and grouped/MoE arms, and admits both
dtypes in
DeviceKeepQuantSupported's ROCm case. Spec:.agents/specs/kernel-quant-ciq-gemm-rocm-iquant.md(#2996, merged).The spec's one named correctness risk is resolved by measurement, not
assumption: CUDA's
DotIQ4XSneeded non-fused float intrinsics to staybit-exact against the oracle, because nvcc's project-wide
-ffp-contract=offnever reaches
.cufiles. HIP already compiles with that flag project-wide,and a new bit-exact gate over the same real-checkpoint bytes and the same
oracle bits the CUDA gate uses confirms plain arithmetic is sufficient on
this toolchain, so the port carries no workaround CUDA needed.
The row's actual acceptance criterion also landed: an end-to-end reload of
the real motivating checkpoint,
Nail-Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf(19.39 GB). It loads, generates coherent tokens, and its resident footprint
(~19.33 GiB) matches the on-disk size instead of the ~70 GiB a bf16
expansion would produce — the fix for the host-RAM SIGSEGV this checkpoint
hit on ROCm before this row.
Gated on
isravale(RX 9060 XT, gfx1200, ROCm 7.2.3) under the GPU filemutex:
test_backend_cross_device41/42 (the one failure,MoeSiluMul, isthe pre-existing, already-tracked #1954 flake — confirmed unrelated);
test_gguf_keep_quant52/52;test_gguf_device_fit/test_gguf_device_fit_reachgreen; every other ROCm suite green with zero regression on the four
existing k-quant formats.
Owed, named rather than silently dropped: the
ROCM-KQUANT-NWARPS-DECODEre-measurement for these two formats (
PENDING, norocprofv3profilingsetup staged on this box yet — does not block this row, whose own scope is
coverage and correctness); a WMMA/tensor-core tile for these two formats
(follow-on row
KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-RDNA4, spec drafted,depends on this PR merging first); the other five formats named in the same
ROCm refusal message (Q4_0/Q2_K/Q3_K/IQ2_XXS/IQ2_S/MXFP4, tracked by #1940).
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-sonnet-5 [Claude Code]