feat(KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT): port IQ4_XS/IQ3_XXS to ROCm

.agents/specs/kernel-quant-ciq-gemm-rocm-iquant.md

@@ -12,7 +12,8 @@
   2026-09-05).
 - Pull request shape: separate spec and implementation pull requests
   (developer decision 2026-09-05, recorded in
-  `.agents/developer-preferences.md`). This pull request lands the spec only.
+  `.agents/developer-preferences.md`). The spec landed in its own pull
+  request first; this implementation is the second.
 
 ## Scope
 
@@ -155,26 +156,31 @@ llama.cpp, pin `b10451` per `.agents/upstream-sync.md`.
 
 ## Risks
 
-- **FMA contraction on IQ4_XS's float-accumulation body.** IQ4_XS's dot is
-  the one format in this row (and in the whole quant-dot family) whose core
-  is not a single integer accumulator: it forms `d1`/`d2` as f32 and folds
-  in per-sub-block `sumf +=` steps, eight per super-block
-  (`cuda_quant_dot.cu:606-680`, extensively commented on exactly this
-  point). On CUDA that required `__fmul_rn`/`__fadd_rn` in place of ordinary
-  `*`/`+`, because nvcc's default `-fmad=true` silently contracts the
-  textual two-rounding sequence into a single-rounding FMA and two of eight
-  real super-blocks then disagreed with the oracle by 1-4 ULP. **This may
-  not reproduce on ROCm**: `CMakeLists.txt:414` already applies
-  `-ffp-contract=off` to `$<COMPILE_LANGUAGE:HIP>` project-wide, unlike CUDA
-  where the project's `-ffp-contract=off` is CXX-only and never reaches
-  `.cu`/`.cuh` translation units. Verify this empirically before assuming it
-  (a W0-style probe: compile the naive `sumf += d1 * x` form, diff against
-  the CPU oracle on the same real super-blocks CUDA's golden vectors use,
-  and inspect the generated ISA for `v_fma_f32` if any block disagrees) —
-  do not carry the CUDA workaround over unexamined, and do not assume the
-  flag alone is sufficient without a measured check, matching how the CUDA
-  side only added the intrinsics after measuring a real disagreement rather
-  than as a precaution.
+- **FMA contraction on IQ4_XS's float-accumulation body — MEASURED, RESOLVED
+  IN FAVOR OF THE SIMPLER PATH.** IQ4_XS's dot is the one format in this row
+  (and in the whole quant-dot family) whose core is not a single integer
+  accumulator: it forms `d1`/`d2` as f32 and folds in per-sub-block
+  `sumf +=` steps, eight per super-block (`cuda_quant_dot.cu:606-680`,
+  extensively commented on exactly this point). On CUDA that required
+  `__fmul_rn`/`__fadd_rn` in place of ordinary `*`/`+`, because nvcc's
+  default `-fmad=true` silently contracts the textual two-rounding sequence
+  into a single-rounding FMA and two of eight real super-blocks then
+  disagreed with the oracle by 1-4 ULP. **W0/W1 measured this directly on
+  the target hardware (RX 9060 XT, gfx1200, ROCm 7.2, `isravale`):** plain
+  `*`/`+` (no non-fused intrinsics) in `DotIQ4XS` is BIT-EXACT against the
+  oracle's own per-super-block numbers, over the SAME four real
+  `unsloth/GLM-5.3-Flash-GGUF` super-blocks and the SAME expected bits
+  (`iq2xs_iq4xs_dot_golden.h`) CUDA's gate uses, both isolated (k=256, one
+  contributing lane, zero reassociation possible) and combined (k=1024,
+  four lanes, the same `__shfl_down_sync` tree CUDA's comment derives) —
+  `tests/vt/test_backend_cross_device.cpp`, "ROCm IQ4_XS dots the ORACLE's
+  own numbers on REAL checkpoint bytes", 13/13 assertions green. The
+  hypothesis held: `CMakeLists.txt:414`'s project-wide
+  `-ffp-contract=off` on `$<COMPILE_LANGUAGE:HIP>` is sufficient on its own,
+  so `DotIQ4XS` on ROCm uses plain `*`/`+` and does **not** carry CUDA's
+  `__fmul_rn`/`__fadd_rn` workaround. This is a measured result, not an
+  assumption carried over — the whole point of naming this as a risk was to
+  force the check rather than inherit the CUDA fix by habit.
 - **The nwarps=8 decode table (`ROCM-KQUANT-NWARPS-DECODE`) may not transfer.**
   Both new formats share the existing `nsb = K/256` decomposition, so they
   compile against the same launch shape as Q4_K/Q5_K/Q6_K with no code
@@ -207,31 +213,88 @@ llama.cpp, pin `b10451` per `.agents/upstream-sync.md`.
 
 ## Tests
 
-- Extend `test_ops_quant_dot.cpp`'s existing IQ4_XS/IQ3_XXS `vec_dot`
-  golden-vector gates (`iq2xs_iq4xs_dot_golden.h`, already committed and
-  sourced from real `unsloth/GLM-5.3-Flash-GGUF` checkpoint bytes) to a new
-  `test_rocm_quant_dot.cpp`, same shape as the CUDA gate
-  (`test_cuda_quant_dot.cpp`): NMSE ≤ 5e-4 vs the independent f64
-  dequant-then-dot reference for IQ3_XXS; bit-exact (not NMSE) for IQ4_XS
-  against the same real-checkpoint golden values CUDA's gate uses, since
-  bit-exactness is the property the FMA-contraction risk above is actually
-  about.
-- `test_backend_cross_device.cpp`: add both formats to the CPU-vs-ROCM
-  cross-check, NMSE ≤ 5e-4 (matching the existing four formats' gate shape
-  there).
-- Rerun `ROCM-KQUANT-NWARPS-DECODE`'s own measurement recipe
-  (`rocprofv3 --kernel-trace` on a real quant-matched trace workload) for
-  IQ4_XS/IQ3_XXS specifically, to answer the nwarps question this issue was
-  filed to test — record the result (transfers / does not transfer) rather
-  than assuming either.
-- `ctest -R 'rocm|cross_device'`, zero regression on the four existing
-  formats' numerics.
-- End-to-end: reload the motivating checkpoint (or a same-format synthetic
-  fixture if the real 35B-A3B artifact is not staged on the gate host) on
-  `isravale` (RX 9060 XT, gfx1200) or an `rc`-leased ROCm fleet device, and
-  confirm keep-quant residency replaces the prior bf16 SIGSEGV — this is
-  the row's actual acceptance criterion, not merely the unit-level dot
-  gates.
+Landed, on `isravale` (RX 9060 XT, gfx1200, ROCm 7.2.3), GPU work under
+`flock ${GPU_LOCK:-$HOME/gpu.lock}` throughout:
+
+- **`test_backend_cross_device.cpp`**, three cases touched/added, run
+  standalone and as part of the full file (41/42 cases, 83998/83999
+  assertions — the one failure is `MoeSiluMul matches the CPU oracle within
+  NMSE <= 5e-4`, confirmed PRE-EXISTING and unrelated: byte-identical
+  mismatch reproduced on an independent binary built from the sibling
+  `KERNEL-QUANT-CIQ-GEMM-ROCM-RDNA4-w1` worktree, which touches neither this
+  kernel nor this dtype):
+  - "non-grouped keep-quant GEMM (...IQ4_XS/IQ3_XXS) matches the CPU
+    oracle" — both new formats added to the existing table-driven CPU-vs-
+    ROCm case, NMSE ≤ 5e-4, random valid blocks (unconstrained lookup
+    indices need no in-range fixture change).
+  - "grouped quant expert GEMM (...IQ4_XS/IQ3_XXS) matches the CPU oracle"
+    — same extension on the grouped/MoE path, the one the motivating
+    checkpoint's routed experts actually use.
+  - "ROCm IQ4_XS dots the ORACLE's own numbers on REAL checkpoint bytes"
+    (NEW) — the bit-exact gate the FMA-contraction risk needed, ported from
+    `test_cuda_quant_dot.cpp`'s `CheckCudaOracleDot` shape onto the same
+    golden vectors: bit-exact per-superblock (k=256, one contributing lane)
+    and warp-reduction-order-exact combined (k=1024, four lanes, primary
+    bit-equality + secondary reassociation-bound check). 13/13 assertions.
+- **`test_gguf_keep_quant.cpp`**: the exhaustive per-device totality table's
+  hand-mirrored ROCm predicate and its `gemm_kept` constant (8 → 10) updated
+  to admit IQ4_XS; IQ3_XXS is not in this test's `all_types` enumeration
+  (a pre-existing gap shared with Q2_K, not closed by this row) and is left
+  to the cross-device gate above. 52/52 cases, 10325/10325 assertions.
+- **`test_gguf_device_fit.cpp`**: `#2516`'s two ROCm residency pins split
+  per-tensor (IQ4_XS's `down_exps` now expects `kKeepQuant` on ROCm;
+  IQ2_XS's `gate_exps` is unaffected and still expects `kExpandBf16`,
+  since #1940's other five formats stay owed); the all-or-nothing
+  "NO PLAN" case is unchanged in outcome (`CHECK_FALSE` still holds, because
+  the still-unsupported IQ2_XS tower alone fails the lane) with its comment
+  corrected to say why. 24/24 cases, 182/182 assertions.
+- `ctest -R 'rocm|cross_device'` (plus the individually-run ROCm suites
+  `test_rocm_arch`/`test_rocm_backend`/`test_exl3_rocm`/
+  `test_gemma4_rocm_fp8_seams`/`test_rocm_fp8_kv_cache`): zero regression,
+  all green.
+
+**End-to-end reload — the row's actual acceptance criterion — LANDED.**
+`Nail-Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf` (19.39 GB on disk, `isravale`
+`/home/justin/Nail/`) is the real motivating checkpoint, not a stand-in: its
+own header histogram is `{BF16: 2, F32: 308, IQ4_XS: 391, Q5_K: 51,
+Q6_K: 1}` — every quantized tensor in the file is one of the three dtypes
+this row's target hardware now has a keep-quant kernel for (read with
+`docs/bench-evidence/limb3-vehicle-search-20260904/gguf_header.py` before
+running anything, not assumed from the filename).
+
+```
+VT_DEVICE_WEIGHT_BUDGET_BYTES=13000000000 \
+./build-hip/examples/vllm-cli --model /home/justin/Nail/Nail-Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf \
+  --device auto --max-num-seqs 1 --kv-cache-dtype fp8 --kv-cache-memory 2000000000 \
+  --prompt "The capital of France is" --max-tokens 16
+```
+
+```
+engine: device placement INSTALLED: 15 layers run their routed experts on cpu, the rest on rocm (resolved against 40 layers, origin fit)
+engine: device placement: --fit placed 15 layer(s) (6417285120 B) to bring a 19333564672 B footprint under a 13000000000 B budget
+vllm-cli: run=1/1 finish_reason=length prompt_tokens=5 completion_tokens=16 secs=3.919 tok_s=4.083
+ Paris. The capital of Germany is Berlin. The capital of Italy is Rome.
+```
+
+The decisive number is the **19,333,564,672 B (~18.01 GiB) footprint** --
+it matches the file's on-disk size, not the ~70 GiB a bf16 expansion of
+these tensors would produce. That is the keep-quant residency actually
+taking effect on ROCm, not merely compiling: before this row,
+`DeviceKeepQuantSupported` routed every IQ4_XS tower to `kExpandBf16` here
+and the streamed-expert lane's blow-up SIGSEGV'd this same box on this
+family of checkpoint (`vllm-cpp-rocm-crash-iq4xs` session memory). Clean
+exit, coherent completion, zero crash. Not a synthetic fixture, not a
+narrower stand-in geometry -- the actual artifact the row exists for.
+
+Not done in this wave (see Owed):
+
+- The `ROCM-KQUANT-NWARPS-DECODE` re-measurement (`rocprofv3 --kernel-trace`
+  on a real quant-matched trace workload) — this issue's own stated reason
+  for existing beyond plain coverage. `isravale` has no `rocprofv3` profiling
+  set up in this session; the correctness gates above stand on their own,
+  but the nwarps question is still open. The 4.083 tok/s figure above is NOT
+  a substitute measurement for it: it is a mixed CPU+ROCm run at a
+  CPU-offload-heavy split, not an isolated ROCm-kernel throughput number.
 
 ## Owed
 
@@ -241,9 +304,10 @@ llama.cpp, pin `b10451` per `.agents/upstream-sync.md`.
 - A WMMA/tensor-core tile for IQ4_XS/IQ3_XXS, if the scalar tier's measured
   throughput warrants one (mirroring how `KERNEL-QUANT-CIQ-GEMM-ROCM`
   followed the existing four formats' scalar tier): not attempted here.
-- The nwarps re-measurement itself, if it is not completed within this
-  row's implementation wave for lack of GPU time: record as `PENDING` on a
-  named lease/box, never silently dropped.
+- The nwarps re-measurement itself: `PENDING`, not completed within this
+  implementation wave for lack of a `rocprofv3` profiling setup on
+  `isravale`, not silently dropped. The correctness gates (Tests) are
+  unaffected by this being open.
 
 ## Stop conditions
 
@@ -260,8 +324,24 @@ llama.cpp, pin `b10451` per `.agents/upstream-sync.md`.
 
 ## Now
 
-`SPIKE`. This pull request lands the spec only; no product code changes in
-this change. Next: W0 probes the FMA-contraction question on target
-hardware (gfx1200), then W1 ports `DotIQ4XS` (the harder, float-accumulation
-body) and W2 ports `DotIQ3XXS`, each with its own focused gate before the
-combined `ctest` sweep.
+`ACTIVE`. W0 (FMA-contraction probe), W1 (`DotIQ4XS`) and W2 (`DotIQ3XXS`)
+are LANDED in this pull request, on both the plain (`MatmulBTQuantKernelRocm`
+/ `KQuantGemmK`) and grouped/MoE (`MatmulBTQuantGroupedKernelRocm` /
+`GroupedKQ8K`) arms, plus `DeviceKeepQuantSupported`'s ROCm admission list.
+Gated per the Tests section above, on target hardware (`isravale`,
+RX 9060 XT / gfx1200), zero regression. The FMA-contraction risk resolved in
+favor of the simpler path: HIP's project-wide `-ffp-contract=off` is
+sufficient, no CUDA-style non-fused-multiply workaround needed.
+
+**The real-checkpoint end-to-end reload also LANDED**, after this pull
+request was first drafted: `Nail-Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf` loads and
+generates coherent tokens on `isravale`, with the resident footprint
+(~18.01 GiB) matching the on-disk size rather than a bf16 blow-up — see
+Tests. That was the row's actual acceptance criterion, and it is now
+satisfied on the artifact that motivated the row, not a synthetic
+stand-in.
+
+Remaining before `DONE`: only the `ROCM-KQUANT-NWARPS-DECODE`
+re-measurement (`PENDING`, see Owed) — it does not block this pull request,
+since the row's own scope is coverage and correctness, and it is named
+rather than silently dropped.

.agents/specs/rocm-iquant-integration-3029.md

@@ -0,0 +1,52 @@
+# Integrate the ROCm I-quant contribution and test loader admission
+
+Row: `KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT`.
+Issue: [#1940](https://github.com/mudler/vllm.cpp/issues/1940), which remains
+open for the other quantized formats.
+Contribution: [#3029](https://github.com/mudler/vllm.cpp/pull/3029).
+Parent spec: [ROCm I-quant port](kernel-quant-ciq-gemm-rocm-iquant.md).
+Contribution base: `7aa0aa00a8eb79d53e65685e78e8da6d3f10482a`.
+Initial integration target: `415d17859500caf2a4cac00511820e4f4760e86f`.
+Final integration target: `08a34c3a74d78046f83886f242d07110a70ff45e`, which
+includes the prerequisite README scan repair from #3064.
+
+## Scope and source
+
+Resolve the three integration conflicts in `gguf_keep_quant.cpp`,
+`rocm_grouped_gemm.hip`, and `test_backend_cross_device.cpp`.
+Preserve both the target's behavior and the contribution's IQ4_XS and IQ3_XXS
+admission, dense kernels, grouped kernels, and tests.
+Do not import #3036 or redesign a kernel.
+
+The parent spec defines the source algorithms and device gates.
+`git log -S kIQ3_XXS -- src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`
+identifies `acd7d457d` as the loader admission change.
+The contributor's routing test explicitly omits IQ3_XXS. Its direct operation
+tests cannot detect deletion of IQ3_XXS from loader admission.
+
+## Design and tests
+
+Add a test through `RouteGgufTensor` for ROCm IQ3_XXS matrix and stacked
+expert weights. Both aligned roles must keep their blocks. Ragged shapes,
+disabled keep-quant, and CPU-reference mode must still expand to bf16.
+Keep the broad existing routing table unchanged except for its obsolete
+coverage comment. The new test owns the formerly missing admission guarantee.
+
+Before accepting the regression, remove IQ3_XXS from the ROCm admission arm
+in a scratch copy and require the test to fail. Restore the original source
+and require the loader suite to pass.
+Run focused loader and device-fit tests plus `scripts/agent-preflight.sh`.
+The operator builds HIP and runs the contributor's device tests under a lease.
+Generic I-quant checks on Strix do not establish gfx1200 performance.
+
+## Records and stop conditions
+
+If a keyed record conflicts, start from its complete target version and reapply
+only this row's edit. Verify unrelated keys against the target byte-for-byte.
+No lifecycle change or new benchmark publication belongs to this repair.
+Stop if conflict resolution requires choosing between incompatible behaviors,
+changes residency-budget semantics, or needs a new kernel design.
+
+## Now
+
+ACTIVE: integration and loader regression specified before code changes.

.agents/specs/rocm-iquant-table-seal-3067.md

@@ -0,0 +1,59 @@
+# Seal every ROCm I-quant table against its CPU reference
+
+Row: `KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT`.
+Issue: [#3067](https://github.com/mudler/vllm.cpp/issues/3067).
+Contribution: [#3029](https://github.com/mudler/vllm.cpp/pull/3029).
+Parent spec: [ROCm I-quant port](kernel-quant-ciq-gemm-rocm-iquant.md).
+Repair base: `b2ee9d8389caf974f3178613a6313e788dd93c4b`.
+
+## Gap and source
+
+The parent spec and `rocm_quant_iq_tables.h:17-20` require a complete seal.
+Review found no executing comparison of the four ROCm device tables against
+their CPU references. Host-parsed arrays currently agree, but that inspection
+does not pin every executing device byte in a regression test.
+
+Mirror the existing CUDA snapshot in `cuda_quant_dot.cu:2637` and the test in
+`test_cuda_quant_dot.cpp:1702`. The CPU tables carry the parent spec's pinned
+llama.cpp reference. This change adds no quantization algorithm or oracle.
+
+## Design and scope
+
+Add a HIP-free internal snapshot declaration for the four arrays:
+`d_kmask_iq2xs`, `d_ksigns_iq2xs`, `d_iq3xxs_grid`, and `d_kvalues_iq4nl`.
+Define the copy in `rocm_grouped_gemm.hip`, which defines the device symbols.
+Use `hipMemcpyFromSymbol` and the existing HIP error checker. Compile-time
+extent checks prevent truncation. Do not change table values, storage classes,
+arithmetic, dispatch, or any other quantized format.
+
+Add a HIP test to `test_backend_cross_device.cpp`. Compare each complete
+snapshot array with its CPU reference using `memcmp`. Check all four extents
+and the number of comparisons. Do not substitute host literals for device
+copies. Follow the executable's missing-backend convention; a skipped device
+case is not device evidence.
+
+## Tests and gates
+
+Commit the test and interface before the copy implementation. The missing
+implementation is the initial compile/link gap. The operator then executes
+the completed seal under a HIP lease and mutates one entry in each of the four
+device tables separately. Every mutation must fail its named comparison;
+restore the table byte-for-byte between runs. Deleting a snapshot copy must
+also fail, since a seal must observe each symbol rather than compare nothing.
+
+Run the focused CPU loader and device-fit tests and the complete host
+preflight. Host builds do not establish HIP correctness. Run the existing
+I-quant numerical device gates with the seal under the operator's lease.
+Independent scoped review and the operator's own gate remain required.
+
+## Risks and stop conditions
+
+A passing tolerance-based dot test can miss an unvisited table entry. The
+byte-exact seal closes only that gap, not gfx1200 performance or the remaining
+formats owned by #1940. Keep that broader issue open.
+Stop if the registered HIP implementation cannot expose its actual device
+symbols, or if source/oracle disagreement requires changing table values.
+
+## Now
+
+ACTIVE: the device-byte seal is specified before its test and implementation.

docs/USAGE.md

@@ -853,8 +853,8 @@ skips with that refusal quoted.
   to inspect or override the architecture-scoped quantizer during
   troubleshooting.
 - On ROCm, GGUF mixture-of-experts checkpoints compute on the quantized
-  expert blocks (Q8_0, Q4_K, Q5_K, Q6_K) instead of being dequantized to
-  bf16 at load time.
+  expert blocks (Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS, IQ3_XXS) instead of being
+  dequantized to bf16 at load time.
 - On ROCm, mixture-of-experts models run the shared-expert gate and both
   expert-combine steps on device. Before these ops were registered the
   engine refused with `no kernel for op` on that path.