Ptq1/vulkan port - #32
Open
LaurentZuijdwijk wants to merge 5 commits into
Open
LaurentZuijdwijk wants to merge 5 commits into
LaurentZuijdwijk wants to merge 5 commits into
Conversation
…rnels Ports GGML_TYPE_PTQ1_0 (143) from PrismML-Eng/llama.cpp, the packing used by prism-ml/Ternary-Bonsai-2-27B-gguf. The format is upstream TQ1_0's base-3 trit packing at group 128 instead of 256: block_ptq1_0 is 28 B per 128 weights (24 B qs at 5 trits/byte + 2 B qh at 4 trits/byte + fp16 scale) = 1.75 bpw. Covered: ggml type + ftype plumbing, the CPU reference codec and vec_dot, and the full Vulkan path (dequant, get_rows, mul_mat_vec f32/f16, matmul and matmul_id including subgroup and coopmat1 variants) -- 50 SPIR-V shaders. coopmat2 is skipped deliberately: dequant_funcs_cm2.glsl has no PTQ1_0 decoder and emitting cm2 for it fails the whole Vulkan build, so it falls back to the scalar and coopmat1 paths. PTQ1_0 has no arch-specific vec_dot on any target, so arch-fallback.h routes every architecture to the generic one -- including x86, which is not covered by the tq1_0 fallback list. PQ2_0 (142) is not ported: it has no Vulkan kernels in the source fork either. Known incomplete: the prism.hadamard.* weight-rotation runtime is NOT ported yet. Bonsai weights are stored in a rotated basis, so a Bonsai GGUF currently loads and reports its ftype correctly but generates garbage. The activation transform (llama-model.cpp metadata + validation, the build_lora_mm hook in llama-graph.cpp, and the token_embd inverse) is the remaining piece; the FWHT primitive and llama_mul_mat_hadamard already exist here from the KV work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K3NeoeVLYx9mNqMSEJHRvS
Completes the Bonsai path. Bonsai weights are stored in a blockwise-Hadamard
rotated basis (W x = (WQ)(Q^T x), Q folded into the weights offline), so the
runtime must apply the matching transform to activations or the model loads
cleanly and emits fluent garbage -- which is exactly what the previous commit
did on its own.
Ported from PrismML-Eng/llama.cpp:
- llama-model.cpp: prism.hadamard.* metadata parsing and validation, plus
construction of the per-(block_size, buffer type) rotation matrices and the
per-width explicit sign vectors, after buffers are allocated.
- llama-graph.{h,cpp}: llama_hadamard_transform, the transform hook in
build_lora_mm and build_lora_mm_id (memoized per (activation, rotation) so
weights sharing an input build it once), and the inverse applied to the
token-embedding lookup, whose table stores rotated rows.
- llama-context.{h,cpp}: llama_verify_hadamard_graph, run once on the first
pristine graph. It throws if any folded weight reaches a matmul without its
transform, so an unsupported graph fails loudly instead of computing wrong
results.
- llama-model-loader.cpp: the three string-key get_arr/get_key instantiations
the metadata reads need.
Deviations from the source fork: LLM_ARCH_DSPARK is Prism-private and its three
special cases are dropped; the verified-architecture list keeps QWEN35 and the
rest, so a Bonsai GGUF (arch qwen35) is accepted.
No new kernels were needed -- llama_mul_mat_hadamard and fwht.comp already live
here from the KV-rotation work, and the Vulkan backend picks the FWHT kernel off
the GGML_HINT_SRC0_IS_HADAMARD hint.
Verified on Ternary-Bonsai-2-27B-PTQ1_0 (Radeon 8060S, RADV STRIX_HALO): loads,
reports 401 folded weights, and answers correctly on both CPU and Vulkan.
llama-bench matches the source fork within noise -- pp512 210.13 vs 206.38,
tg128 3.96 vs 4.00 -- so the port is faithful.
Known performance gap, not introduced here: tg128 is ~22 GiB/s effective against
~193 GiB/s for IQ3_XXS on the same box and build. PTQ1_0 decode is bound by the
serial dependent-multiply trit unpack in ptq1_0_trit (up to 4 iterations per
element, trip count varying by element position, so it diverges within a
subgroup), not by bandwidth. A LUT or position-unrolled decoder is the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K3NeoeVLYx9mNqMSEJHRvS
ptq1_0_trit walked to trit n with a loop of up to four dependent v = (v*3) & 0xFF steps, and the trip count is a function of element position, so lanes in a subgroup ran different iteration counts. Mod-256 multiplication is associative, so that walk is just b * 3^n mod 256. This is already how dequantize_row_ptq1_0 in ggml-quants.c decodes -- only the GLSL kept the serial form. 3^n is computed with selects rather than a const array, because a dynamically indexed local array can land in scratch on some drivers and this is the hot path of every ternary matmul. Equivalence was checked exhaustively over the whole input domain: all 256 byte values x 5 trit positions agree with the loop form, and all 3^5 = 243 encoder codes round-trip. Applied to both copies -- ptq1_0.glsl (mul_mat_vec, get_rows, mul_mm) and the standalone decoder in dequant_ptq1_0.comp. NOT YET VALIDATED ON GPU. The shaders compile and the arithmetic is proven equivalent offline, but nothing has been run on the Radeon 8060S since the change, so there is no measured speedup and no on-device correctness check yet. Baseline to beat, from the same box and build: pp512 210.13, tg128 3.96, which is ~22 GiB/s effective against ~193 GiB/s for IQ3_XXS on the same hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K3NeoeVLYx9mNqMSEJHRvS
Assisted-by: Marshall
Upstream replaced the per-type CREATE_MM/CREATE_MM2 pipeline macros with keyed pipeline creation and generic non-LUT quantization loops. Keep that architecture and re-register the dedicated PTQ1_0 shaders through the new per-type macros (CM1, CM1_ID, SG, SG_ID, SG_ID_SUB, SG_FP32, SG_ID_FP32, SG_ID_SUB_FP32), preserving PTQ1_0 support in both MUL_MAT and MUL_MAT_ID. The PTQ1_0 packed 8-wide loader is kept in the refactored load_a_to_shmem, and shader generation still emits the 8-wide LOAD_VEC_A per-type variants. Coopmat2 remains unsupported for PTQ1_0 (dequant_funcs_cm2.glsl has no PTQ1_0 entry), so those shaders are skipped instead of failing the build. Assisted-by: Marshall
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimize Vulkan PTQ1_0 decoding and matrix-vector multiplication by reusing packed quantization bytes instead of decoding them repeatedly.
• Add a PTQ1_0-specific matvec path using eight lanes per quantization block: six for packed qs words, one for the qh tail, and one inactive.
• Reuse activation loads across output rows and apply the block scale after accumulating each lane’s contribution.
• Preserve the existing reduction, batch/expert offsets, and fused-output handling.
• Extend PTQ test coverage and classify PTQ1_0 under the existing ternary quantization error limits.
• Add strict CPU dot-product checks against dequantized values to test arithmetic independently of quantization loss.
Benchmarks
Measured against the preceding local PTQ1_0 implementation on the same machine using Vulkan0.
End-to-end inference
Model: Ternary-Bonsai-2-27B-PTQ1_0.gguf
▎ sh
▎ llama-bench
▎ -m Ternary-Bonsai-2-27B-PTQ1_0.gguf
▎ -dev Vulkan0 -ngl 99
▎ -p 512 -n 128 -r 3
The generation improvement comes with an observed prompt-processing regression; its cause has not yet been isolated.
Matrix-vector kernel latency
PTQ1_0 weights, F32 input, one input vector. Dimensions are output rows × input columns.
These results compare the byte-reuse optimization with the preceding PTQ implementation, not with upstream master or another quantization format.
Validation
• Full configured Release/Vulkan build passed, including server, tools, and tests.
• All 45 focused PTQ backend correctness cases passed.
• All four model-shape correctness cases passed.
• All 16 selected CTest regression tests passed.
• Six strict PTQ dot-product checks passed at lengths 128, 256, 384, 4096, 5120, and 17408.
• git diff --check passed.
AI assistance
Implementation and validation were assisted by Marshall.