(AI policy violation) Metal: TurboQuant GPU dequant kernels + host buffer type

README.md

@@ -30,6 +30,86 @@ LLM inference in C/C++
 
 ----
 
+## TurboQuant KV Cache (This Fork)
+
+> **This fork extends [PR #21089](https://github.com/ggerganov/llama.cpp/pull/21089) (elusznik's CPU TurboQuant) with two optimizations that improve speed and quality.**
+
+TurboQuant compresses the KV cache using Householder rotation + Lloyd-Max codebooks, enabling **3.9–5.2× memory compression** with minimal quality loss. On a 96GB Mac Studio, Qwen3-32B goes from **278K → 1.45M token context**.
+
+### Benchmark: Quality (Perplexity)
+
+Qwen3-32B Q4_K_M, wikitext-2, Mac Studio M3 Ultra 96GB, `-ngl 99`, 8 chunks:
+
+| KV Type | Origin | PPL ↓ | vs F16 | Bits/elem | Compression |
+|---------|--------|-------|--------|-----------|-------------|
+| F16 | baseline | 5.798 | — | 16.00 | 1.0× |
+| Q8_0 | baseline | 5.797 | −0.01% | 8.50 | 1.9× |
+| Q4_0 | baseline | 5.857 | +1.0% | 4.50 | 3.6× |
+| **TBQ4_0** | **TurboQuant** | **5.872** | **+1.3%** | **4.06** | **3.9×** |
+| **TBQ3_0** | **TurboQuant** | **6.068** | **+4.7%** | **3.06** | **5.2×** |
+
+### Benchmark: Speed (Prompt Evaluation)
+
+| KV Type | Prompt Eval (t/s) | vs F16 | Notes |
+|---------|-------------------|--------|-------|
+| F16 | 278 | — | baseline |
+| Q8_0 | 269 | −3% | |
+| Q4_0 | 271 | −3% | |
+| **TBQ4_0** | **228** | **−18%** | Householder dequant overhead (CPU) |
+| **TBQ3_0** | **228** | **−18%** | same as TBQ4_0 |
+
+Speed overhead is due to the Householder matrix-vector multiply during dequantization, currently running on CPU via Apple Accelerate. A Metal SET_ROWS kernel would eliminate this penalty.
+
+### Benchmark: Max Context Length (96GB Mac Studio)
+
+| Model | Weights | F16 | Q4_0 | TBQ4_0 | TBQ3_0 |
+|-------|---------|-----|------|--------|--------|
+| Llama-3.1-8B (Q4_K_M) | ~5 GB | 696K | 2.47M | **2.74M** | **3.64M** |
+| Qwen3-32B (Q4_K_M) | ~19 GB | 278K | 988K | **1.10M** | **1.45M** |
+| Llama-3.3-70B (Q4_K_M) | ~40 GB | 164K | 583K | **646K** | **857K** |
+| Qwen2.5-72B (Q4_K_M) | ~43 GB | 154K | 548K | **606K** | **805K** |
+
+### Our Optimizations (On Top of PR #21089)
+
+1. **Apple Accelerate `cblas_sgemv`** (`95fb4fe`) — 2× faster dequant on M-series via AMX coprocessor
+2. **L2 norm correction** (`c2621e3`) — re-normalize unit vector after dequant, halves PPL gap vs F16
+
+### Experimental: QJL Error Correction
+
+See branch [`experimental/qjl-error-correction`](https://github.com/SeKondBrainAILabs/llama.cpp-turboquant/tree/experimental/qjl-error-correction) for TBQP4_0 and TBQP3_0 types that add 1-bit QJL (Quantized Johnson-Lindenstrauss) sign correction. Results: QJL reduces 2-bit PPL by 28%, but straight codebook types (TBQ3_0, TBQ4_0) remain more efficient at equivalent bit-rates.
+
+### Usage
+
+```bash
+cmake -B build -DGGML_METAL=ON
+cmake --build build -j
+
+# Perplexity benchmark
+./build/bin/llama-perplexity \
+    -hf bartowski/Qwen2.5-32B-Instruct-GGUF:Q4_K_M \
+    -f wiki.test.raw -ctk tbq4_0 -ctv tbq4_0 --chunks 8 -ngl 99
+
+# Server with mixed K/V (best quality/compression trade-off)
+./build/bin/llama-server \
+    -hf bartowski/Qwen2.5-32B-Instruct-GGUF:Q4_K_M \
+    -ctk tbq4_0 -ctv tbq3_0 --host 0.0.0.0 --port 8080 -ngl 99
+```
+
+### Known Limitations
+
+- **Speed**: 18% slower prompt eval due to CPU-based Householder dequant. Needs Metal kernel.
+- **CPU-only KV cache**: TBQ types require KV cache on CPU (no Metal SET_ROWS kernel yet).
+- **Model compatibility**: Qwen2.5-VL-7B broken with any KV quant below Q8_0 (upstream issue).
+
+### Roadmap
+
+- [ ] **Metal SET_ROWS kernel**: eliminate speed penalty by moving dequant to GPU
+- [ ] **Cross-architecture validation**: benchmark on Llama, Mistral, Gemma
+- [ ] **Needle-in-a-haystack eval**: verify retrieval quality at 500K+ tokens
+- [ ] **Upstream PR**: submit Accelerate + norm correction to PR #21089
+
+----
+
 ## Quick start
 
 Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

common/arg.cpp

@@ -387,6 +387,8 @@ const std::vector<ggml_type> kv_cache_types = {
     GGML_TYPE_IQ4_NL,
     GGML_TYPE_Q5_0,
     GGML_TYPE_Q5_1,
+    GGML_TYPE_TBQ3_0,
+    GGML_TYPE_TBQ4_0,
 };
 
 static ggml_type kv_cache_type_from_str(const std::string & s) {

ggml/include/ggml.h

@@ -429,7 +429,9 @@ extern "C" {
         GGML_TYPE_MXFP4   = 39, // MXFP4 (1 block)
         GGML_TYPE_NVFP4   = 40, // NVFP4 (4 blocks, E4M3 scale)
         GGML_TYPE_Q1_0    = 41,
-        GGML_TYPE_COUNT   = 42,
+        GGML_TYPE_TBQ3_0  = 42, // TurboQuant 3-bit
+        GGML_TYPE_TBQ4_0  = 43, // TurboQuant 4-bit
+        GGML_TYPE_COUNT   = 44,
     };
 
     // precision
@@ -467,6 +469,8 @@ extern "C" {
         GGML_FTYPE_MOSTLY_MXFP4   = 25, // except 1d tensors
         GGML_FTYPE_MOSTLY_NVFP4   = 26, // except 1d tensors
         GGML_FTYPE_MOSTLY_Q1_0    = 27, // except 1d tensors
+        GGML_FTYPE_MOSTLY_TBQ3_0  = 28, // except 1d tensors
+        GGML_FTYPE_MOSTLY_TBQ4_0  = 29, // except 1d tensors
     };
 
     // available tensor operations:

ggml/src/CMakeLists.txt

@@ -205,6 +205,9 @@ add_library(ggml-base
             ggml-threading.h
             ggml-quants.c
             ggml-quants.h
+            ggml-turboq.c
+            ggml-turboq.h
+            ggml-turboq-tables.h
             gguf.cpp)
 
 set_target_properties(ggml-base PROPERTIES
@@ -213,6 +216,15 @@ set_target_properties(ggml-base PROPERTIES
 )
 
 target_include_directories(ggml-base PRIVATE .)
+
+# Link Accelerate framework for TurboQuant cblas_sgemv on Apple Silicon
+if (APPLE)
+    find_library(ACCELERATE_FRAMEWORK Accelerate)
+    if (ACCELERATE_FRAMEWORK)
+        target_link_libraries(ggml-base PRIVATE ${ACCELERATE_FRAMEWORK})
+    endif()
+endif()
+
 if (GGML_BACKEND_DL)
     target_compile_definitions(ggml-base PUBLIC GGML_BACKEND_DL)
 endif()

ggml/src/ggml-common.h

@@ -277,6 +277,22 @@ typedef struct {
 } block_tq2_0;
 static_assert(sizeof(block_tq2_0) == sizeof(ggml_half) + QK_K / 4, "wrong tq2_0 block size/padding");
 
+// TurboQuant blocks
+
+// 3.0625 bpw
+typedef struct {
+    uint8_t qs[QK_K * 3 / 8];
+    ggml_half d;
+} block_tbq3_0;
+static_assert(sizeof(block_tbq3_0) == sizeof(ggml_half) + QK_K * 3 / 8, "wrong tbq3_0 block size/padding");
+
+// 4.0625 bpw
+typedef struct {
+    uint8_t qs[QK_K / 2];
+    ggml_half d;
+} block_tbq4_0;
+static_assert(sizeof(block_tbq4_0) == sizeof(ggml_half) + QK_K / 2, "wrong tbq4_0 block size/padding");
+
 //
 // Super-block quantization structures
 //

ggml/src/ggml-cpu/arch-fallback.h

@@ -19,6 +19,8 @@
 #define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
 #define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
 #define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 #define ggml_vec_dot_q2_K_q8_K_generic ggml_vec_dot_q2_K_q8_K
 #define ggml_vec_dot_q3_K_q8_K_generic ggml_vec_dot_q3_K_q8_K
 #define ggml_vec_dot_q4_K_q8_K_generic ggml_vec_dot_q4_K_q8_K
@@ -71,6 +73,9 @@
 #define ggml_gemm_q8_0_4x4_q8_0_generic ggml_gemm_q8_0_4x4_q8_0
 #define ggml_gemm_q8_0_4x8_q8_0_generic ggml_gemm_q8_0_4x8_q8_0
 #elif defined(__aarch64__) || defined(__arm__) || defined(_M_ARM) || defined(_M_ARM64)
+// quants.c
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 // repack.cpp
 #define ggml_quantize_mat_q8_K_4x4_generic ggml_quantize_mat_q8_K_4x4
 #define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
@@ -84,6 +89,8 @@
 // quants.c
 #define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
 #define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 // repack.cpp
 #define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
 #define ggml_quantize_mat_q8_K_4x4_generic ggml_quantize_mat_q8_K_4x4
@@ -117,6 +124,8 @@
 #define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
 #define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
 #define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 #define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
 // repack.cpp
 #define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
@@ -160,6 +169,8 @@
 #define quantize_row_q8_K_generic quantize_row_q8_K
 #define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
 #define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 #define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
 #define ggml_vec_dot_mxfp4_q8_0_generic ggml_vec_dot_mxfp4_q8_0
 #define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
@@ -203,6 +214,8 @@
 #define ggml_gemm_q8_0_4x8_q8_0_generic ggml_gemm_q8_0_4x8_q8_0
 #elif defined(__riscv)
 // quants.c
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 #define ggml_vec_dot_nvfp4_q8_0_generic ggml_vec_dot_nvfp4_q8_0
 #define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
 // repack.cpp
@@ -248,6 +261,8 @@
 #define ggml_vec_dot_q1_0_q8_0_generic ggml_vec_dot_q1_0_q8_0
 #define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
 #define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 #define ggml_vec_dot_q2_K_q8_K_generic ggml_vec_dot_q2_K_q8_K
 #define ggml_vec_dot_iq2_xxs_q8_K_generic ggml_vec_dot_iq2_xxs_q8_K
 #define ggml_vec_dot_iq2_xs_q8_K_generic ggml_vec_dot_iq2_xs_q8_K
@@ -298,6 +313,8 @@
 #define ggml_vec_dot_q4_1_q8_1_generic ggml_vec_dot_q4_1_q8_1
 #define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
 #define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_tbq3_0_q8_K_generic ggml_vec_dot_tbq3_0_q8_K
+#define ggml_vec_dot_tbq4_0_q8_K_generic ggml_vec_dot_tbq4_0_q8_K
 #define ggml_vec_dot_iq2_xxs_q8_K_generic ggml_vec_dot_iq2_xxs_q8_K
 #define ggml_vec_dot_iq2_xs_q8_K_generic ggml_vec_dot_iq2_xs_q8_K
 #define ggml_vec_dot_iq2_s_q8_K_generic ggml_vec_dot_iq2_s_q8_K

ggml/src/ggml-cpu/ggml-cpu.c

@@ -396,6 +396,18 @@ static const struct ggml_type_traits_cpu type_traits_cpu[GGML_TYPE_COUNT] = {
         .vec_dot_type             = GGML_TYPE_Q8_K,
         .nrows                    = 1,
     },
+    [GGML_TYPE_TBQ3_0] = {
+        .from_float               = quantize_row_tbq3_0,
+        .vec_dot                  = ggml_vec_dot_tbq3_0_q8_K,
+        .vec_dot_type             = GGML_TYPE_Q8_K,
+        .nrows                    = 1,
+    },
+    [GGML_TYPE_TBQ4_0] = {
+        .from_float               = quantize_row_tbq4_0,
+        .vec_dot                  = ggml_vec_dot_tbq4_0_q8_K,
+        .vec_dot_type             = GGML_TYPE_Q8_K,
+        .nrows                    = 1,
+    },
     [GGML_TYPE_I32] = {
         .from_float               = (ggml_from_float_t) ggml_cpu_fp32_to_i32,
     },