feat: TurboQuant KV cache (TBQ3_0/TBQ4_0) with CPU + CUDA support - #21

Merged

marksverdhei merged 10 commits into

htfrom

feat/turboquant-kv-cache

Mar 31, 2026

Commits on Mar 31, 2026

feat: welcome agentic contributions, remove upstream AI restrictions (#6)

marksverdhei
and
claude
committed
aed1606
webui: add cancel button for in-progress model loading (#8)

marksverdhei
and
claude
committed
7ce7e86
feat: add CPU TurboQuant KV cache types

elusznik
authored and
marksverdhei
committed
94dcdbe
ggml : limit the first TurboQuant CPU PR to TBQ

elusznik
authored and
marksverdhei
committed
39974a3
fix(turboq): add malloc NULL checks for memory safety

marksverdhei
and
claude
committed
1410041
chore(turboq): remove unused QJL projection scaffolding

marksverdhei
and
claude
committed
4847f38
feat(turboq): add CUDA backend support for TBQ3_0/TBQ4_0 KV cache types

marksverdhei
and
claude
committed
5a76777
fix(turboq): add CUDA SET_ROWS support and disable graph capture for TBQ

marksverdhei
and
claude
committed
ca3f79b
perf(turboq): fused on-device SET_ROWS kernel, re-enable CUDA graphs

marksverdhei
and
claude
committed
2254e22
fix(turboq): free d_turboq_norms buffer in cleanup to prevent memory leak

marksverdhei
and
claude
committed
c9b66da