From c04b380cbf334c93ec3a01b626b41a4c0984d155 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Thu, 9 Jul 2026 13:28:57 -0400 Subject: [PATCH] Bump llama.cpp to a646006 (b9932) for v0.8.34 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Submodule cb295bf59 -> a646006f0 (44 commits). Header diffs in the range are additive only (Q2_0 quant enums, common_speculative_init_result helper, server_base param) — no NIF changes required. Full suite against the rebuilt NIF: 191 tests, 0 failures. Also documents speculative-type support status (MTP only) in README and the MTP moduledoc, including measured DFlash-on-Metal results at b9932. --- CHANGELOG.md | 22 ++++++++++++++++++++++ README.md | 6 ++++++ lib/llama_cpp_ex/mtp.ex | 8 ++++++++ mix.exs | 2 +- vendor/llama.cpp | 2 +- 5 files changed, 38 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d00c031..f3eb84d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # Changelog +## v0.8.34 + +Maintenance release: llama.cpp bump to b9932. Full suite against the rebuilt +NIF: 191 tests, 0 failures. + +### Changed + +- **llama.cpp submodule** — Updated from cb295bf59 to a646006f0 (44 commits, tag b9932). No NIF changes were required: the header diffs in this range are additive only — new `GGML_TYPE_Q2_0`/`GGML_FTYPE_MOSTLY_Q2_0`/`LLAMA_FTYPE_MOSTLY_Q2_0` enum values in `ggml.h`/`llama.h`, a new `common_speculative_init_result` helper in `common/speculative.h`, and a `server_base` field plus `` include in `common/common.h`. + - **llama core**: fix allowed decreasing positions in a sequence in llama-batch (#25449); add `n_keep_tail` in `split_equal` for recurrent models (#25278); refactor fused ops (#24646); fix quantized KV cache for dsv4 (#25202); fix OOB reads in the UGM tokenizer's `precompiled_charsmap` handling (#18750). + - **speculative**: fix out-of-bounds read in ngram-map on prompt shrink (#23936); fix draft-model fit vs load inconsistency in the server (#25056); naming/spacing cleanup (#25410). + - **ggml**: add Q2_0 quantization type definition + CPU backend (#24448); CPU f16→f16 `GGML_OP_SET_ROWS` (#25344); fix A-indexing in the simd_gemm scalar tail-column path (#25390); make `ggml_time_init` idempotent (#24422); better default thread count on ppc/AIX (#25237). + - **Metal**: add `set_rows` with f16 src0 (#25434); add `col2im_1d` op for f32/f16/bf16 (#25176). + - **CUDA**: f16→f16 `SET_ROWS` (#25367); fuse MMVQ post-scale for NVFP4 (#24481); remove `-sm row`, refactor cuBLAS (#24216). + - **Vulkan**: disable FA `mask_opt` on GCN (#24362); reduce submission threshold on small AMD GPUs by CU count (#25240); guard unimplemented f16 `SET_ROWS` (#25351). + - **OpenCL**: ragged-tile MoE prefill FP16 GEMM optimization (#25433); flash-attention decode perf (#25366); fix potential crash in aos reconstruct (#25383). + - **SYCL/HIP/hexagon**: eight SYCL commits (col2im_1d, cross-entropy-loss ops, argsort coverage, AOT double fix, env-var renames); HIP `-fno-finite-math-only` alongside `-ffast-math` (#25373); hexagon VTCM layouts + pipeline improvements for MUL_MAT/MUL_MAT_ID/FLASH_ATTN_EXT (#25425). + - **tools/server/ui** (not linked into the binding): llama-cli moved to an HTTP-based implementation (#24948); SSE replay-buffer follow-up (#25047); timings/progress in `/responses` API streams (#25348); prompt-cache RAM limit enforcement (#25070); fix `load_models()` deadlock (#25358); context-usage gauge in the web UI (#25340). + +### Added + +- **Speculative-decoding docs** — README and `LlamaCppEx.MTP` moduledoc now state which upstream speculative types the binding exposes (MTP only) and document DFlash status on Apple Silicon: functional end-to-end on Metal at b9932 via upstream tools, but measured slower than plain decoding at small target sizes (Qwen3.5-4B + 0.6B drafter on M4 Max: 42 tok/s vs 85 tok/s plain, 30% acceptance greedy), with community drafter-GGUF conversions still incompatible across converter revisions (#25116, #25110). + ## v0.8.33 Performance and robustness release for the batching `Server` and generation paths diff --git a/README.md b/README.md index 305f851..7f9de43 100644 --- a/README.md +++ b/README.md @@ -494,6 +494,12 @@ Multi-Token Prediction speculative decoding (upstream PR [#22673](https://github > > Larger `n_draft` hurts on Metal because verify cost grows faster than acceptance benefit. On NVIDIA, `n_draft: 3` is the right default — that's what the upstream 2× number assumes. +### Other speculative types (EAGLE-3, DFlash, n-gram) + +Upstream llama.cpp implements more speculative types behind the same `common_speculative` API — `draft-eagle3`, `draft-dflash` (block-diffusion drafting via a separate drafter GGUF), and several n-gram self-speculation modes. **This binding currently exposes only MTP**: `MTP.init/2` pins `COMMON_SPECULATIVE_TYPE_DRAFT_MTP` and builds both contexts from the same model, so there is no way to load a separate drafter model yet. + +> **DFlash status (July 2026, llama.cpp b9932).** DFlash runs end-to-end on Metal via upstream `llama-cli`/`llama-server`, but we measured it *slower* than plain decoding on Apple Silicon at small target sizes: Qwen3.5-4B target + z-lab 0.6B drafter on M4 Max reached 42 tok/s with DFlash vs 85 tok/s plain (greedy sampling; 30% draft acceptance, mean accepted run 2.8 — and stochastic sampling at `temp 0.8` collapses acceptance to ~7%). The Metal economics are the same as the MTP note above (wide verify batches are expensive), and the community drafter-GGUF conversions are still churning: of three third-party Qwen 4B drafter repos tested, only one loads with current upstream (the others hit the `dflash-draft` arch mismatch [#25116](https://github.com/ggml-org/llama.cpp/issues/25116) or lack the `target_layers` metadata added by the conversion refactor [#25110](https://github.com/ggml-org/llama.cpp/pull/25110)). Worth revisiting when the drafter format settles; the natural entry point is a `spec_type` + drafter-model option on `speculative_init`. + ### Models with MTP heads - [`ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/ggml-org/Qwen3.6-35B-A3B-MTP-GGUF) (recommended: `Q4_K_M`, ~21 GB) diff --git a/lib/llama_cpp_ex/mtp.ex b/lib/llama_cpp_ex/mtp.ex index 6c35e30..ce807ff 100644 --- a/lib/llama_cpp_ex/mtp.ex +++ b/lib/llama_cpp_ex/mtp.ex @@ -30,6 +30,14 @@ defmodule LlamaCppEx.MTP do that — a single MTP session decodes one sequence at a time. Reuse the same `%MTP{}` value across calls to `stream/3` / `generate/3` to avoid rebuilding the contexts; KV caches are cleared on each call. + + MTP is the only speculative type this binding exposes. Upstream llama.cpp + also implements EAGLE-3, DFlash (block-diffusion drafting via a separate + drafter GGUF), and n-gram self-speculation behind the same + `common_speculative` API, but the NIF pins the MTP type and both contexts + are built from the same model, so a separate drafter model cannot be loaded + yet. See the "Speculative decoding" section of the README for the current + status of DFlash on Apple Silicon. """ alias LlamaCppEx.{Context, Model, Sampler, Tokenizer} diff --git a/mix.exs b/mix.exs index 2b4e41c..662ecbe 100644 --- a/mix.exs +++ b/mix.exs @@ -37,7 +37,7 @@ end defmodule LlamaCppEx.MixProject do use Mix.Project - @version "0.8.33" + @version "0.8.34" @source_url "https://github.com/nyo16/llama_cpp_ex" def project do diff --git a/vendor/llama.cpp b/vendor/llama.cpp index cb295bf..a646006 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit cb295bf59663cd3577389315636772f4060bd1f5 +Subproject commit a646006f09d2f76f2d62d6c0d5e8e8490d570720