From cd16d4d2974667d8506c498fe95d6f896d53d514 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Mon, 13 Jul 2026 04:51:46 +0000 Subject: [PATCH 1/2] Link CUDA runtime, cuBLAS and driver libs into the NIF ggml-cuda.a leaves the CUDA runtime, cuBLAS/cuBLASLt and CUDA driver API symbols unresolved, but the Linux link line only ever added -lstdc++ -lm -lpthread. The resulting .so failed to load at runtime with: Failed to load NIF library: undefined symbol: cuMemCreate (cuMemCreate is a driver API symbol used by ggml-cuda's VMM pool.) Add the CUDA libraries to LDFLAGS whenever the CUDA backend is selected, deriving the toolkit root from nvcc's location rather than hardcoding /usr/local/cuda. The stubs directory is included so -lcuda resolves on build hosts without a driver installed (e.g. release CI); the real libcuda.so.1 is picked up from the driver at load time. Verified on 3x RTX 3090 / CUDA 13.3: LlamaCppEx.devices() now reports the GPU with backend "CUDA" instead of silently falling back to CPU. Co-Authored-By: Claude Opus 4.8 (1M context) --- Makefile | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/Makefile b/Makefile index 2167a9d..e37d3d5 100644 --- a/Makefile +++ b/Makefile @@ -75,6 +75,16 @@ else ifneq ($(shell $(CXX) -fopenmp -E - < /dev/null 2>/dev/null && echo yes),) LDFLAGS += -lgomp endif + # ggml-cuda.a leaves the CUDA runtime, cuBLAS, and driver API unresolved. + # The stubs dir lets -lcuda link on hosts without a driver (e.g. release CI); + # the real libcuda.so.1 is picked up from the driver at load time. + ifneq (,$(filter -DGGML_CUDA=ON,$(CMAKE_FLAGS))) + CUDA_HOME ?= $(patsubst %/bin/nvcc,%,$(shell which nvcc 2>/dev/null)) + ifneq ($(CUDA_HOME),) + LDFLAGS += -L$(CUDA_HOME)/lib64 -L$(CUDA_HOME)/lib64/stubs + endif + LDFLAGS += -lcudart -lcublas -lcublasLt -lcuda + endif endif # CPU count for parallel builds From b01e711d1afcc244eb4e93d978476fb5263b0730 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Mon, 13 Jul 2026 04:51:46 +0000 Subject: [PATCH 2/2] Fix off-by-one corrupting the KV cache on MTP hybrid rollback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On the needs_ckpt path (hybrid models such as Qwen 3.6, where partial seq_rm is unsupported and the recurrent state is snapshot/restored), the rollback re-decode rebuilt the KV from the last n_accepted_total entries of `prompt`. That slice is wrong at both ends: - it omits the token that was `sampled` at the top of the iteration, which is batch element 0 at pos n_past and is NOT yet in the KV, so a real token was silently dropped from the context; and - it includes the final emitted token, which becomes the next iteration's `sampled` and is decoded again as batch element 0 — so that token ended up in the context twice. Every rollback therefore deleted one token and duplicated another. The visible symptom was degenerate output, e.g. Qwen3.6-35B-A3B emitting "when a message arrives arrives arrives arrives ..." for an entire generation, with an inflated acceptance rate (the draft head correctly predicts the repetition it just created). Start the slice one token earlier so the redo batch is [sampled, tok_0 .. tok_{t-2}], leaving the final emitted token to be decoded next iteration. The dense path (native partial seq_rm) was already correct and is unchanged. After this fix, greedy MTP output is invariant to n_draft — the property correct speculative decoding must have — and matches plain decoding exactly on low-entropy prompts. Co-Authored-By: Claude Opus 4.8 (1M context) --- c_src/llama_cpp_ex/llama_nif.cpp | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index 31ffd67..35716c6 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -1794,11 +1794,19 @@ fine::Ok<> generate_mtp_tokens( // Re-decode the accepted tokens on the target so the next // iteration's draft starts from a consistent state. + // + // The KV must be rebuilt with the token that was `sampled` at + // the top of this iteration (batch element 0, at pos n_past), + // followed by all but the LAST emitted token. The last emitted + // token becomes the next iteration's `sampled` and is decoded + // then — including it here would duplicate it in the context. + // `sampled` sits at prompt[size - n_accepted_total - 1], since + // the accept loop pushed n_accepted_total tokens after it. if (n_accepted_total > 0) { llama_batch redo = llama_batch_init(n_accepted_total, 0, 1); for (int i = 0; i < n_accepted_total; i++) { llama_token tok = - prompt[prompt.size() - n_accepted_total + i]; + prompt[prompt.size() - n_accepted_total - 1 + i]; common_batch_add(redo, tok, n_past + static_cast(i), { seq_id },