Skip to content

Link CUDA runtime, cuBLAS and driver libs into the NIF #70

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nyo16 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Link CUDA runtime, cuBLAS and driver libs into the NIF #70

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cd16d4d
Link CUDA runtime, cuBLAS and driver libs into the NIF
nyo16 Jul 13, 2026
b01e711
Fix off-by-one corrupting the KV cache on MTP hybrid rollback
nyo16 Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,16 @@ else
ifneq ($(shell $(CXX) -fopenmp -E - < /dev/null 2>/dev/null && echo yes),)
LDFLAGS += -lgomp
endif
# ggml-cuda.a leaves the CUDA runtime, cuBLAS, and driver API unresolved.
# The stubs dir lets -lcuda link on hosts without a driver (e.g. release CI);
# the real libcuda.so.1 is picked up from the driver at load time.
ifneq (,$(filter -DGGML_CUDA=ON,$(CMAKE_FLAGS)))
CUDA_HOME ?= $(patsubst %/bin/nvcc,%,$(shell which nvcc 2>/dev/null))
ifneq ($(CUDA_HOME),)
LDFLAGS += -L$(CUDA_HOME)/lib64 -L$(CUDA_HOME)/lib64/stubs
endif
LDFLAGS += -lcudart -lcublas -lcublasLt -lcuda
endif
endif

# CPU count for parallel builds
Expand Down
10 changes: 9 additions & 1 deletion c_src/llama_cpp_ex/llama_nif.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1794,11 +1794,19 @@ fine::Ok<> generate_mtp_tokens(

// Re-decode the accepted tokens on the target so the next
// iteration's draft starts from a consistent state.
//
// The KV must be rebuilt with the token that was `sampled` at
// the top of this iteration (batch element 0, at pos n_past),
// followed by all but the LAST emitted token. The last emitted
// token becomes the next iteration's `sampled` and is decoded
// then — including it here would duplicate it in the context.
// `sampled` sits at prompt[size - n_accepted_total - 1], since
// the accept loop pushed n_accepted_total tokens after it.
if (n_accepted_total > 0) {
llama_batch redo = llama_batch_init(n_accepted_total, 0, 1);
for (int i = 0; i < n_accepted_total; i++) {
llama_token tok =
prompt[prompt.size() - n_accepted_total + i];
prompt[prompt.size() - n_accepted_total - 1 + i];
common_batch_add(redo, tok,
n_past + static_cast<llama_pos>(i),
{ seq_id },
Expand Down