Link CUDA runtime, cuBLAS and driver libs into the NIF#70
Open
nyo16 wants to merge 2 commits into
Open
Conversation
ggml-cuda.a leaves the CUDA runtime, cuBLAS/cuBLASLt and CUDA driver API
symbols unresolved, but the Linux link line only ever added
-lstdc++ -lm -lpthread. The resulting .so failed to load at runtime with:
Failed to load NIF library: undefined symbol: cuMemCreate
(cuMemCreate is a driver API symbol used by ggml-cuda's VMM pool.)
Add the CUDA libraries to LDFLAGS whenever the CUDA backend is selected,
deriving the toolkit root from nvcc's location rather than hardcoding
/usr/local/cuda. The stubs directory is included so -lcuda resolves on
build hosts without a driver installed (e.g. release CI); the real
libcuda.so.1 is picked up from the driver at load time.
Verified on 3x RTX 3090 / CUDA 13.3: LlamaCppEx.devices() now reports the
GPU with backend "CUDA" instead of silently falling back to CPU.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the needs_ckpt path (hybrid models such as Qwen 3.6, where partial
seq_rm is unsupported and the recurrent state is snapshot/restored), the
rollback re-decode rebuilt the KV from the last n_accepted_total entries
of `prompt`. That slice is wrong at both ends:
- it omits the token that was `sampled` at the top of the iteration,
which is batch element 0 at pos n_past and is NOT yet in the KV, so a
real token was silently dropped from the context; and
- it includes the final emitted token, which becomes the next
iteration's `sampled` and is decoded again as batch element 0 — so
that token ended up in the context twice.
Every rollback therefore deleted one token and duplicated another. The
visible symptom was degenerate output, e.g. Qwen3.6-35B-A3B emitting
"when a message arrives arrives arrives arrives ..." for an entire
generation, with an inflated acceptance rate (the draft head correctly
predicts the repetition it just created).
Start the slice one token earlier so the redo batch is
[sampled, tok_0 .. tok_{t-2}], leaving the final emitted token to be
decoded next iteration. The dense path (native partial seq_rm) was
already correct and is unchanged.
After this fix, greedy MTP output is invariant to n_draft — the property
correct speculative decoding must have — and matches plain decoding
exactly on low-entropy prompts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ggml-cuda.a leaves the CUDA runtime, cuBLAS/cuBLASLt and CUDA driver API symbols unresolved, but the Linux link line only ever added -lstdc++ -lm -lpthread. The resulting .so failed to load at runtime with:
(cuMemCreate is a driver API symbol used by ggml-cuda's VMM pool.)
Add the CUDA libraries to LDFLAGS whenever the CUDA backend is selected, deriving the toolkit root from nvcc's location rather than hardcoding /usr/local/cuda. The stubs directory is included so -lcuda resolves on build hosts without a driver installed (e.g. release CI); the real libcuda.so.1 is picked up from the driver at load time.
Verified on 3x RTX 3090 / CUDA 13.3: LlamaCppEx.devices() now reports the GPU with backend "CUDA" instead of silently falling back to CPU.