[pull] master from mudler:master - #1504
Merged
Merged
Conversation
…6df7d2ef28fa612` (#11842) ⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
…a31cbaf871daba1` (#11846) ⬆️ Update NVIDIA/NeMo-Speech.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
…2c4e9535c094b` (#11829) ⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries. Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests. Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* fix(ds4): build CUDA kernels for the target architecture The ds4 backend compiled its CUDA objects with no -arch. Upstream's Makefile leaves CUDA_ARCH empty and its `cuda` target refuses to build without one, offering `cuda-spark` (sm_121) and `cuda-generic` (native) instead. We invoke its object targets directly, which bypasses that guard, so nvcc fell back to its default architecture and the kernels ran as JIT'd PTX on the real GPU. On GB10 (sm_121) that silently corrupted inference: any prompt over roughly 128 tokens produced text unrelated to the input and never closed its thinking block, so content came back empty and the chat showed only reasoning; longer prompts failed with "cuda decode failed". It also cost close to two orders of magnitude of prefill throughput. Measured on one box, same model, same prompt, same GPU, upstream ds4 at the pinned commit, differing only in the nvcc flags: make -B ds4 (archless, as we build it) garbage output 4.21 t/s make cuda-spark (compute_121a/sm_121a) correct output 325.70 t/s Select an architecture list from CUDA_MAJOR_VERSION, which the backend matrix already declares for both ds4 cublas entries but Dockerfile.ds4 never forwarded. Upstream's CUDA_ARCH takes a single value, so it cannot express the fat binary these images need; NVCC_ARCH_FLAGS is overridden instead, since a command-line assignment wins over its `:=`. The lists are copied from vllm-cpp rather than invented so the two CUDA images cover the same GPUs, with l4t/arm64 covering Orin, Thor and GB10. An empty CUDA_MAJOR_VERSION keeps upstream's `native` behaviour for local developer builds, and no CI runner has a GPU to enumerate. DS4_CUDA_HAVE_MXF4 is deliberately left unset: upstream defines it only for single-arch sm_120/sm_121 builds and guards it with a plain #ifdef rather than __CUDA_ARCH__, so it cannot be combined with older archs. It gates an optional MXFP4 indexer fast path whose #ifndef branch returns 0 and falls back cleanly, so omitting it costs speed on GB10, not correctness. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Claudio Maradonna <git@codeshifter.xyz> * test(ds4): cover the multi-batch prefill regression The architecture fix has no automated guard: every existing e2e spec uses a short prompt, and the miscompiled backend answered short prompts correctly. The corruption only appears once a prompt spans more than one prefill batch, so the whole suite passed against a backend that produced garbage in normal use. Add an opt-in "long_prefill" capability to the backend e2e suite that sends a prompt well past one batch with a known needle and asserts the answer still reflects it, and document in the ds4 guide why the build must never omit an nvcc architecture, how to check which flags a configuration resolves to without compiling, and how to run the new spec. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Claudio Maradonna <git@codeshifter.xyz> --------- Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )