DiffusionGemma by danielhanchen · Pull Request #24423 · ggml-org/llama.cpp

danielhanchen · 2026-06-10T15:56:37Z

Worked on prelim Diffusion Gemma support!

Has normal chat similar to llama-cli via llama-diffusion-cli -cnv -n 2048
Has a visualization method to show diffusion live via llama-diffusion-cli -cnv -n 2048 --diffusion-visual

To try this PR:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
gh pr checkout 24423
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release --target llama-diffusion-cli

then use a GGUF (any can work but for eg)

pip install -U "huggingface_hub[cli]"
hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
    --local-dir unsloth/diffusiongemma-26B-A4B-it-GGUF \
    --include "*Q8_0*" # Use "*Q4_K_M*" for a smaller 16 GB download

then use chat or visualization:

./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -cnv -n 2048

or

./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -cnv -n 2048 --diffusion-visual

Example below (a bit blurry to limit to 10MB on Github :()

Disclaimer Heavy usage of AI, but verified logits matching with transformers, checked FP16 vs FP32 KV cache, long context checks and much more

Some diffusion cli and visual updates

ggml-gh-bot · 2026-06-10T16:01:39Z

Hi @danielhanchen, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

pwilkin · 2026-06-10T16:38:47Z

Oof, that's a big one.

There's a ton of debugging stuff left in there that needs throwing out, for one. I'm also not convinced about the idea to make a server just for one model - I think if we're intending to support diffusion models in a server mechanism, it should be a general diffusion-server (but that's just my opinion, probably have to wait for what @ggerganov thinks about this one).

danielhanchen · 2026-06-10T16:42:31Z

Haha sorry - this PR was more of a direct translation / proof of concept that it works!

danielhanchen · 2026-06-10T16:42:44Z

I'll edit the PR - sorry we're juggling multiple things haha

gaugarg-nv · 2026-06-10T16:43:20Z

Another PR for DiffusionGemma: #24427

CISC · 2026-06-10T16:50:14Z

You have some failing tests to fix. :)

coder543 · 2026-06-10T16:50:37Z

if we're intending to support diffusion models in a server mechanism, it should be a general diffusion-server

With a block diffusion model, couldn't the regular server just return each block when it is finished diffusing? It would be nice to just have one server, and the API could remain fully compatible so clients don't need to be aware that they're dealing with a diffusion model. (We don't show the distribution of sampled logits for AR models, and I don't see why people would need to see the intermediate diffusion steps either, since those won't be useful.)

quasar-of-mikus · 2026-06-10T16:56:48Z

Doesn't build on Windows:

[421/433] Building CXX object examples\diffusion\CMakeFiles\llama-diffusion-cli.dir\diffusion-cli.cpp.obj
FAILED: examples/diffusion/CMakeFiles/llama-diffusion-cli.dir/diffusion-cli.cpp.obj
ccache C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\Llvm\x64\bin\clang-cl.exe  /nologo -TP -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_CPU -DGGML_USE_CUDA -DLLAMA_SHARED -D_CRT_SECURE_NO_WARNINGS -IC:\Textgen\llama.cpp\src\..\include -IC:\Textgen\llama.cpp\ggml\src\..\include -IC:\Textgen\llama.cpp\common\. -IC:\Textgen\llama.cpp\common\..\vendor /DWIN32 /D_WINDOWS /EHsc /O2 /Ob2 /DNDEBUG -std:c++17 -MD /utf-8 /bigobj /showIncludes /Foexamples\diffusion\CMakeFiles\llama-diffusion-cli.dir\diffusion-cli.cpp.obj /Fdexamples\diffusion\CMakeFiles\llama-diffusion-cli.dir\ -c -- C:\Textgen\llama.cpp\examples\diffusion\diffusion-cli.cpp
C:\Textgen\llama.cpp\examples\diffusion\diffusion-cli.cpp(10,10): fatal error: 'sys/ioctl.h' file not found
   10 | #include <sys/ioctl.h>
      |          ^~~~~~~~~~~~~
1 error generated.

danielhanchen · 2026-06-10T16:57:01Z

Yep will fix haha - I also added a short GIF of it working edited in description!

…s, drop debug hooks - guard sys/ioctl.h behind _WIN32 and add a GetConsoleScreenBufferInfo fallback for the visual viewport size, so diffusion-cli builds on Windows - skip diffusion-gemma in test-llama-archs like gemma4 (shared ISWA backbone, no synthetic fixture params yet) - remove the DG_DUMP_KV_LAYER / DG_NSWA debug scaffolding and its llama.h API - fix flake8 E306 in conversion/diffusion_gemma.py

stepfunction83 · 2026-06-10T19:53:44Z

I was able to compile it successfully on Linux for my 4090, but when running it, I get the following error after sending a user message:

0.13.483.785 E ggml_cuda_compute_forward: SOFT_MAX failed
0.13.483.796 E CUDA error: invalid argument
0.13.483.798 E   current device: 0, in function ggml_cuda_compute_forward at /home/LLM/DiffusionGemma/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:3163
0.13.483.798 /home/LLM/DiffusionGemma/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:103: CUDA error
E   err
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(+0x1c1ab)[0x7bb8567401ab]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x21c)[0x7bb85674062c]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x15b)[0x7bb85674080b]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-cuda.so.0(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xb7)[0x7bb853260997]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-cuda.so.0(+0x27a810)[0x7bb85327a810]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x817)[0x7bb85675def7]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7bb855ee04c1]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x114)[0x7bb855ee2c94]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6encodeERK11llama_batch+0x240)[0x7bb855ee6c80]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(llama_decode+0xf)[0x7bb855eec1ef]
./build/bin/llama-diffusion-cli(+0x1941c)[0x5a66e697341c]
./build/bin/llama-diffusion-cli(+0x8143)[0x5a66e6962143]
./build/bin/llama-diffusion-cli(+0x60e4)[0x5a66e69600e4]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7bb85562a1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7bb85562a28b]
./build/bin/llama-diffusion-cli(+0x6a45)[0x5a66e6960a45]
Aborted (core dumped)

The command I'm running is:

 CUDA_VISIBLE_DEVICES="" ./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \
  -ngl 99 -cnv -n 2048 --system-prompt-file sysprompt.txt \
  --diffusion-eb auto \
  --diffusion-eb-max-steps 48 \
  --diffusion-eb-t-max 1.0 \
  --diffusion-eb-t-min 0.6 \
  --diffusion-visual

To compile it, I used:

rm -rf build
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=$(which nvcc)
cmake --build build -j --config Release --target llama-diffusion-cli

quasar-of-mikus · 2026-06-10T20:06:24Z

Yep will fix haha - I also added a short GIF of it working edited in description!

Builds and runs on Windows now.
1x 3090 Q4KM: time per step: 326.13ms
2x 3090 Q8_0: time per step: 878.83ms

arkham000 · 2026-06-10T20:10:32Z

is it only cli at this point? no llama-server ?

kroaton · 2026-06-10T20:16:25Z

Thank you for putting this together! An Issue I found is that --fit doesn't work with this PR.

icedream · 2026-06-10T21:54:09Z

Test run on my system with AMD hardware (7900 XTX) in it, Q4_K_M - time per step: 364.45ms

Screencast_20260610_234032_c.webm

lucasbinder · 2026-06-10T21:57:15Z

@icedream What (equivalent) tokens per second are you getting? I also tried running it with an AMD GPU (R9700 w/ vulkan) and only got ~27t/s.

icedream · 2026-06-10T22:24:59Z

@lucasbinder Not 100% sure if that's the right way to calculate it but based on two more runs with the same prompt, calculating with 256 tokens per full canvas diffused (I left out the last canvas as tail end of response), taking the start/end timings per canvas from the --verbose output:

Run 1

2.327213 - 9.803998 = 7.476785 = 34.24 t/s
10.119182 - 17.995429 = 7.876247 = 32.50 t/s
18.412472 - 23.738067 = 5.325595 = 48.07 t/s
24.261660 - 32.754977 = 8.493317 = 30.14 t/s
33.345178 - 51.575057 = 18.229879 = 14.04 t/s

Run 2

2.328457 - 9.777839 = 7.449382 = 34.37 t/s
10.092776 - 17.935576 = 7.8428 = 32.64 t/s
18.349154 - 23.654470 = 5.305316 = 48.25 t/s
24.176041 - 32.655410 = 8.479369 = 30.19 t/s
33.245244 - 51.415716 = 18.170472 = 14.09 t/s

(Also I should clarify I used ROCm, not Vulkan in my case so that may be influencing the performance as well.)

gbgh1 · 2026-06-11T01:49:10Z

Unofficial prebuilt binaries for anyone who wants to test this PR without setting up a CUDA toolchain:

https://github.com/gbuznote-beep/llama-diffusion-cli-prebuilt

Linux x86_64 / WSL2 — CUDA 12.8, sm_86 (RTX 30-series / A4000–A6000), glibc ≥ 2.39, self-contained (cudart/cublas/cublasLt/nccl bundled)
Windows x64 — CPU-only
Pinned to c84e85a; SHA256SUMS + reproducible build scripts included (other GPU archs rebuild in ~10 min)

Data points from testing (256 tokens, EB sampler): A5000 full-GPU 0.98 s/step; RTX 3070 Ti Laptop 8 GB via WSL2 (-ngl 99 --n-cpu-moe 22) 5.9 s/step; i7-12700H CPU-only 17.1 s/step. Output quality looks coherent (thinking-style drafts + self-critique). Thanks @danielhanchen for the implementation!

corporatepiyush · 2026-07-18T20:15:25Z

what are we waiting for ?

any unresolved issues which are not discussed in this thread ?

Refreshing the ggml-org#25731 pin was not enough on its own. It and ggml-org#24523 conflict with each other in common/chat.cpp and src/llama-arch.h: both append a new llm_arch value immediately before LLM_ARCH_UNKNOWN and both add a chat parser plus a detection block in the same regions. Whichever entry is applied second conflicts, and reordering does not help. #40 carries ggml-org#24523 with ggml-org#25731 merged into it and the conflict resolved, keeping every function and dispatch entry from both sides. It also adapts Inkling to the renamed thinking_end_tags API, so it builds against b10133 and newer. Replaying the resolver sequence on b10133 with these pins: OK ggml-org ggml-org#24423 @ c3fb972 OK ggml-org ggml-org#24523 @ a58a7fa6e OK unslothai #40 @ fee66f6a8 Drop the #40 entry and repin ggml-org#25731 upstream once either PR lands.

ggml-org ggml-org#24523 now places LLM_ARCH_MINIMAX_M3 next to LLM_ARCH_MINIMAX_M2 instead of at the tail of the enum, matching how llama-arch.cpp already groups the name table. That removes the src/llama-arch.h half of the collision with ggml-org#25731, which appends LLM_ARCH_INKLING at the tail. ggml-org ggml-org#25731 has picked up current master and the thinking_end_tags rename, so it now builds on its own against b10133 and newer. Both head SHAs moved, so the pins and the carrier branch are refreshed. unslothai #40 now only resolves what is left: both PRs still add a chat parser and a detection block in the same region of common/chat.cpp, immediately before "namespace workaround {". There is no principled alternative anchor there, since chat.cpp has no MiniMax-M2 parser to sit beside, so that half stays with the carrier. Replaying the resolver sequence on b10133: OK ggml-org ggml-org#24423 @ c3fb972 OK ggml-org ggml-org#24523 @ baee0f5 OK unslothai #40 @ 233cedb

#39 landed the first pin refresh only, so master still points at commits that have since moved on and that conflict with each other. ggml-org ggml-org#24523 was force-pushed: it now places LLM_ARCH_MINIMAX_M3 next to LLM_ARCH_MINIMAX_M2 rather than at the tail of the enum, matching how llama-arch.cpp already groups the name table. That removes the src/llama-arch.h half of its collision with ggml-org#25731, which appends LLM_ARCH_INKLING at the tail. New head baee0f5. ggml-org ggml-org#25731 has picked up current master and the thinking_end_tags rename, so it now builds on its own against b10133 and newer. Its entry is replaced by unslothai #40, which carries what is still unresolvable between the two: both add a chat parser and a detection block in the same region of common/chat.cpp, immediately before "namespace workaround {". chat.cpp has no MiniMax-M2 parser to sit beside, so there is no principled alternative anchor for that half. Replaying the resolver sequence on b10133: OK ggml-org ggml-org#24423 @ c3fb972 OK ggml-org ggml-org#24523 @ baee0f5 OK unslothai #40 @ 233cedb Drop the #40 entry and repin ggml-org#25731 upstream once either PR lands. Co-authored-by: Daniel Han <unslothai@gmail.com>

Firstbober · 2026-08-01T13:10:27Z

What's the status of this PR? All checks are passing, and I don't see any code issues to be addressed and acted upon?

nonetrix · 2026-08-10T20:24:45Z

Thanks for still working on this even though it's been a while since weights dropped :D

# Conflicts: # common/arg.cpp

Prebuilt: repin ggml-org#24423 and ggml-org#25731 onto carry branches

Ports ggml-org/llama.cpp#24423 at head daca8075 onto the 3f545be bump. The PR is open and unmerged; drop the patch when it lands. - diffusion-gemma arch, entropy-bound decoder, --diffusion-eb* flags, gguf-py diffusion keys, CUDA sampling kernel (not built on Metal) - examples/diffusion and examples/diffusion-gemma-server relocated to tools/ so the shipped LLAMA_BUILD_EXAMPLES=OFF build produces llama-diffusion-cli, llama-diffusion-gemma-server and llama-diffusion-gemma-visual-server alongside llama-server; nothing is fetched at build or run time - two fork-local fixups: -no-cnv re-targeted for the narrowed upstream example set; visual server parses via common_json for chat.h Verified on diffusiongemma-26B-A4B-it-Q4_K_M: CLI (-n 2048, and --diffusion-eb off) produce coherent text, visual server answers two consecutive requests from one resident process, otool -L clean.

didlawowo · 2026-09-10T07:27:23Z

Hi — I operate DiffusionGemma behind an OpenAI-compatible HTTP API with a resident model process on an RTX 3090. I saw this PR now contains a native DiffusionGemma server path, so I do not want to duplicate that work.
I would like to contribute a small, well-scoped follow-up. Would a generic diffusion-server direction, an HTTP regression test for consecutive requests on a resident process, or a reproducible CUDA issue/fix be most helpful?
I’m happy to first validate the current branch against our deployment and follow the project’s contribution and AI-disclosure requirements.

danielhanchen · 2026-09-16T01:20:22Z

@didlawowo Yes go ahead!

ServeurpersoCom · 2026-09-17T17:34:10Z

Heads up, #29042 makes the saver write the SWA pattern, so once it lands this architecture no longer needs to be excluded from llama_model_saver_supports_arch and can get the test-llama-archs roundtrip.

diffusion-visual updates

c5fe75b

Some diffusion cli and visual updates

danielhanchen requested review from a team, CISC and am17an as code owners June 10, 2026 15:56

github-actions Bot added model Model specific examples python python script changes labels Jun 10, 2026

danielhanchen marked this pull request as draft June 10, 2026 15:58

danielhanchen changed the title ~~diffusion-visual updates~~ DiffusionGemma Jun 10, 2026

github-actions Bot added the testing Everything test related label Jun 10, 2026

danielhanchen mentioned this pull request Jun 10, 2026

diffusion_studio: serve DiffusionGemma in Unsloth Studio with the optimized visual decoder unslothai/unsloth-zoo#748

Merged

pearsonkyle mentioned this pull request Jun 10, 2026

Add DiffusionGemma (Gemma-4 MoE block-diffusion) support: MoE imatrix variants, AWQ MoE groups, llama.cpp bump, diffusion eval backend pearsonkyle/Quant-Tuner#4

Draft

danielhanchen mentioned this pull request Jul 19, 2026

Repin PR #25731 to current head after force-push unslothai/llama.cpp#36

Merged

shanevcantwell mentioned this pull request Jul 23, 2026

GGUF loading via llama.cpp — wrap Unsloth GGUF + diffusion-cli for ComfyUI node shanevcantwell/ComfyUI-DiffusionGemma#131

Open

danielhanchen mentioned this pull request Aug 4, 2026

Report git's stderr when a merge fails without leaving conflicts unslothai/llama.cpp#63

Merged

danielhanchen added 2 commits August 10, 2026 10:39

Fix merge conflicts

1d87a1b

Fix merge conflicts

daca807

Fix merge conflicts

ec5fbb4

# Conflicts: # common/arg.cpp

danielhanchen added a commit to unslothai/llama.cpp that referenced this pull request Aug 26, 2026

Merge pull request #109 from unslothai/prset-repin-24423-25731

1bf8dc3

Prebuilt: repin ggml-org#24423 and ggml-org#25731 onto carry branches

Fix merge conflicts

843e05a

danielhanchen force-pushed the diffusion-visual-updates branch from 3a16140 to 843e05a Compare September 4, 2026 08:11

tests: make diffusion-gemma testable instead of excluded

c6f8d60

Fix merge conflicts

168b21a

iodeh referenced this pull request in iodeh/unsloth-llama.cpp Sep 14, 2026

Prebuilt: repin #24423 and #25731 onto carry branches

9527d8b

Fix merge conflicts

12e0a96

ServeurpersoCom mentioned this pull request Sep 17, 2026

Model-Saver: Write the SWA pattern, 15 more architectures roundtrip #29042

Open

Conversation

danielhanchen commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggml-gh-bot Bot commented Jun 10, 2026

Uh oh!

pwilkin commented Jun 10, 2026

Uh oh!

danielhanchen commented Jun 10, 2026

Uh oh!

danielhanchen commented Jun 10, 2026

Uh oh!

gaugarg-nv commented Jun 10, 2026

Uh oh!

CISC commented Jun 10, 2026

Uh oh!

coder543 commented Jun 10, 2026

Uh oh!

quasar-of-mikus commented Jun 10, 2026

Uh oh!

danielhanchen commented Jun 10, 2026

Uh oh!

stepfunction83 commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

quasar-of-mikus commented Jun 10, 2026

Uh oh!

arkham000 commented Jun 10, 2026

Uh oh!

kroaton commented Jun 10, 2026

Uh oh!

icedream commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lucasbinder commented Jun 10, 2026

Uh oh!

icedream commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbgh1 commented Jun 11, 2026

Uh oh!

corporatepiyush commented Jul 18, 2026

Uh oh!

Firstbober commented Aug 1, 2026

Uh oh!

nonetrix commented Aug 10, 2026

Uh oh!

didlawowo commented Sep 10, 2026

Uh oh!

danielhanchen commented Sep 16, 2026

Uh oh!

ServeurpersoCom commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

danielhanchen commented Jun 10, 2026 •

edited

Loading

stepfunction83 commented Jun 10, 2026 •

edited

Loading

icedream commented Jun 10, 2026 •

edited

Loading

icedream commented Jun 10, 2026 •

edited

Loading