Add TML Inkling architecture

[danielhanchen]

Adds support for the Inkling architecture, a Python safetensors-to-GGUF converter, the graph build, and the kernel changes needed for correct and deterministic inference.

1. Had to use int64_t on some ops since large MoEs would go out of index
2. Made a Flash Attention banded attention kernel
3. Checked logits, outputs and long context + needle in a haystack and others
4. And more - will enumerate later
5. Has audio + vision support
6. Testing: Runs GGUFs made at https://huggingface.co/unsloth/inkling-GGUF multimodal well

I tried to keep changes unbreaking - might need to redesign some parts later.

Used AI for kernels, but hand verified and checked everything carefully.

[ggml-gh-bot]

Hi @danielhanchen, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[jaholmesuk]

First: thank you for day-0 support, and it works. We ran inkling UD-Q4_K_XL (587GB) fully resident across two cloud VMs (4x A100 80GB each, 8 total) split over the RPC backend on ordinary Ethernet, and output is coherent and correct (it first-shot passed our hardest structured Terraform generation task). Text only, audio/vision untested here.

We also profiled it, and the data below may be useful for the "might need to redesign some parts later" list. Decode runs at roughly 25 percent of what the hardware gives comparable MoEs, and the evidence points at per-token graph rebuild and launch overhead, not compute.

Numbers (all this PR at ce16fff, same boxes, sustained single-stream, ctx 8192)

Config	Decode t/s	Note
Qwen3-235B UD-Q4, 4x A100 single box	53.2	matches master (53.7), so this PR does not regress existing archs
Qwen3-235B UD-Q4, 8x A100 over RPC	44.5	matches master (45.9), so RPC is healthy
GLM-5.2-744B UD-Q4, 8x A100 over RPC (master, same boxes, previous day)	24.0	the expected class for a ~40B-active MoE at Q4
inkling UD-Q4_K_XL, 8x A100 over RPC, fully resident	6.45	41B active, expected ~GLM-class
inkling UD-Q2_K_XL, 4x A100 single box, fully resident	20.65	isolates RPC from the equation

Decode is flat from empty context to 2.2k depth (6.2 to 6.6), so it is not KV/attention scaling. Single box, inkling runs at ~80 percent of the comparable resident MoE. Over RPC it drops to ~27 percent.

Where the time goes

Per token at 6.45 t/s = 155 ms. During decode:

• All 8 GPUs sit at 0 to 6 percent utilization.
• llama-server burns ~35 percent of one host core.
• The host pushes ~5.5 MB per token to the RPC worker (36 MB/s at 6.5 t/s). Qwen-235B on the same pair moves ~0.75 MB per token.
• The server log reports graphs reused = 0 on every request, on both topologies.
• sched_reserve reports graph nodes = 8985, graph splits = 23, plus a 35 MiB CPU compute buffer (a few ops fall to CPU).
• GGML_CUDA_DISABLE_GRAPHS=1 changes nothing (20.74 vs 20.65 single box), so CUDA graph capture is not engaging either way and every node pays a discrete launch.

perf record on llama-server during decode gives a maximally flat profile: 1,783 distinct symbols, top entry 3.5 percent. Named entries in the top 20: cudaStreamSynchronize / ggml_backend_cuda_synchronize, pthread_mutex_lock under cuLaunchKernel, ggml_backend_sched_split_graph, ggml_gallocr_alloc_graph, and ggml-rpc add_tensor + get_alloc_size (the whole graph re-serialized to the worker every token, which is the 5.5 MB).

So the shape looks like: a ~9k-node graph rebuilt, re-split, re-allocated, re-serialized and launched node by node every token while the GPUs idle. Our guess is the banded rel-bias / shortconv path defeats the llama-side graph reuse, and the node count does the rest. If graph reuse engages for this arch, the RPC number should improve several-fold on its own.

Happy to re-run any branch of this PR on the 2-node A100 cluster or the single box and report the same measurements. Full logs, perf.data and exact commands available on request.

[MatioLK]

Performance data on gfx1151 (Strix Halo, ROCm) — decode degrades linearly with context depth.

Ran a depth sweep with llama-bench -d on Inkling-Small UD-IQ2_M (76.76 GiB), since CUDA-only testing may not surface this.

Setup: AMD Ryzen AI Max+ 395 (gfx1151), 128 GB LPDDR5-8000, Ubuntu 26.04 (in-tree amdgpu), ROCm 7.14.0 in container, built from 1e6f9e4 with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151.

test	t/s	ms/token
tg32	20.96	47.7
tg32 @ d4096	8.86	112.9
tg32 @ d16384	3.36	297.6
pp512	121.94	—
pp512 @ d4096	127.89	—
pp512 @ d16384	75.10	—

The decode slope is constant — roughly 15.5 us of added latency per token of context:

   0 -> 4096  :  +65.2 ms / 4096  = 15.9 us/token
4096 -> 16384 : +184.7 ms / 12288 = 15.0 us/token

So it's O(n), not O(n^2) — but the constant seems high for this architecture. With sliding_window=512 on most layers, I'd expect decode to stay closer to flat as depth grows, with only the global layers scaling. Rough estimate: even if all 42 layers read their full KV window, that's under 1 us/token at this memory bandwidth, so the cost looks compute-bound in the banded bias rather than bandwidth-bound.

Possibly GGML_OP_FLASH_ATTN_EXT_BANDED isn't exploiting rel_extent_swa to bound the work per layer? I haven't read the kernel closely enough to say.

Practical impact: a 19.8k-token prompt with ~2k output takes about 18.5 min end-to-end on this hardware.

Two other notes for anyone testing on Strix Halo:

• --load-mode none (or --no-mmap) is required — without it, loading stalls badly past ~64 GB (looks like Rocm slow model weight loading past ~64GB mark on Strix Halo (all fine with Vulkan) #15018).
• -ub 256 is needed to avoid the ???? output issue reported earlier in this thread; the sweep above uses it, which may account for the modest prefill numbers.

Happy to run additional configurations if useful.

[MatioLK]

Three findings from testing on gfx1151 (Strix Halo, ROCm) that may be useful — in particular, the ???? output issue does not appear to be a ubatch problem.

Setup: AMD Ryzen AI Max+ 395 (gfx1151), 128 GB LPDDR5-8000, Ubuntu 26.04 (in-tree amdgpu), ROCm 7.14.0 in a container, built from 1e6f9e4 with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151. Model: Inkling-Small UD-IQ2_M.

---

1. ???? output is caused by n_slots > 1, not by ubatch size

The workaround discussed earlier in this thread (-ub 256) did not help in my case. What did:

-np 1 --no-kv-unified

Reproduction is trivial and does not need a long prompt — an 18-token request is enough:

llama-server -m Inkling-Small-UD-IQ2_M-00001-of-00003.gguf -ngl 99 -fa 1 -ub 256 --load-mode none -c 32768 --jinja

Default n_slots = 4, kv_unified = true:

{"role":"assistant","content":"????????????????????????"}

Same server, same build, same model, with -np 1 --no-kv-unified:

{"role":"assistant","content":"Hello! How can I help you today? 😊"}

Notes that may narrow it down:

• llama-cli was always clean, at any prompt length — it runs a single slot.
• Broken output occurs at 18 input tokens, so the ~260-token final chunk boundary reported earlier isn't the mechanism, at least not the only one.
• -fa 0 does not avoid it either; only slot count does.
• Generation speed is normal while producing ???? — the model is computing at full rate, just emitting garbage.

Since the architecture carries per-layer short convolution state (shortconv_kernel = 4), my guess is that this recurrent state isn't isolated per slot, so slots overwrite each other. I haven't read the implementation, so that's speculation — but the -np 1 fix is reproducible.

---

2. -fa 0 is ~3x faster than -fa 1 at depth on this backend

Depth sweep with llama-bench -d:

test	-fa 1	-fa 0
tg32	20.96	—
tg32 @ d4096	8.86	—
tg32 @ d16384	3.36	13.85
pp512 @ d16384	75.10	118.39

With FA on, the decode slope is roughly 15.5 us of added latency per token of context, and it is linear:

   0 -> 4096  :  +65.2 ms / 4096  = 15.9 us/token
4096 -> 16384 : +184.7 ms / 12288 = 15.0 us/token

So O(n), not O(n^2), but the constant looks high. With sliding_window = 512 on most layers I'd expect only the global layers to scale with depth. Rough estimate: even if all 42 layers read their full KV window each step, that's under 1 us/token at this memory bandwidth — so the cost looks compute-bound inside the banded bias rather than bandwidth-bound. Possibly GGML_OP_FLASH_ATTN_EXT_BANDED isn't using rel_extent_swa to bound per-layer work?

Real-world, 8421-token prompt via /v1/chat/completions:

-fa 1 : prompt 102.2 t/s, decode  5.36 t/s
-fa 0 : prompt 111.3 t/s, decode 16.04 t/s

---

3. But -fa 0 degrades output quality noticeably

This part is subjective, so treat it accordingly — but the difference was consistent and large enough to report.

Same 8421-token prompt, same seed-free sampler settings, same preset except flash-attn. Task: summarise a document into a structured factual account. With -fa 1 the output covered the central facts. With -fa 0 the same prompt produced output that:

• omitted five of the six key facts present in the input (including the one uncontested admission, the date of the event, and the named witness),
• was ~40% shorter in substance,
• in one run, invented a witness who does not appear in the input, along with a fabricated relationship and a paragraph of testimony,
• in the reasoning trace, hallucinated an input detail ("broken glass") that isn't in the prompt and then built the whole analysis around contradicting it.

I don't have a numerical benchmark for this — no perplexity runs — so it may be worth someone reproducing on CUDA before drawing conclusions. But if the non-FA path is numerically diverging rather than just being a different code path, that seems worth knowing, since -fa 0 is currently the only way to get usable decode speed at depth on this hardware.

---

Also, for anyone else testing on Strix Halo: --load-mode none (or --no-mmap) is required, otherwise loading stalls badly past ~64 GB — looks like #15018.

Happy to run additional configurations or provide full logs if useful.

[ServeurpersoCom]

Heads up, #29042 makes the saver write the SWA pattern, so once it lands this architecture no longer needs to be excluded from llama_model_saver_supports_arch and can get the test-llama-archs roundtrip.