Qwen3.8-27B MTP: the flag was free the whole time

One llama.cpp flag unlocks +33% to +145% decode speed for Qwen3.8-27B on consumer GPUs, depending on the card, and the knobs the community mapped push further still. No new files, no conversion, no custom build. The MTP head already ships inside the GGUF you downloaded on launch night.

Opened hours after the Aug 14 2026 release. Within four days the community grew it into a living record: 53 configurations, 40 contributors, a decade of silicon from 2016 Pascal to Blackwell workstation cards plus Apple Silicon and an unlocked mining card, three quant makers, and seven tuning rules nobody knew at launch, see Community numbers.

The flag

--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1

Full launch command

llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  -c 131072 -ngl 999 -fa 1 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 --parallel 1

The KV cache flags matter on their own: without them, context creation fails past roughly 90K next to 17GB of weights. With them, the full 262K window fits a 24GB card at 22.2GB (drop -c to 262144 and remove the spec flags if you want maximum window instead of maximum speed).

Weights: unsloth/Qwen3.8-27B-GGUF. Official model: Qwen/Qwen3.8-27B.

Tuning quickstart

Start with the launch command above, --spec-draft-n-max 2. Then sweep n-max 2-4 for your card, and on bandwidth-poor rigs try --spec-draft-p-min 0.60-0.75, per rules 1 and 2 below. Every contributor's full sweep lives in sweeps/.

The seven rules the community found

Discovered by contributors in the table across the first three days, detailed in the sections and footnotes below:

1. The n-max sweet spot is card- and topology-dependent. 24GB cards peak at n-max 2, bigger or faster cards at 3-4, and switching split mode moves it too, so re-sweep after any config change (@lingster, @Jackwwg83).
2. --spec-draft-p-min helps starved cards and hurts fast ones. The ~0.60 confidence gate makes deep drafting nearly free on bandwidth-poor rigs (@tomertec), and inverts on desktop Blackwell, where three independent RTX 5090s ran fastest ungated. Acceptance is a vanity metric there: gating raised it and lowered throughput (@taco-devs, @paulomcg, @jcr211). Sweep it, don't adopt it.
3. The gain scales with generation length where overhead dominates. Short generations can pay more than they win (@Spadav); on rigs whose baseline is already bandwidth-bound the full gain shows at 400 tokens and length adds nothing (@Jackwwg83).
4. On multi-GPU boxes, fix the split mode before touching spec flags. The default --split-mode layer serializes decode; tensor was +68% on its own on a 5060 Ti pair, and the two levers stack to 3.14x (@Jackwwg83).
5. Speculative decode is a single-stream optimization. The advantage is gone by --parallel 4, and a --parallel 2 BASELINE reads ~20% low, which inflates your gain claim, so measure both arms at --parallel 1 (@Jackwwg83, @paulomcg).
6. Rebuild llama.cpp before you tune anything, and re-derive whole flag stacks per card class. A config that pays on a bandwidth-starved APU inverted below baseline on a 960 GB/s card (@Splizard). Upstream is optimizing this arch weekly: a current build was +10-15% on every quant before any flag (@taco-devs), and a fresh 3090 baseline now equals the day-one with-flag number (@hauntedhost).
7. A shared desktop halves everything, silently. A live compositor and browser spilled 3.5 GB of weights to host RAM over PCIe while /health stayed green; decode halved with no error anywhere. Bench headless, or verify the weights are actually resident (mem_info_gtt_used or your vendor's equivalent) before trusting a number (@Splizard).

How it works

Qwen trained multi-token-prediction (nextn) layers into Qwen3.8. The quantizers kept them: unsloth's GGUFs carry the blk.*.nextn.* tensors, which llama.cpp loads and, without the flag, ignores. llama.cpp added draft-mtp speculative decoding in PR #22673 (July 2026): the server drafts tokens with the built-in head and verifies them with the main model, so accepted drafts cost a fraction of a full forward pass. Everything was in place on release night. The flag connects it.

Caveats

• --parallel 1 for measurement, always: some hosts do serve with higher parallel, but the spec advantage is gone by 4 concurrent streams, and a parallel-2 baseline corrupts the A/B (see rule 5)
• prompt processing takes a small hit from device-to-host embedding transfers
• these are day-one llama.cpp speeds through the qwen3_5 code path, the hybrid attention kernels are young and the floor should rise with upstream work
• your absolute numbers will differ with hardware, drivers, and thermals, the deltas are the durable part

Measure it yourself

probe.py is the streaming client behind every number here. It clocks every generated token (reasoning and content deltas both) against a live server and prints per-prompt medians.

python3 probe.py                 # defaults to http://127.0.0.1:8080
python3 probe.py http://127.0.0.1:8090

Run it once against a baseline serve and once with the flag, same everything otherwise. That pairing is the whole method.

Community numbers

Ran the A/B on your card? Open a PR and add a row.

Card	Baseline	With flag	n-max	Acceptance	Contributor
RTX 3090 24GB	31.0	41.3	2	0.78	@sudoingX
RTX 5090 mobile 24GB	36.7	50.9	2	0.79	@sudoingX
RTX 4090 24GB	47.7	76.3	2	0.56	@Spadav_
RTX A6000 48GB (Ada)	26.7	52.5	2	0.54-0.98	@lingster
RX 7900 XTX 24GB	30.7	43.9	2	0.60-0.95	@Jqianggu
2× RTX 3090 + 3090 Ti 24GB (TP)	49.1	81.1	2	0.52-0.96	@guilhermedemelocabral
RTX 4090 24GB (UD-Q4_K_XL)	36.1	74.8	2	0.56-0.94	@rkvhtd
2x RX 9070 16GB (Vulkan)	22.1	41.6	2	0.73	@tomertec
AMD Radeon AI PRO R9700 32GB	27.0	43.3	2	0.60-0.94	@ajnytebot
Ryzen AI Max+ 395 / Radeon 8060S	11.5	23.7	2	0.52-0.94	@shiwuxiu
AMD Radeon 890M iGPU (Strix Point) 48GB UMA	2.7	5.7	2	0.59-0.91	@davidglogan
RTX PRO 6000 Blackwell 96GB	63.8	91.5	2	0.72-0.81	@commdata2338
RTX 5090 32GB (desktop)	61.4	135.0	4	0.38-0.87	@taco-devs
RTX 5090 32GB (Q6_K, 128K)	61.9	130.0	2	0.52-0.95	@hypertectonic
RTX 5090 32GB (Q6_K, 256K)	62.0	121.7	2	0.50-0.95	@hypertectonic
RTX 3090 Ti 24GB (Q4_K_M, 128K)	42.0	60.9	2	0.47-0.93	@hypertectonic
RTX 3090 Ti 24GB (Q4_K_M, 256K)	41.2	61.4	2	0.55-0.94	@hypertectonic
RTX 5090 32GB desktop (UD-Q5_K_XL)	66.3	144.2	4	0.32-0.89	@TrickRiggin
3× RTX 3060 12GB (layer split)	17.3	24.5	8	0.87	@EamonMcKiernan05
AMD Radeon 9060 XT 16 GB	15.2	28.7	2	0.62-0.93	@kdrapel
RTX 3090 24GB (turboquant, n-max 6)	39.8	61.5	6	0.61-0.90	@NicholaiVogel
RTX 3090 24GB (b10450)	41.3	63.5	2	0.69-0.87	@hauntedhost
GMK EVO-X2, Ryzen AI Max+ 395 (64GB unified, ROCm/HIP)	10.5-11.1	21.4-22.2	12	0.95-1.0	@KyaniteLabs
2× RTX 5060 Ti 16GB (PP, default -sm layer)	22.1	42.8	2	0.53-0.94	@Jackwwg83
2× RTX 5060 Ti 16GB (TP, -sm tensor)	37.1	65.9	2	0.51-0.88	@Jackwwg83
RTX 5090 32GB (desktop)	62.7	108.7	3	0.72	@jcr211
RTX 5090 32GB (desktop)	69.3	129.1	4	0.55	@paulomcg
RX 7900 GRE 16GB (Vulkan, packed)	28.7 *	47.8 avg (36.6–53.8)	3	0.87–0.96 (avg ~0.93)	@lsunay (Hermes agent on PC-12)
RTX 5090 32GB (Q4_K_M, 131K)	76.9	155.5	2	0.50-0.95	@anstaendig
RTX 5090 32GB (UD_Q4_K_XL, 262K)	74.3	179.7	4	0.35–0.91	@anstaendig
RTX PRO 6000 Blackwell Max-Q 96GB	45.7	97.1	2	0.52-0.95	@awilliamson
2× Tesla P40 24GB (tensor split)	11.7	22.6	4	0.68-0.87	@lyesrock
RTX 5090 32GB (UD-Q5_K_XL, 262K)	69.6	147.5	6	0.69-0.87	@lyesrock
2× RTX 5060 Ti 16GB (TP, Q4_K_M)	38.3	76.0	3	0.43-0.93	@mgoswick
RTX 5090 32GB (UD-Q4_K_XL, 262K, q4_0 KV)	76.3	171.7	4	0.69-0.87	@lyesrock
2× Tesla P40 24GB (UD-Q4_K_XL, 262K, q4_0 KV)	13.2	23.4	4	0.82	@lyesrock
RX 7900 XTX 24GB (Vulkan/RADV)	28.8	70.7	3	0.43-0.95	@Splizard
RTX 3090 24GB (UD-Q2_K_XL, OC)	52.4	85.6	2	0.76	@dcrey7
RTX 3090 24GB (UD-Q4_K_XL, OC)	43.9	79.6	2	0.78	@dcrey7
RTX 3060 Ti 8GB + RTX 3080 Ti 12GB (TP, -sm tensor)	39.4	49.2	2	0.895	@randyjohnson11
Apple M4 24GB (Metal)	5.8	5.8	2	0.48-0.95	@sternryan
Mac Studio (M3 Ultra, 96GB UMA, Q6_K, Metal)	22.8	24.2	2	0.53-0.95	@adityavsingh
RTX 3090 24GB (Q4_K_M, 131K, 250W)	31.9	52.8	4	0.65	@ctaylor83
RTX 5090 32GB (UD-Q4_K_XL, 131K, q4_0 KV)	74.4	182.0	4	0.38-0.90	@cmoro-deusto
RTX 5080 16GB + RTX 3090 24GB (TP)	55.6	92.9	2	0.51-0.95	@plyra
RTX 5060 Laptop 8GB	4.2	7.3	2	0.75-0.97 (0.90 aggregate)	@ENY66n
RTX PRO 3000 Blackwell Laptop 12GB	5.3	10.8	2	0.67-0.93	@ENY66n
RTX 3080 10GB (UD-IQ2_XXS, 16K, headless)	45.1	64.4	2	0.44-0.93	@ntindle
NVIDIA CMP 170HX 64GB (unlocked HBM2e)	33.0	46.7	2	0.53-0.94	@shiwuxiu
RTX 4090 24GB	47.0	68.8	2	—	@snakeyjay63-png
RTX 4090 24GB (Purusha, LoRA merge)	47.0	83.4	2	—	@snakeyjay63-png
Tesla V100 32GB (Volta)	33.3	48.0	2	0.75	@cameron
RTX 5060 Ti 16GB (Q4-XYZ-v2, 32K)	26.3	59.5	4	0.34-0.69	@jaisusx

* A6000 row: unsloth Q8_K_XL, 256K context, q8_0 KV cache — 40.0 GB VRAM baseline, 41.4 GB with spec (rows above: Q4_K_M, 131K, q4_0 KV). * RX 7900 XTX row: unsloth Q4_K_M, 131K context, q4_0 KV cache — 18.9 GB VRAM baseline, 19.7 GB with spec. * 3×24GB TP row: two RTX 3090 + one 3090 Ti, Unsloth UD-Q6_K_XL, tensor-parallel --split-mode tensor, --parallel 4, 500K unified KV pool, q8_0 KV, f16 draft KV, mmproj Q8, temp 1.0. VRAM ~16.6 GB/GPU baseline, ~18.7 GB/GPU with spec (tightest card). Method: stock probe.py. --parallel 1 was not required on this host. * RTX 4090 row: unsloth UD-Q4_K_XL, 160K context, q4_0 KV cache, llama.cpp b10360, Windows/CUDA, 275W power limit — 20.6 GB VRAM baseline, 21.9 GB with spec. * RX 9070 row: two 16GB cards on one 31.84 GiB pool, Vulkan build b10426, unsloth UD-Q4_K_XL, 262K context, q8_0 KV cache — 28.0 GiB across the pool with spec. Method differs from the rows above and is spelled out under the sweep below. * R9700 row: unsloth UD-Q4_K_XL, 262K context, q4_0 KV cache, llama.cpp b10433, Vulkan/RADV — 22.53 GB VRAM baseline, 24.55 GB with n-max 2. Method: unchanged probe.py at commit 67c20536, three runs x three prompts, thinking off. * Ryzen AI Max+ 395 row: 64GB unified memory, unsloth UD-Q4_K_XL, 32K context, q8_0 KV cache, llama.cpp b10437, Windows build 26200, Vulkan with AMD driver 32.0.31035.1003. Method: unchanged probe.py at commit 67c2053, three runs x three prompts, thinking off. * RTX 4090 Spadav_ row: unsloth Q4_K_M, 200K context, q4_0 KV cache, q8_0 draft KV, mmproj loaded (888MB on GPU) — method: stock probe.py + 4096-token curl (MTP crossover: overhead dominates at ≤400 tokens, +60% at 4096 tokens). * Radeon 890M row: Ryzen AI 9 HX 370 (Strix Point, gfx1150), 48 GB UMA carve of 96 GB DDR5, unsloth UD-Q4_K_XL, 131K context, q4_0 KV cache, llama.cpp Vulkan/RADV, Ubuntu 26.04 — 19.15 GB VRAM baseline, 20.13 GB with n-max 2. Method: unchanged probe.py, three runs x three prompts, thinking off. Note this is a different APU class from the Ryzen AI Max+ 395 row above: Strix Point 890M is 16 CUs on a 128-bit bus, Strix Halo 8060S is 40 CUs on 256-bit, and this row holds 131K context resident against that row's 32K — both differences push this baseline down. * RTX PRO 6000 row: unsloth Q4_K_M, 131K context, q4_0 KV cache, llama.cpp b10335, CUDA — method: stock probe.py, thinking off, --parallel 1 both sides. n-max 4 on this card: 85.7 overall (code 105.7 up, prose 58.8 down, acceptance 0.65-0.77) — overall peaks at n-max 2 here, same code-up/prose-down shape as the A6000 and 3×3090 sweeps. Cross-engine bonus on the same card: vLLM 0.27.1 with unsloth's NVFP4 build and MTP (--speculative-config '{"method":"mtp","num_speculative_tokens":N}') gives 63.3 -> 96.1 at n=2 and 116.3 (+84%) at n=4 — vLLM keeps climbing where llama.cpp has peaked. (An earlier revision carried a streamed-word-drop caveat for vLLM MTP; it was traced to the benchmark harness's own SSE parsing — a grep regex truncating multi-token deltas at escaped quotes — not to vLLM. Retracted with verification at vllm-project/vllm#52469, closed; streamed output is byte-identical to non-streamed with MTP on.) * RTX 5090 desktop row: unsloth UD-Q4_K_XL, 192K context, q8_0 KV cache, llama-server self-built from the PR #26704 branch (master-equivalent for this path, CUDA arch 120), Windows 11 native — ~26.3 GB VRAM baseline, ~28.2 GB at n-max 4. Method differs from probe.py: server timings.predicted_per_second over 900-token generations, 2 runs x the same 3 prompts, thinking off, warmup discarded. Acceptance range is per-request (prose low end, Python high end), 0.60 aggregate. * RTX 5090 32GB rows: unsloth Q6_K, 128K/256K context, q4_0 KV cache, llama.cpp 62bf73d, Windows/CUDA. Loaded VRAM was 24,441/27,385 MiB baseline and 25,797/29,381 MiB with spec. MTP gained 110.0% at 128K and 96.3% at 256K. * RTX 3090 Ti rows: Q4_K_M, 128K/256K context, q4_0 KV cache, llama.cpp 62bf73d, Linux/CUDA. Loaded VRAM was 18,842/21,786 MiB baseline and 20,134/23,718 MiB with spec. MTP gained 45.0% at 128K and 49.0% at 256K. The 256K MTP arm had 846 MiB free at load. * RTX 5090 32GB and RTX 3090 Ti method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. Baseline and MTP used the same model and serving config at each context. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2. * RTX 5090 desktop UD-Q5_K_XL row: 131K context, q4_0 KV cache, llama.cpp b10448 (ad1de39e0), Windows/CUDA 13.3. Loaded VRAM was 22,449 MiB baseline and 24,083 MiB at n-max 4. Method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. N-max 2 reached 129.4 tok/s with 80.9% aggregate acceptance; n-max 4 reached 144.2 tok/s with 63.8% aggregate acceptance, a 117.5% gain over baseline. Only the MTP flags changed within each pair. * 3×3060 row: unsloth UD-Q4_K_XL, 262K context, q8_0 KV cache, layer split --split-mode layer --tensor-split 35,37,28 (no NVLink), llama.cpp b10068, --parallel 1, --spec-draft-n-max 8 --spec-draft-p-min 0.85. Method differs from the rows above: serve-timing averages from llama-server logs, not probe.py — baseline = weighted mean of 126K decoded tokens with MTP off (17.3 t/s); with flag = weighted mean over 23.4M decoded tokens of live traffic (24.5 t/s, short bursts 28-39 t/s, long-context drops to 14-16 t/s). Acceptance = aggregate 0.87 (176,446/202,620 drafted tokens, per-task median 0.884, range 0.70-1.00) from draft acceptance log lines. * Radeon 9060 XT 16 row: AtomicChat AD-IQ3_S_IQ3_XXS, 128k context, q4_0 KV cache — method: stock probe.py. Windows, ROCM * RTX 3090 b10450 row: unsloth Q4_K_M, 131K context, q8_0 KV cache, q8_0 draft KV, llama.cpp b10450 (master ece963f41), CUDA/Linux (CachyOS), froggeric fixed chat template, --reasoning-format deepseek — method: unchanged probe.py, three runs x three prompts, thinking off. Worth noting: the b10450 baseline (41.3) equals the original day-one with-flag number for this card — the young hybrid-attention kernels caught up upstream, and the flag now stacks on top of that (+54%). * GMK EVO-X2 row: Ryzen AI Max+ 395 (Strix Halo), 64GB unified memory, Linux, ROCm/HIP llama.cpp (gfx1151, ROCm 7.2.4), unsloth UD-Q4_K_XL, 96K context, f16 KV cache, --parallel 1, thinking off — 10.5-11.1 tok/s spec-off baseline; 21.4-22.2 tok/s with --spec-draft-n-max 12 at 0.95-1.0 acceptance on the bench prompt (novel-traffic acceptance 0.345); stacking --spec-type draft-mtp,ngram-mod --spec-ngram-mod-n-min 24 takes the streamed count bench to 59.7-64.0 cold and 148-163 warm on back-to-back repeats — the warm figure is an ngram repetition artifact on that prompt, not a general speedup (real novel traffic: prose 11-24, code 30-40 tok/s); production metric is time-per-task, 7.6-14.3 s per correct task across the thermal band. Method: streamed HTTP bench against a live llama-server (not probe.py), medians of 3+ runs; one-command reproducer: bench.sh, full writeup: one week with Qwen3.8-27B on Strix Halo. * 2× RTX 5060 Ti rows: unsloth UD-Q4_K_XL (sha256 bee238bb…1372), 131K context, q4_0 KV cache, llama.cpp built from source at commit ece963f4 with -DCMAKE_CUDA_ARCHITECTURES=120, CUDA 13.0 / driver 580.173.02, PCIe 3.0 x8, PHB topology (no P2P). Method: unchanged probe.py, three runs x three prompts, thinking off. VRAM: PP 21.7 GB baseline / 23.0 GB with spec; TP 20.9 GB / 22.1 GB. The 16.68 GiB of weights do not fit one 16GB card, so two cards is the floor on this box — the split-mode choice is not optional here, which is what makes the two rows worth reading side by side. Both rows verified at n_ctx_slot = 131072. Details under the sweep below. * RTX 5090 desktop row: unsloth Qwen3.8 Dynamic NVFP4 (FP8-as-Q8 unified-mtp), 163,840 context, q8_0 KV cache, llama.cpp b10430, Windows/CUDA, driver 610.88 — first NVFP4 quant in the table. Full n-max sweep: 2 → 98.4, 3 → 108.7 (+73%), 4 + p-min 0.60 → 103.9, 8 → 92.6 — deep-draft optimum consistent with the A6000 48GB pattern; n-max 8 confirmed worst spec setting. Method: unchanged probe.py, three runs x three prompts, thinking at template default (xhigh). Acceptance 0.721 aggregate (1845/2558 from server logs). * RTX 5090 32GB row: unsloth UD-Q4_K_XL, 192K context, q8_0 KV cache, mmproj loaded (vision, --image-min-tokens 1024), llama-swap unified-cuda-2026-08-14, Linux/CUDA — 26.5 GB VRAM baseline, 28.5 GB with spec. Ungated — see the p-min A/B below. Both arms at --parallel 1 so only the spec flags differ. Method: stock probe.py, medians of three runs x three prompts, thinking off. n-max sweep at p-min 0.60 below. * RTX 3090 turboquant row: unsloth Q4_K_M, 131K context, q4_0 KV cache, custom turboquant llama.cpp at commit 95b18c0, NVIDIA driver 610.43.03, --parallel 1, all layers on GPU, thinking off. Method: unchanged probe.py, medians of three runs x three prompts, same setup both arms. MTP at n-max 6 with p-min 0.75. * RX 7900 GRE row: the only packed-16GB row in the table (96% VRAM with MTP; 86% spec-off). spec-off baseline back-filled 2026-08-16 on the same card (MTP removed as the only variable, live agent traffic to ~45K) — 28.5–28.8 tok/s at 35–45K context. custom AtomicChat IQ3_XXS quant (not unsloth Q4_K_M), 90K context, turbo3/turboquant KV cache, llama.cpp 1655 (2168b0cd8) in a custom llama-cpp-turboquant Docker image, Debian 13 trixie, kernel 6.12.101, Vulkan/AMD Navi 31, --parallel 1, --reasoning-budget 512, flash-attn on. Both arms live Hermes agent traffic (not probe.py), context bands differ by ~1.5K tokens. Full study under the section below. * RTX 3090 turbo3 section: same host (RTX 3090 24GB, Debian 12, Ryzen 7 9700X, CPB disabled), unsloth Q4_K_M, both arms MTP-on (--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 --spec-default, -np 1) — this row compares two KV cache configurations, not spec-off vs spec-on. Baseline arm: q4_0 KV, 185K context, llama27b-mtp:cuda image (mtp-clean fork), live Hermes agent traffic (14 completed turns, 100–2700 output tokens, avg 37.2 tok/s, acceptance 0.57). Flag arm: turbo3 KV, 200K (204800) context (262K exceeded 24 GB at model load), llama-cpp-turboquant:cuda-latest (v10465, fca3093c9), clean probe 3×400 tokens, medians of 3, avg 54.2 tok/s (48.06–58.34), acceptance 0.64–0.84. VRAM 23.0/23.3 GB of 24.6 GB (~94–95%), 52–63°C. Confound disclosed: the two arms run different llama.cpp builds (mtp-clean vs turboquant v10465), so part of the delta is the newer build, not only the KV cache. Method differs from the rows above and is spelled out below. * RTX 5090 32GB (Q4_K_M, 131K): Unsloth Q4_K_M, 131K context, q4_0 KV cache - 19.54GB baseline, 20.38GB with spec. * RTX 5090 32GB (UD_Q4_K_XL, 262K): Unsloth Q4_K_XL, 262K context, q4_0 KV cache - 23.27GB baseline, 24.75GB with spec. * RTX PRO 6000 Blackwell Max-Q row: lmstudio-community/Qwen3.8-27B-GGUF Q8_0, 131K context, q4_0 K/V cache, llama.cpp 0.1.0-dev build 10454 (4df29be4f), built with GNU 16.1.1 for Linux x86_64 / CUDA 13.3.1-1, RTX PRO 6000 Blackwell Max-Q Workstation Edition 96GB at 300W, --parallel 1. VRAM: 32.072 GiB baseline / 33.540 GiB with spec (+1468 MiB). Method: unchanged probe.py, three runs x three prompts, thinking off; qwen38-mtp (28527fc55) commit used for probe.py. * 2× Tesla P40 row: unsloth UD-Q5_K_XL, 131K context, q4_0 KV cache, llama.cpp b10453 (3cb7ffb1a), Ubuntu 25.04 / CUDA 12.9, driver 580.178.04, --tensor-split 1,1 with numactl --interleave=all, P2P disabled (cards on separate NUMA nodes, SYS path), --parallel 1, thinking off. ~11 GB/GPU baseline, ~12 GB/GPU with spec. Method: probe.py unchanged, three runs x three prompts, both arms otherwise identical. MTP arm: --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75; prefill 354 → 295 tok/s (20K tokens). Full n-max sweep and a multi-model companion sweep below. * RTX 5090 32GB UD-Q5_K_XL 262K row: unsloth UD-Q5_K_XL, 262K context, q4_0 KV cache, llama.cpp b10453 (3cb7ffb1a), Linux/CUDA, driver 610.57.04. Loaded VRAM ~32.1 GB. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Both arms at --parallel 1. Baseline 69.6 tok/s; with --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 147.5 tok/s (+112%). Acceptance 0.69-0.87 (avg 0.78). N-max 4 ungated reached 155.8 tok/s (0.31-0.87 acceptance) — ungated is faster but the gated config is the safer daily bet. * 2× RTX 5060 Ti Q4_K_M row: unsloth Q4_K_M, 131K context, q4_0 KV cache, --split-mode tensor, llama.cpp b10450 (ece963f), Windows 11 / CUDA 13.2 native sm_120a, driver 596.36, --parallel 1. VRAM per GPU: 9,808 MiB baseline / 10,684 MiB at n-max 3. Method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 3. * RTX 5090 32GB UD-Q4_K_XL 262K q4_0 KV row: unsloth UD-Q4_K_XL, 262K context, q4_0 KV cache (both K and V), llama.cpp b10453 (3cb7ffb1a), Linux/CUDA, driver 610.57.04. Loaded VRAM ~25.8 GB. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Both arms at --parallel 1. Baseline 76.3 tok/s; with --spec-type draft-mtp --spec-draft-n-max 4 (ungated, no p-min) 171.7 tok/s (+125%). Acceptance 0.69-0.87 (avg 0.69). N-max 6 gated (p-min 0.75) reached 154.5 tok/s (0.77-0.87 acceptance) — ungated is faster, gated is the safer daily bet. * 2× Tesla P40 UD-Q4_K_XL 262K q4_0 KV row: unsloth UD-Q4_K_XL, 262K context, q4_0 KV cache (both K and V), llama.cpp b10453 (3cb7ffb1a), Linux/CUDA, --tensor-split 1,1 (no NVLink), --parallel 1. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Baseline 13.2 tok/s; with --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75 23.4 tok/s (+77%). Acceptance 0.82 avg. N-max 4 ungated reached 23.6 tok/s (0.68 acceptance) — only +0.8% faster (noise) with much lower acceptance; gated is the clear winner on bandwidth-starved Pascal. N-max 6 gated: 22.5 tok/s (0.80 acceptance) — n-max 4 is the sweet spot. KV f16 was within 2% of q4_0 on all arms (13.4 / 23.6 / 22.4 / 22.6) and also fits 262K ctx on 2×24GB with this quant — q4_0 KV buys headroom, not speed; f16 is the default choice when VRAM allows. * RX 7900 XTX Vulkan row: unsloth Q4_K_M, 131K context, q4_0 KV cache, llama.cpp master 4695f00 (2026-08-17), Vulkan backend (RADV, Mesa 26.1.5) on Void Linux musl, kernel 7.0.14, ReBAR enabled, Ryzen 9 5900XT host — no ROCm installed; this is mainline Mesa only. 0.6 GB VRAM before serve, 18.6 GB serving baseline, 19.9 GB with spec. Method: unchanged probe.py at dc18736, three runs x three prompts, thinking off, both arms --parallel 1. Row is the n-max 3 arm (aggregate acceptance 0.72); n-max 2 on the same config measured 57.9 at 0.80 aggregate (1652/2055), the range's low end is the prose prompt in both arms. Full sweep and gate A/B in the section below. Relative to the earlier XTX row: this baseline reads lower (28.8 vs 30.7) while the flag arms read far higher — different backend and a 2026-08-17 build (see rule 6); the deltas are the durable part. * RTX 3090 UD-Q2_K_XL row: unsloth UD-Q2_K_XL (10.7 GB), 32K context, f16 KV cache, llama.cpp master source snapshot of 2026-08-14 (built from archive, so no commit hash; it contains the ssm_scan state-rollback that makes deep MTP drafts work on the DeltaNet layers), CUDA 13 self-built, CachyOS/Linux, driver 610.43.03. VRAM 13.9 GB baseline, 14.7 GB with spec. The card is overclocked: memory +1200, core +180, power limit 430 W against a 390 W default. Method: unmodified probe.py at a4c3028da1, three full passes per arm, the table shows the median of the three pass medians; passes were 52.5/52.4/52.3 and 85.6/85.1/87.2. Both arms --parallel 1, thinking off. * RTX 3090 UD-Q4_K_XL row: same machine, same build, same method, same overclock; unsloth UD-Q4_K_XL (17.9 GB), 32K context, f16 KV. VRAM 20.5 GB baseline, 21.4 GB with spec. Passes were 43.9/43.9/43.9 and 79.8/79.6/78.2. This row is the one to compare against the other 3090 rows, since those use ~17 GB Q4 files; the UD-Q2_K_XL row above is faster mainly because it reads 38% fewer weight bytes per token, not because of the overclock. See the section below. * 3060 Ti + 3080 Ti row: Unsloth Dynamic IQ3_XXS (Qwen38_27B-UD-IQ3_XXS.gguf) + mmproj-BF16, tensor-parallel across an 8GB 3060 Ti and a 12GB 3080 Ti (--split-mode tensor, --tensor-split 11,9), 131K context, q4_0 KV cache, llama.cpp local build @ f9779dd, Linux/CUDA (driver 595.84). VRAM: ~0.2/0.9 GB pre-load baseline (desktop residual) vs 7.5/11.3 GB with spec. Method: stock probe.py @ dc18736, both arms --parallel 1, medians of 3 runs; with-flag arm run three times at the exact spec-on config (49.2 / 46.4 / 52.4) — reported value is the median. Prose prompt runs ~3 tok/s slower with the flag in every arm; the gain is carried by code prompts. * Apple M4 row: first Apple Silicon row. M4 base (10-core GPU, ~120 GB/s), 24GB unified memory, macOS 15 / Metal, iogpu.wired_limit_mb=21504. unsloth UD-Q3_K_XL (12.5 GiB), 32K context, q4_0 KV cache, llama.cpp b10450 (ece963f41, Homebrew bottle), -b 512 -ub 512 — the default -b 2048 OOMs the spec arm mid-generation and takes the server down, see the section. VRAM 13.2 GB baseline, 13.9 GB with spec. Method: unmodified probe.py at a4c3028da1 (which is itself medians of three runs per prompt), run as two independent full passes per arm — six samples per prompt, --parallel 1, thinking off. Runs were 5.8/5.8 baseline and 5.6/5.9 with the flag, so the two runs straddle parity and the overall number is a wash within noise (band ~±0.2 tok/s on ~6). The reproducible part is the per-prompt split: code +9-10%, prose -22-24%. Batch-8 decode amortizes only 1.2x on this backend against 3.3x on CUDA, which is why; a matched RTX 3090 control on the same GGUF, flags and commit is in the sweep rather than this table, since it is a method check and not a new card. Full analysis in sweeps/apple-silicon.md. * Mac Studio M3 Ultra row: Apple M3 Ultra with 96 GB unified memory, macOS/Metal; unsloth Q6_K (21 GB file), 131K context, Flash Attention, q4_0 K/V main and draft KV caches, --parallel 1, LM Studio llama.cpp Metal backend 2.29.0 (llama-server version: 1 (dd1ea52)). Process RSS was 23.20 GiB baseline / 24.34 GiB at MTP n-max 2; Apple has no discrete VRAM, so these are process-RSS measurements, not VRAM allocations. Method: unmodified probe.py, warmup discarded, three runs x three prompts, thinking off, and Qwen3.8 was the only resident model. The table uses the overall probe median. Only the MTP arm enabled --spec-type draft-mtp --spec-draft-n-max 2 and its q4_0 draft KV cache. Aggregate acceptance was 0.798 (1681/2106); the n-max and p-min sweep is below. * RTX 3090 Q4_K_M 131K 250W row: Unsloth Qwen3.8-27B Q4_K_M, 131072 context, q4_0 K/V cache, llama.cpp b10473 (b75ecd197), Fedora 44/CUDA 13.3, NVIDIA 610.43.03, RTX 3090 24GB capped at 250W. Loaded VRAM was 18,856 MiB baseline and 20,486 MiB at n-max 4. Method: unchanged probe.py at commit 70a699e4, --parallel 1 both arms, thinking off, three complete probe passes per arm; reported values are the median of the three overall medians. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 4. Aggregate n-max 4 acceptance: 5420/8370 = 0.648. * RTX 5090 32GB UD-Q4_K_XL 131K row: ASUS TUF RTX 5090 32GB, unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL, 131K context, q4_0 KV cache (both K and V), llama.cpp build 10451 (10bf611e5), Arch Linux kernel 7.1.8, CUDA. Serving config is stock serve_mtp.sh (-c 131072 -ngl 999 -fa 1 --cache-type-k q4_0 --cache-type-v q4_0 --parallel 1); the spec arm changed only --spec-draft-n-max, 2 to 4. VRAM 20,366 MiB baseline / 21,500 MiB with spec (+1,134 MiB). Method: unchanged probe.py at 70a699e, three runs x three prompts, thinking off, warmup discarded. Table figures are the OVERALL medians; pooled means were 74.5 baseline and 169.1 with spec, so the gain reads +145% on medians and +127% on means. Acceptance aggregate 0.69 (1860/2696 from server logs); the quoted range is per-run and the spread is unusually wide — code 0.87, bash 0.74, prose 0.40 aggregate, with mean draft length falling 4.60 to 2.52 along the same axis. The baseline is exceptionally flat (74.3-74.8 across all nine runs), i.e. fully bandwidth-bound before the flag. Compare the 262K UD-Q4_K_XL row above: 74.3 → 179.7 at the same n-max 4 on a different host, so halving context to 131K moved this pair very little. * RTX 5080 + 3090 row: mixed Blackwell + Ampere, no NVLink. unsloth UD-Q4_K_XL (sha256 bee238bb…1372), 131K context, q4_0 KV cache (K and V), llama.cpp b10470 (34af94c, official win-cuda-13.3 zip), Windows 11 / CUDA 13.3, driver 610.88, --split-mode tensor, --parallel 1, flash-attn on. VRAM 12,048+10,334 MiB baseline, 12,827+11,130 MiB with spec. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Both arms otherwise identical. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2 (ungated; 5080 is Blackwell). Server log: n_ctx_slot = 131072. Note: backend sampling not supported with SPLIT_MODE_TENSOR; using CPU sampler — MTP still 55.6 → 92.9 (+67%). Per-prompt medians: code 55.7→105.2, prose 54.0→79.2, bash 55.8→92.9. Acceptance 0.51-0.95 (agg ~0.79 from draft acceptance lines, warmup excluded). No n-max sweep yet. * RTX 5060 Laptop 8GB row: unsloth/Qwen3.8-27B-GGUF IQ4_XS (15,705,861,088 bytes, sha256 9fd40d70...ce666), 64K context, q4_0 K/V cache, 25 GPU layers with the remaining layers on CPU, llama.cpp b10453 (3cb7ffb1a), Windows build 26200 / CUDA 12.4, driver 582.05, Ryzen 7 260, 31.31 GiB RAM, --parallel 1, Flash Attention on, batch 2048 / ubatch 512. Loaded VRAM: 6,796 MiB baseline / 7,540 MiB with MTP (6,868 / 7,614 MiB after the probe). Method: unchanged probe.py at commit 70a699e, three runs x three prompts, thinking off, warmup discarded. Both arms were otherwise identical; MTP added --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.70. Acceptance 0.75-0.97, 0.90 aggregate (1,594 / 1,770). This is intentional hybrid CPU/GPU offload; the full model weights are not resident in 8GB VRAM. * RTX PRO 3000 Blackwell Laptop 12GB row: unsloth Qwen3.8-27B IQ4_XS (SHA-256 9fd40d70...ace666), 64K context, 38/66 layers on GPU, with the remaining layers on CPU, q4_0 K/V cache, batches 2048/512, 24/24 threads, Flash Attention, llama.cpp b10456 (f275595dd) official Windows CUDA 12.4 build, Windows 11 Enterprise, NVIDIA driver 573.71 (CUDA 12.8 compatibility), --parallel 1, thinking off. Peak monitored dedicated VRAM was 10,112 MiB baseline / 10,922 MiB MTP. Method: unchanged probe.py at commit 70a699e4, three runs x three prompts, warmup discarded. Overall nine-run median was 5.3 tok/s baseline / 10.8 tok/s with --spec-type draft-mtp --spec-draft-n-max 2 (+104%); only those two MTP options changed. Draft acceptance was 1,747/2,047 aggregate (0.85), per-request range 0.67-0.93. Laptop was on AC; peak temperature 67/68 C baseline/MTP. * RTX 3080 10GB row: the smallest-VRAM card in the merged table, and the only one holding the weights fully resident under 12 GB. unsloth UD-IQ2_XXS (Qwen3.8-27B-UD-IQ2_XXS.gguf, 9,010,048,064 B, sha256 8d1b3729…c35b8524, verified byte-exact against the HF repo), 16K context, q8_0 KV cache, llama.cpp b10472 (60eeeb608), Windows 11 build 26200 / CUDA 12.4, driver 591.86, RTX 3080 10GB GA102 SM86 at stock 370 W on PCIe 4.0 x16, Ryzen 9 7900X + 32 GB DDR5-6000, full offload (-ngl 99, every layer), --parallel 1, --jinja, thinking off, both arms otherwise identical. VRAM 9,156 MiB baseline / 9,734 MiB with spec (+578 MiB) of 10,240 MiB. llama-bench -ngl 99 -fa 1 on the same file: pp512 1132.46 ± 15.80, tg128 45.86 ± 0.04. Method: unchanged probe.py at 70a699e, medians of three complete passes per @dcrey7's note (baseline 45.8/45.1/44.8; n-max 2 63.2/65.3/64.4). Aggregate acceptance 0.74 (4,543/6,179 from server logs). n-max 3 measured 62.9 at 0.67 — n-max 2 is the peak, matching rule 1. The card must be headless and that is load-bearing: with both monitors on the 7900X iGPU the desktop's footprint on the 3080 is 64 MiB; with them attached dwm alone holds 1,846 MiB, full offload does not fit, and the same box baselines 33.0 at ngl 62. The MTP arm's extra ~580 MiB, not the baseline's fit, sets the context ceiling: 32K inverts to 45.4 → 24.7 (−46%) while acceptance stays at 0.73 and nvidia-smi still reports 446 MiB free. Measured at the bus, the inverted arm moves 11.3 GB/s rx / 13.6 GB/s tx against 0.66 / 1.34 healthy, and pulls 252 W against 366 W — starved on PCIe, not throttling. Full study in sweeps/rtx-3090.md. * CMP 170HX row: unsloth Q4_K_M (15.93 GiB), 131K context, q4_0 KV cache, llama.cpp 0.1.0-dev commit f275595dd (reported as build 1), Linux/CUDA, driver 610.43.02, headless Ubuntu VM. The card is a GA100 CMP 170HX (70 SMs) with Hynix HBM2e unlocked from 8GB to 64GB, 200W cap, PCIe Gen1 x4. VRAM 18,864 MiB baseline, 20,152 MiB with spec. Method: unchanged probe.py at 70a699e, three runs x three prompts, thinking off, both arms --parallel 1. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2. Per-prompt medians: baseline 33.1 / 33.0 / 32.7, with flag 51.4 / 38.3 / 46.7 (code / prose / bash). Overall mean 32.9 -> 45.5, median 33.0 -> 46.7 (+41.5%). Draft acceptance from slot print_timing 0.53-0.94 (prose low end, Python high end). No n-max sweep in this PR; 64GB has headroom to try 3-4 later. See sweeps/cmp-170hx.md. * RTX 4090 snakeyjay63-png rows: four configs measured on same hardware, Ryzen 7 9800X3D, Linux/CUDA, 500W power limit (raised from 450W stock default — likely pushes tokens/s a touch higher than stock-limit rows). Row 1 (Q4_K_M): unsloth Q4_K_M, 131K context, q4_0 KV cache, ngl 999, flash attention on, MTP n-max 2 — overall 68.8, per-prompt: Python 47.0→91.4 (+94%), Prose 47.0→64.6 (+38%), Bash 47.3→67.4 (+43%). Row 2 (Purusha ★): LoRA merge (18 safetensors on stock Qwen3.8-27B), three variants tested — (b) 65K/ngl 79/p-split 0.05 → 81.5, (c) 131K/ngl 999 → 83.1, (d) 65K/ngl 79/p-split 0.06 → 83.4, overall +78%. Simplest config = highest score. First row with GPU metrics: baseline 353W/64°C/97% mem util vs MTP 403W/69°C/88% mem util — +50W, +5°C, +519 MiB VRAM, but GPU util drops 3% and mem util drops 9% (fewer forward passes). Method: stock probe.py, 3 runs × 3 prompts, thinking off. * Tesla V100 32GB row: one V100 32GB in a Dell PowerEdge R720, unsloth Q4_K_M, 4K context, f16 KV cache, flash attention on, llama.cpp b10358 (030ebb558), NixOS / CUDA driver 580.142, both arms at --parallel 1 with only --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0 added. VRAM 18,022 MiB spec-off against 18,864 MiB with the head, measured at 32K context rather than at this row's 4K. Method differs from probe.py: non-streaming requests scored on the server's own timings.predicted_per_second, greedy, n_predict 400 with ignore_eos, cache_prompt false, medians of five samples after a discarded warmup — prose 33.6 → 47.9, code 33.0 → 48.1, each arm inside 2.2% of its own minimum. Depth and build sweeps in sweeps/workstation.md. * RTX 5060 Ti 16GB row: quimmedes/Qwen3.8-27B-XYZ Q4-XYZ-v2 (15.06 GB — the largest Q4-tier file that fits one 16 GB card; unsloth Q4_K_M 17.11 GB, IQ4_NL 16.34 GB and UD-Q4_K_XL 17.92 GB do not fit a single card), 32K context, q4_0 KV cache, llama.cpp b10472 (60eeeb6, 2026-08-17) built from source with CUDA 12.8 (-DCMAKE_CUDA_ARCHITECTURES=120), Ubuntu 24.04 / CUDA, driver 595.84, Ryzen 5 3600. Method: unchanged probe.py at 70a699e, three runs x three prompts, thinking off, --parallel 1 both arms. VRAM 14,514 MiB baseline / 15,704 MiB at n-max 4. Full n-max sweep (2/3/4/6 + p-min) and an unsloth IQ4-family study in sweeps/rtx-5060-ti.md: draft depth keeps paying on this card (n2→n4: 50.0→59.3), n-max 6 OOMs at load (99.1% VRAM at n4), p-min 0.65 gating costs 11.5% throughput (52.5 vs 59.3) despite raising acceptance, and a June-22 vs b10472 build A/B shows only +1.5% baseline / +0.3% at n4 (rule 6's +10-15% did not reproduce on this card).

Deep dives

Every contributor sweep and study is in sweeps/, grouped by card family and PR-able like the table: RTX 5090 · RTX 3090 · Radeon · APUs and iGPUs · Multi-GPU · Workstation · RTX 5060 Ti · Apple Silicon · CMP 170HX

License

Apache-2.0. The numbers and verdicts are real, the conclusions are mine.