Qwen3.6-35B-A3B-PARO: config.json declares mtp_num_hidden_layers=1 but no MTP weights shipped → 0% accept + ~50% decode overhead by default

[sangemaru]

Summary

z-lab/Qwen3.6-35B-A3B-PARO's config.json declares mtp_num_hidden_layers=1 (which advertises an MTP head) but the shipped safetensors contain no MTP weights. When a runtime takes the config at face value and engages its MTP path, the result is 0% accept rate + ~50% decode overhead from the empty-tensor verify step. The fix on the operator side is to manually set mtp_enabled=false in oMLX model_settings.json (or equivalent), but anyone who tries this model with defaults wastes a lot of confusing debug time before figuring that out.

Affected runtime behavior observed

On oMLX 0.3.9.dev2 (M1 Max) without overriding mtp_enabled:

• MTP path engages because config declares an MTP layer.
• Forward pass through the empty MTP head returns an empty/uninitialized tensor each step.
• Accept rate: 0%.
• Decode wall ~1.5× the no-MTP baseline (the MTP head is still computed, just produces no useful drafts).

With mtp_enabled=false set explicitly:

• MTP path skipped.
• 31.3 t/s warm @ 32K, 200-tok decode — the actual PARO performance (this is our Mac long-ctx workhorse number).

Reproducer

1. huggingface-cli download z-lab/Qwen3.6-35B-A3B-PARO.
2. Inspect config.json — note mtp_num_hidden_layers: 1.
3. Inspect model.safetensors.index.json — note no language_model.mtp.* keys (for comparison, Jundot/Qwen3.6-27B-oQ6-mtp has 29 such keys, samwang0041/Qwen3.6-27B-MLX-4bit-MTP has 31, Jundot/Qwen3.6-35B-A3B-oQ4-mtp has 42).
4. Serve via oMLX 0.3.9.dev2 with model_settings defaulting mtp_enabled=true.
5. Observe 0% accept rate and ~50% decode overhead in the MTP[N] server log lines.

Fix options

(a) Drop mtp_num_hidden_layers from the released config.json. Cleanest fix — config matches what's actually shipped.

(b) Ship the MTP head weights in the safetensors release. Adds size but enables MTP for downstream users.

(c) Document the required mtp_enabled=false override in the model card / README and accept that downstream stacks default-on MTP from config. Operator-side fix only.

Most other paroquant PARO releases I checked (gemma-4-*-it-PARO, Qwen3.6-27B-PARO) ship clean configs that don't trigger this. The 35B-A3B-PARO is the outlier.

Why this matters

z-lab/Qwen3.6-35B-A3B-PARO is the best Mac long-context workhorse model currently available for M1 Max (256K cold/warm, 36 t/s @ 32K via oMLX). I'd rather not have it cost downstream operators a wasted afternoon on first try. Happy to confirm option (a) works end-to-end with a quick re-pull and bench if useful.

Companion data

This is captured in our test fleet's per-model facts card alongside the MTP sampler-floor finding (filed separately at jundot/omlx) and the MoE Gemma-4 Metal OOM (filed separately, also paroquant). Happy to share full server logs or bench JSONLs.

[sangemaru]

A relevant data point for triage: guru87/Qwen3.6-27B-MTP on HuggingFace ships the standalone MTP head extracted from the official Qwen/Qwen3.6-27B checkpoint as a single ~810 MB mtp.safetensors file, specifically designed to be injected into PARO checkpoints missing their MTP weights:

"This repo is only useful as a drafter for some other quantized Qwen3.6-27B checkpoint that is missing its MTP weights."

Tested on "2× RTX 3090 + paroquant + this MTP head"; vLLM speculative decoding example targets ./Qwen3.6-27B-PARO-MTP. Reports 81-91% draft acceptance at position 0, 71-79% at position 1, sustained ~580 tok/s gen at parallel=24.

So there's an existing community workaround for the 27B variant, but no equivalent for the MoE 35B-A3B — I just searched HF and guru87 only ships heads for Qwen3.6-27B-MTP + four sizes of the older Qwen3.5 family; the 35B-A3B (256-expert MoE) likely needs a careful extraction the community hasn't done yet. I've opened a discussion at https://huggingface.co/guru87/Qwen3.6-27B-MTP/discussions asking if they'd consider it.

Either way, two options for closing this from the paroquant side:

1. Drop mtp_num_hidden_layers from Qwen3.6-35B-A3B-PARO/config.json. Defaults-on-MTP runtimes (oMLX 0.3.9.dev2's MTP path) would stop wastefully engaging the empty head, and operators wouldn't need the mtp_enabled=false override.

2. Ship the MTP head bundled or as a sibling repo (z-lab/Qwen3.6-35B-A3B-PARO-MTP or similar). This is what guru87 is doing for the 27B side. Would unlock vLLM + paroquant speculative decoding on NVIDIA for the 35B-A3B workhorse.

Option 1 is the immediate-user-experience fix; option 2 is the "actually useful MTP" fix. They're independent — option 1 is cheap and stops the bug, option 2 is value-add that takes more engineering. Happy to test whichever you ship on M1 Max + NVIDIA-class hardware.

[liang2kl]

Thanks for reporting this. I think bundling the MTP weights should be the best option.