[5/5] feat(convert): build the published Qwen3.6-35B-A3B NVFP4 checkpoint into an artifact - #290
Open
MichaelDementii wants to merge 5 commits into
Open
MichaelDementii wants to merge 5 commits into
MichaelDementii wants to merge 5 commits into
Conversation
…essing The four decode kernels addressed a matrix from the call site: a flat group index built from the row, and three plane pointers picked by hand. That holds while every codec keeps one scale per group per row. It does not generalise - a codec whose scale plane is swizzled needs the row to form an address, and the call site cannot compute it without already knowing the codec. A codec now receives the matrix as a plane set plus a row, with the column count as a template argument. The shared expert becomes a codec parameter in the same move: every registered profile stores it as Q8 today, but that is a property of the profiles rather than of the kernels, and the kernels no longer assert it. No behaviour change. The arithmetic, the accumulation order and the launch geometry are untouched for all four registered codecs.
…d tensor A tensor object holds one NVFP4 weight divisor, which is right for a plane quantised as one matrix and wrong for a plane assembled from several that were quantised apart. Published NVFP4 checkpoints of MoE models are the second case: each expert matrix carries its own weight_global_scale, so the converter has to refuse the pack. The tensor object gains an optional `divisors`: the number of separately quantised source matrices stacked into the plane, default 1. The payload keeps that many FP32 words immediately after the scale plane, in stacking order, so payload_bytes = weight_divisor_offset + 4 * divisors, and row r of an N-row plane uses word r / (N / divisors). A divisor's share must be a whole number of 128-row scale tiles, so a boundary never splits one. Absent means one word and the present layout exactly, so every existing artifact stays valid and no reader changes behaviour. The count is decided where the parent's shape is chosen, in the recipe plan, rather than being discovered at encode time; the converter refuses it outside NVFP4, and `validate_nvfp4_weight` refuses a plane whose divisor count does not divide its rows.
The operation admitted expert banks as Q4+Q5/Q6 or Q8+Q8 and nothing else, so an artifact whose experts are NVFP4 had no route at any token count. This registers the profile across all three: decode, small-token and prefill. Decode reads NVFP4 against represented BF16 activations. The kernels are CUDA-core dot products, so four-bit activations buy no arithmetic there and would cost a quantiser of their own; keeping the activation represented puts rel_l2 at T=1 at 1.60e-3 instead of 1.70e-1. The D4 block takes two hidden rows, because the scale plane puts rows r and r+1 in one 32-byte sector. Prefill quantises the chunk once and runs W4A4 over the existing work list, and the gate/up epilogue writes the SwiGLU intermediate already encoded, so `down` needs no separate quantiser. The per-source divisor is applied in that epilogue rather than inside the MMA loop, by a shift where the stride is a power of two. The frontier between the small-token and prefill routes is 13 for this profile, swept cold over [2,46] against its own routes rather than inherited from the groupwise ones, and the small-token schedule is chosen by the same sweep. The staging buffers the route needs are charged to this profile alone. Three launches written for it - the small-token router, the fused selection/scan/index block and a narrower route-job bound - are gated on it for the same reason: they would otherwise change what a groupwise call runs on the ragged tail of a sliced chunk. The wrapper also requires the routed and shared banks to agree on their activation divisor, because the route encodes the chunk once and both gate/up GEMMs read that one plane. The operator test walks the profile at T = 1, 2, 12, 13, 64, 768, 4097, with a separate tolerance either side of the frontier, and its fixture stacks gate and up under divisors that differ, as the published checkpoint does.
`prepare_sparse_moe_weights` enumerates the admissible bank formats and had no NVFP4 arm, so an artifact whose experts carry the format could not be bound even once the operation could run it. The profile takes all four expert matrices as NVFP4 - they are inseparable, because the gate/up epilogue emits an encoded intermediate that `down` reads. It requires AllowA4 on every expert input, since the prefill route quantises the hidden state to four bits, and every other NVFP4 consumer refuses an input without it.
…nto an artifact A recipe for the NVFP4 releases of this model: the routed and shared experts are imported as they are stored, the projections stay Q8 and the vocabulary Q8/Q6, as in the groupwise recipe. The expert banks are grouped explicitly, because automatic grouping still separates parents whose divisors differ - correctly, since its result is also read by dense routes whose kernels take one scalar, and only the MoE kernels can read a stack.
Contributor
Author
|
Last of the five: the conversion recipe for Qwen3.6-35B-A3B on NVFP4. It depends on [2/5] and @codex review |
iamwavecut
added a commit
to iamwavecut/ninfer-3090
that referenced
this pull request
Sep 25, 2026
…nto an artifact A recipe for the NVFP4 releases of this model: the routed and shared experts are imported as they are stored, the projections stay Q8 and the vocabulary Q8/Q6, as in the groupwise recipe. The expert banks are grouped explicitly, because automatic grouping still separates parents whose divisors differ - correctly, since its result is also read by dense routes whose kernels take one scalar, and only the MoE kernels can read a stack. Ported from upstream PR Neroued#290. Co-authored-by: MichaelDementii <136074657+MichaelDementii@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue: #283
Last of five. It needs the third; it is what produces the artifact the series exists for.
What changes
A recipe,
qwen3_6_35b_a3b_nvfp4, that builds the published NVFP4 releases of this model into anartifact: routed and shared experts imported as they are stored, projections Q8, vocabulary Q8/Q6 —
the same assignment as the groupwise recipe everywhere except the experts.
The expert banks are grouped explicitly rather than by the automatic rule, and only in the text
component. Automatic grouping still separates parents whose divisors differ, and that is right: its
result is also read by dense routes whose kernels take one scalar, and only the MoE kernels can read a
stack. An explicit group is the statement "this one is a stack".
Contract
A new recipe name in
docs/weight-conversion.md; no existing recipe changes. Conversion needs noexternal script and rewrites no block scale, so the artifact holds the vendor's own codes and scales
byte for byte.
Reproducing it needs three files the release omits and the loader requires:
preprocessor_config.jsonand
video_preprocessor_config.json, taken from the BF16 release, and atokenizer_config.jsoncarrying the
added_tokensarray of the release's owntokenizer.json— the loader cross-checks thetwo, so an abbreviated one is refused.
Verification
pytest tests/artifact tests/convert: 55 at this commit against 47 on master; the two new onescover the source mapping for a routed and a shared expert, and its refusal on a missing one.
RedHatAI/Qwen3.6-35B-A3B-NVFP4converts in one command; the artifact is 21 759 564 036 bytes againstthe groupwise control's 22 244 450 816. Quality and the perplexity control are on the Issue.
Checks not run
AllowA4policy or the grouping rule.They do in the RedHatAI release, every one 131.0; a release where they differ would convert and
then be refused at bind. Naming the tensor at conversion time would be better and is not done.
too; unsloth additionally stores its projections as
fp8_e4m3fn_row_bf16at K=2048, a shapesrc/ops/linear/fp8/fp8_shapes.hhas no family for.🤖 Generated with Claude Code