[2/5] feat(artifact): carry one NVFP4 divisor per source matrix in a stacked tensor - #287
Open
MichaelDementii wants to merge 2 commits into
Open
MichaelDementii wants to merge 2 commits into
MichaelDementii wants to merge 2 commits into
Conversation
…essing The four decode kernels addressed a matrix from the call site: a flat group index built from the row, and three plane pointers picked by hand. That holds while every codec keeps one scale per group per row. It does not generalise - a codec whose scale plane is swizzled needs the row to form an address, and the call site cannot compute it without already knowing the codec. A codec now receives the matrix as a plane set plus a row, with the column count as a template argument. The shared expert becomes a codec parameter in the same move: every registered profile stores it as Q8 today, but that is a property of the profiles rather than of the kernels, and the kernels no longer assert it. No behaviour change. The arithmetic, the accumulation order and the launch geometry are untouched for all four registered codecs.
…d tensor A tensor object holds one NVFP4 weight divisor, which is right for a plane quantised as one matrix and wrong for a plane assembled from several that were quantised apart. Published NVFP4 checkpoints of MoE models are the second case: each expert matrix carries its own weight_global_scale, so the converter has to refuse the pack. The tensor object gains an optional `divisors`: the number of separately quantised source matrices stacked into the plane, default 1. The payload keeps that many FP32 words immediately after the scale plane, in stacking order, so payload_bytes = weight_divisor_offset + 4 * divisors, and row r of an N-row plane uses word r / (N / divisors). A divisor's share must be a whole number of 128-row scale tiles, so a boundary never splits one. Absent means one word and the present layout exactly, so every existing artifact stays valid and no reader changes behaviour. The count is decided where the parent's shape is chosen, in the recipe plan, rather than being discovered at encode time; the converter refuses it outside NVFP4, and `validate_nvfp4_weight` refuses a plane whose divisor count does not divide its rows.
Contributor
Author
|
This one adds a @codex review |
iamwavecut
added a commit
to iamwavecut/ninfer-3090
that referenced
this pull request
Sep 25, 2026
…d tensor A tensor object holds one NVFP4 weight divisor, which is right for a plane quantised as one matrix and wrong for a plane assembled from several that were quantised apart. Published NVFP4 checkpoints of MoE models are the second case: each expert matrix carries its own weight_global_scale, so the converter has to refuse the pack. The tensor object gains an optional `divisors`: the number of separately quantised source matrices stacked into the plane, default 1. The payload keeps that many FP32 words immediately after the scale plane, in stacking order, so payload_bytes = weight_divisor_offset + 4 * divisors, and row r of an N-row plane uses word r / (N / divisors). A divisor's share must be a whole number of 128-row scale tiles, so a boundary never splits one. Absent means one word and the present layout exactly, so every existing artifact stays valid and no reader changes behaviour. The count is decided where the parent's shape is chosen, in the recipe plan, rather than being discovered at encode time; the converter refuses it outside NVFP4, and `validate_nvfp4_weight` refuses a plane whose divisor count does not divide its rows. Ported from upstream PR Neroued#287. Co-authored-by: MichaelDementii <136074657+MichaelDementii@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue: #283
Second of five, and the only product-contract change in the set. Independent of the first; the third
needs both. No performance claim.
What changes
A tensor object holds one NVFP4 weight divisor. That is right for a plane quantised as one matrix and
wrong for a plane assembled from several quantised apart — which is what every published NVFP4
checkpoint of an MoE model is: each expert matrix carries its own
weight_global_scale, 30 720 ofthem on the routed banks of
qwen3_6_35b_a3b.The object gains an optional
divisors, the number of separately quantised source matrices stackedinto the plane; the layout and the row-to-word rule are on the Issue. Default 1.
Contract
divisorsabsent means one word and today's layout exactly, so the stored form of every existingartifact is unchanged. The reader is not:
validate_nvfp4_weightgains two refusal conditions thatthe six shipped dense NVFP4 wrappers run through, so a
Weightbuilt outsidenative_weightnow hasto carry a divisor pointer. That is why two in-tree fixtures move with it —
tests/ops/quantized_weight.handbench/ops/quantized_weight.cuh.Four refusals in total:
divisorsoutside NVFP4, a count that does not divide the rows, a share thatis not a whole number of scale tiles, and sources with unequal row counts. Two of them are the C++
conditions above; all four are enforced on the Python side. The count itself is decided where the
parent's shape is chosen, in the recipe plan, rather than being discovered at encode time.
The documents that own the rule move with it: the
nvfp4row, the reconstruction equation, and theproducer and reader obligations of
docs/maintainer/tensor-formats.md; thepayload_bytesformulain
docs/maintainer/storage-layouts.md; the object-member table ofdocs/maintainer/artifact-container.md§5.1.validate_nvfp4_weightalso gains a defaultedstackedflag that every route in this commit leavesfalse, so a multi-divisor plane is refused everywhere until the route that can read one arrives.
Nothing in the engine reads more than one divisor yet; that is the next commit, and it is why this
one is separate.
Verification
pytest tests/artifact tests/convert: 53 at this commit against 47 on master. The six newones are the stacked round trip, the collapse to a single divisor when the sources agree, the
128-row tile rule, the refusal outside NVFP4, and two on the activation divisor: that an
AllowA4input with no override carries the source's own calibrated word, and that a stacked parent takes the
smallest of its sources'. All six are synthetic; the end-to-end conversion of a vendor
checkpoint lands with the recipe, in the last commit.
ctest120 of 120 — the C++ refusals and the multi-word read ride on it rather than on a test oftheir own.
Checks not run
validation and the
stackedbranch are exercised only from the commit that adds a reader.conditions are inert for it.
belongs to the commit that adds the reader.
🤖 Generated with Claude Code