Skip to content

[2/5] feat(artifact): carry one NVFP4 divisor per source matrix in a stacked tensor - #287

Open
MichaelDementii wants to merge 2 commits into
Neroued:masterfrom
MichaelDementii:prep/artifact-nvfp4-source-divisors
Open

MichaelDementii wants to merge 2 commits into
Neroued:masterfrom
MichaelDementii:prep/artifact-nvfp4-source-divisors

Conversation

@MichaelDementii

Copy link
Copy Markdown
Contributor

Related Issue: #283

Second of five, and the only product-contract change in the set. Independent of the first; the third
needs both. No performance claim.

What changes

A tensor object holds one NVFP4 weight divisor. That is right for a plane quantised as one matrix and
wrong for a plane assembled from several quantised apart — which is what every published NVFP4
checkpoint of an MoE model is: each expert matrix carries its own weight_global_scale, 30 720 of
them on the routed banks of qwen3_6_35b_a3b.

The object gains an optional divisors, the number of separately quantised source matrices stacked
into the plane; the layout and the row-to-word rule are on the Issue. Default 1.

Contract

divisors absent means one word and today's layout exactly, so the stored form of every existing
artifact is unchanged. The reader is not: validate_nvfp4_weight gains two refusal conditions that
the six shipped dense NVFP4 wrappers run through, so a Weight built outside native_weight now has
to carry a divisor pointer. That is why two in-tree fixtures move with it —
tests/ops/quantized_weight.h and bench/ops/quantized_weight.cuh.

Four refusals in total: divisors outside NVFP4, a count that does not divide the rows, a share that
is not a whole number of scale tiles, and sources with unequal row counts. Two of them are the C++
conditions above; all four are enforced on the Python side. The count itself is decided where the
parent's shape is chosen, in the recipe plan, rather than being discovered at encode time.

The documents that own the rule move with it: the nvfp4 row, the reconstruction equation, and the
producer and reader obligations of docs/maintainer/tensor-formats.md; the payload_bytes formula
in docs/maintainer/storage-layouts.md; the object-member table of
docs/maintainer/artifact-container.md §5.1.

validate_nvfp4_weight also gains a defaulted stacked flag that every route in this commit leaves
false, so a multi-divisor plane is refused everywhere until the route that can read one arrives.
Nothing in the engine reads more than one divisor yet; that is the next commit, and it is why this
one is separate.

Verification

pytest tests/artifact tests/convert: 53 at this commit against 47 on master. The six new
ones are the stacked round trip, the collapse to a single divisor when the sources agree, the
128-row tile rule, the refusal outside NVFP4, and two on the activation divisor: that an AllowA4
input with no override carries the source's own calibrated word, and that a stacked parent takes the
smallest of its sources'. All six are synthetic; the end-to-end conversion of a vendor
checkpoint lands with the recipe, in the last commit.

ctest 120 of 120 — the C++ refusals and the multi-word read ride on it rather than on a test of
their own.

Checks not run

  • No C++ test constructs a plane with more than one divisor. The multi-word read, the per-word
    validation and the stacked branch are exercised only from the commit that adds a reader.
  • No existing artifact of another architecture was loaded again to confirm the two new C++ refusal
    conditions are inert for it.
  • Nothing measured: the divisors are read once per row inside an existing epilogue, and timing
    belongs to the commit that adds the reader.

🤖 Generated with Claude Code

…essing

The four decode kernels addressed a matrix from the call site: a flat group index built from the
row, and three plane pointers picked by hand. That holds while every codec keeps one scale per group
per row. It does not generalise - a codec whose scale plane is swizzled needs the row to form an
address, and the call site cannot compute it without already knowing the codec.

A codec now receives the matrix as a plane set plus a row, with the column count as a template
argument. The shared expert becomes a codec parameter in the same move: every registered profile
stores it as Q8 today, but that is a property of the profiles rather than of the kernels, and the
kernels no longer assert it.

No behaviour change. The arithmetic, the accumulation order and the launch geometry are untouched
for all four registered codecs.
…d tensor

A tensor object holds one NVFP4 weight divisor, which is right for a plane quantised as one matrix
and wrong for a plane assembled from several that were quantised apart. Published NVFP4 checkpoints
of MoE models are the second case: each expert matrix carries its own weight_global_scale, so the
converter has to refuse the pack.

The tensor object gains an optional `divisors`: the number of separately quantised source matrices
stacked into the plane, default 1. The payload keeps that many FP32 words immediately after the
scale plane, in stacking order, so payload_bytes = weight_divisor_offset + 4 * divisors, and row r
of an N-row plane uses word r / (N / divisors). A divisor's share must be a whole number of 128-row
scale tiles, so a boundary never splits one.

Absent means one word and the present layout exactly, so every existing artifact stays valid and no
reader changes behaviour. The count is decided where the parent's shape is chosen, in the recipe
plan, rather than being discovered at encode time; the converter refuses it outside NVFP4, and
`validate_nvfp4_weight` refuses a plane whose divisor count does not divide its rows.
@MichaelDementii

Copy link
Copy Markdown
Contributor Author

This one adds a divisors member to the NVFP4 tensor record so a stacked parent can keep one
per-source divisor instead of one for the whole plane, and it is the PR I am least attached to: if
you would rather have the rescale at conversion time, this withdraws and [5/5] changes with it.
Worth checking closely: that an artifact written before this change still binds (the member defaults
to 1 and the index collapses to word 0), and the activation-divisor rule in tools/convert, which
gates on the number of sources rather than the number of divisors — the other way round silently
breaks the two merged 27B recipes, and there are now two tests standing on exactly that.

@codex review

iamwavecut added a commit to iamwavecut/ninfer-3090 that referenced this pull request Sep 25, 2026
…d tensor

A tensor object holds one NVFP4 weight divisor, which is right for a plane quantised as one matrix
and wrong for a plane assembled from several that were quantised apart. Published NVFP4 checkpoints
of MoE models are the second case: each expert matrix carries its own weight_global_scale, so the
converter has to refuse the pack.

The tensor object gains an optional `divisors`: the number of separately quantised source matrices
stacked into the plane, default 1. The payload keeps that many FP32 words immediately after the
scale plane, in stacking order, so payload_bytes = weight_divisor_offset + 4 * divisors, and row r
of an N-row plane uses word r / (N / divisors). A divisor's share must be a whole number of 128-row
scale tiles, so a boundary never splits one.

Absent means one word and the present layout exactly, so every existing artifact stays valid and no
reader changes behaviour. The count is decided where the parent's shape is chosen, in the recipe
plan, rather than being discovered at encode time; the converter refuses it outside NVFP4, and
`validate_nvfp4_weight` refuses a plane whose divisor count does not divide its rows.

Ported from upstream PR Neroued#287.

Co-authored-by: MichaelDementii <136074657+MichaelDementii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants