[3/5] feat(ops): add an NVFP4 expert profile to sparse_moe

[MichaelDementii]

Related Issue: #283

Third of five. This is the mechanism; the first two are its prerequisites, the last two turn it on.

Level of the claim: operator. The end-to-end numbers are on the Issue; they are the confirmation,
and they need commits 4 and 5 to be reachable at all.

What changes

ops::sparse_moe admitted expert banks as Q4+Q5/Q6 or Q8+Q8 and nothing else. This registers an
NVFP4 profile across all three routes — decode, small-token, prefill — because the operation has to
answer for every token count.

Decode reads NVFP4 against represented BF16 activations. The kernels are CUDA-core dot products,
so four-bit activations buy no arithmetic there and would cost a quantiser of their own; keeping the
activation represented puts rel_l2 at T=1 at 1.60e-3 instead of 1.70e-1. The D4 block takes two
hidden rows, because the scale plane puts rows r and r+1 in one 32-byte sector; the kernel goes
10.2 → 8.0 µs and a third and fourth row start the next sector and buy nothing.

Prefill quantises the chunk once and runs W4A4 over the existing work list. The gate/up epilogue
writes the SwiGLU intermediate already encoded, so down needs no separate quantiser. The per-source
divisor is applied in that epilogue rather than inside the MMA loop — by a shift where the stride is a
power of two, which is every stride this model produces, with a division kept for the general case.

Dispatch. The frontier between the small-token and prefill routes is 13 for this profile, swept
cold over [2,46] against its own two routes rather than inherited from the groupwise 47, and the
small-token schedule comes from the same sweep.

Contract and effect on other paths

The profile costs no workspace. Its 3744 bytes a token of staging are carved out of
routed_storage — the routed SwiGLU plane it does not have, because the gate/up epilogue writes the
intermediate already encoded and the reduction is folded into the shared-down epilogue. That plane
holds 8192 a token; the header states the rule for its three lifetime unions, and this is the
fourth. The bench reports a groupwise profile's arena unchanged, 43 378 176 bytes at T=1024.

Three launches written for this profile are gated on it: the small-token router, whose reduction
order differs from the MMA router's and so cannot be bit-identical; the fused selection/scan/index
block at T ≤ 8; and a narrower route-job bound. Ungated, a groupwise call would take all three on
the ragged tail of a sliced chunk, where 4096k + m produces a final slice of m. Each may be
worth having there — a separate proposal with its own evidence, not a passenger here.

The wrapper also gains an invariant the route depends on: the routed and shared banks must agree on
their activation divisor, because the route encodes the chunk once and both gate/up GEMMs read that
one plane. Two divisors would scale the shared expert's whole contribution by their ratio, silently.

nvfp4_w4a4_mma_kernel gains two defaulted policies and an early-out so the routed launch can fold
its work list into the tile index. Five shipped dense operators launch it across nine translation
units; I compiled all nine from both trees with the production line and compared SASS. 133 of 169
kernels have an identical opcode mix; the other 36 differ by 48 instructions in 237 994 — IADD
against NOP padding on most, LEA/LOP3/MOV/SHF against IMAD/CS2R on six — and the
raster policy computes the blockIdx mapping master computed inline. Equivalent work, not identical
code; I have not localised the deltas or measured whether they cost anything.

Verification

Witness against master on the profiles this does not add — an FNV-1a 64 over the decoded output of
every operator case, two binaries: identical on all 16 shipped-profile cases, with one case added
(Q8+Q8 at T=4097, which the fixture did not carry before).

Operator, ninfer_sparse_moe_bench from the tree, production geometry, graph replay, 256 MiB L2
flush, both routed distributions:

./build/bench/ninfer_sparse_moe_bench --codec all --sweep 1:8192 --distribution trace-like \
    --execution graph --cache cold --warmup 5 --repeat 50 --flush-mib 256

The fixture stacks divisors the way the artifact does — 512 rows a divisor on routed gate/up, 2048 on
routed down — so the epilogue runs its per-source path rather than a uniform one. Median µs, this
profile against Q4+Q5:

T	trace-like Q4+Q5 → NVFP4		same Q4+Q5 → NVFP4		null arm
1	26.624 → 28.672	+7.7 %	26.624 → 28.672	+7.7 %	0.0 %
2	38.624 → 38.912	+0.7 %	32.768 → 36.512	+11.4 %	1.0 %
4	53.248 → 55.296	+3.8 %	45.056 → 47.104	+4.5 %	0.0 %
8	79.872 → 86.016	+7.7 %	69.632 → 73.728	+5.9 %	2.6 %
13	118.784 → 112.640	−5.2 %	104.448 → 51.200	−51.0 %	0.0 %
64	356.416 → 249.856	−29.9 %	108.544 → 71.680	−34.0 %	0.5 %
256	487.424 → 378.880	−22.3 %	167.936 → 81.920	−51.2 %	0.0 %
1024	694.272 → 393.216	−43.4 %	436.224 → 161.792	−62.9 %	0.0 %
4096	1796.096 → 698.368	−61.1 %	1597.472 → 557.056	−65.1 %	0.3 %
8192	3656.736 → 1392.640	−61.9 %	3214.368 → 1120.256	−65.1 %	0.3 %

The last column is the null arm: the same binary over the same row a second time. Taken three times
over the 96 rows of the sweep, its ninetieth percentile lands at 0.29, 0.41 and 0.57 %, and its
worst single row at 1.6, 3.2 and 5.8 % — so the floor itself is only good to a factor of two, and
nothing under about a per cent here is a measurement.

The rows below T=13 are one or two ticks of the instrument. Every median the bench prints is an
exact multiple of 2.048 µs, and those rows differ by exactly that. Both passes land on the same
tick, so the direction is reproducible, but the figure is a quantisation and should be read as one
tick rather than as 3.8 %. From T=13 the difference is tens of ticks.

The other two shipped profiles are in the same sweep as controls: Q4+Q6 runs within 3 % of Q4+Q5
everywhere, and Q8+Q8 is 17 to 120 % slower than it. The comparison is against Q4+Q5 because that
is the profile this model's artifact uses.

Roofline at T=1024, trace-like: 670.9 → 1159.2 GB/s of unique weight bytes, 85.3 → 150.2 TFLOP/s
logical. This stand's own DRAM read peak, measured with ncu on this kernel family, is 1765 GB/s
with 89.8 % of it observed as attainable, so the two points are 38.0 % → 65.7 % of peak, and
73.2 % of what this kernel reaches (raw/ncu_roofline_2026-09-17.txt). On the arithmetic side the
candidate is at 7.5 % of the NVFP4 MMA tier and the Q4+Q5 baseline at 33.7 % of the bf16 one, so
neither arm is near compute-bound: the gain is bytes moved.

Provenance: this ceiling is a direct measurement of the instruction in its product register form on
this card, two independent runs agreeing to within 0.2%. That is more honest than quoting a constant
compiled into a benchmark, but it is our own measurement, not a vendor figure. The tier figures
(2003.9 and 253.4 TFLOP/s) are the same kind of measurement taken on a different RTX 5090.

Where this profile is behind is small T, and that is where speculation runs it. A verify step
runs the MoE at T = k+1 = 4, on the small-token route, and the table puts it one tick of the
instrument behind on both distributions — a figure at the resolution limit, not a measurement of
size. End to end it does not survive: over three runs the Issue's corpus puts MTP3 at medians of
−0.1, −0.5 and −0.6 %, against a null arm on the same benchmark that moves up to 1.20 %. Decode
without speculation is unchanged.

Numerical: the test walks T = 1, 2, 12, 13, 64, 768, 4097, both sides of the frontier, with a
tolerance either side — one threshold per profile was only right while a profile had one
computational form. Represented activation: rel-L2 2.09e-3 against 2.5e-3, worst element 1.79
against 5.32. Quantised: rel-L2 1.011e-1 against 1.8e-1.

gross_relative_to_max_reference moves 2.5e-3 → 4.0e-3, set by the output format rather than by
this fixture: BF16 keeps eight significand bits, so its relative ulp runs over (2⁻⁸, 2⁻⁷] and half
an ulp is at most 2⁻⁸ of the value. The old bound was below that floor — an element whose exact
value is 637.701 lands on 636, the nearest BF16 with the grid stepping by 4, and missed a limit of
1.66 by being correctly rounded. relative_l2 is unchanged and remains the binding term.

Strength control on the lookup the profile turns on: with the divisor index pinned to the plane's
first word, all seven cases fail, by up to 1200 times the L2 bound. The fixture gives every
source matrix its own divisor over a fourfold range.

ctest 120 of 120. RTX 5090 sm_120a, driver 616.64, CUDA 13.1, Release, WSL2.

Checks not run

• The [2,46] frontier sweep is not attached as raw.
• Whether the 48 instructions that move in the dense operators cost anything, and whether any of
  them lands inside an MMA loop rather than in the prologue.
• No register, shared-memory, spill or graph-node delta.
• The fixture's shared gate/up plane carries two divisors where this checkpoint's collapses to one,
  because its two source matrices happen to agree. That is the demanding direction, but it means the
  shared plane's stacked path is covered by the fixture and not by the artifact.
• One card, one architecture, one model; no concurrency above 1.

🤖 Generated with Claude Code

[MichaelDementii]

This is the profile itself, and the part I would most like read adversarially is the workspace: all
six NVFP4 staging planes are carved out of routed_storage, the routed SwiGLU plane this profile
never allocates because gate/up writes the intermediate already encoded and the reduction folds into
the shared-down epilogue. The static_assert in sparse_moe_prefill.h is the whole proof that the
planes fit, so if it is wrong the profile overruns its arena. Second: three launches are gated on
this profile rather than shared, including a small-token router whose reduction order cannot be
bit-identical to the MMA router's — ungated, a groupwise call would take them on the ragged tail of
a sliced chunk. Third: gross_relative_to_max_reference moves 2.5e-3 → 4.0e-3, argued from half a
BF16 ulp, not from this fixture.

@codex review