feat(cuda-device): derive the m16n8k16 accumulator fragment layout#474
Open
honeyspoon wants to merge 1 commit into
Open
feat(cuda-device): derive the m16n8k16 accumulator fragment layout#474honeyspoon wants to merge 1 commit into
honeyspoon wants to merge 1 commit into
Conversation
`wmma::mma_m16n8k16_f32_f16` takes and returns `[f32; 4]`, and those four
values are not four adjacent elements of the output tile: each lane holds a
scattered quarter of a 16x8 tile. The intrinsic's own docs state the mapping as
index arithmetic the caller must reproduce exactly, and the safety section
notes that a different array order computes a different matrix operation.
Getting it wrong does not fail - it silently computes a wrong product. This
derives the mapping once so call sites do not each re-derive it.
The layout is a mixed-radix numeral. Substituting the PTX row/col formulas into
the row-major offset and regrouping:
offset = (group + 8*(j>>1))*8 + 2*tig + (j&1)
= (j&1)*1 + tig*2 + group*8 + (j>>1)*64
digit range place source
j & 1 2 1 fragment index, low bit
tig 4 2 lane % 4
group 8 8 lane / 4
j >> 1 2 64 fragment index, high bit
Place values are the running product of the lower ranges (1, 2, 8, 64), span
128, so the numeral is uniquely decodable and (lane, j) -> offset is injective.
Domain and codomain both have 128 points, so it is a bijection: the warp's
fragments tile the output exactly, no overlap and no gap.
What makes this awkward by hand is that the fragment index is split across two
non-adjacent digit positions, at places 1 and 64, interleaved with the lane
digits - which is why j 0 and 1 differ by one column while j 0 and 2 differ by
eight rows.
Provides `acc_coords`, `acc_offset`, `acc_matrix_offset` for the strided
destination a GEMM epilogue writes, and `acc_decode` as the inverse, exported
so a caller can round-trip an existing hand-rolled layout in a test and catch a
disagreement cheaply. All `const fn`, no unsafe.
Verified against hardware, not only against its own unit tests: real
`mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32` on sm_86 with A and B
gathered from the PTX ISA layouts and the result scattered by `acc_coords`,
compared to a host matmul with asymmetric inputs so a transposed or half-swapped
layout could not coincidentally match. 128 of 128 elements, worst absolute
difference 0. A bijection onto the wrong positions would pass the unit tests; it
would not pass that.
Tests cover the bijection via an ownership map, agreement between the numeral
and row/col forms, decode as a true inverse over all 128 points, the
two-position split of the fragment index, disjointness under strided placement,
and overflow rejection.
Scope: the C/D accumulator only. A and B are different numerals and would each
need the same treatment, though the hardware check exercises them indirectly.
Signed-off-by: abder <bobmatt911@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wmma::mma_m16n8k16_f32_f16takes and returns[f32; 4], and those four values are not four adjacent elements of the output tile — each lane holds a scattered quarter of a 16x8 tile. The intrinsic's own docs give the mapping as index arithmetic the caller must reproduce, and its safety section warns that a different array order computes a different matrix operation.That failure is silent: it produces a wrong product rather than an error. This derives the mapping once so call sites do not each re-derive it.
The layout is a mixed-radix numeral
Substituting the PTX row/col formulas into the row-major offset and regrouping:
Place values are the running product of the lower ranges (1, 2, 8, 64), span 128. So the numeral is uniquely decodable,
(lane, j) -> offsetis injective, and since domain and codomain both have 128 points it is a bijection — the warp's fragments tile the output exactly, no overlap and no gap.What makes this awkward to write by hand is that the fragment index is split across two non-adjacent digit positions, at places 1 and 64, interleaved with the lane digits. That is why
j0 and 1 differ by one column whilej0 and 2 differ by eight rows.API
All
const fn, nounsafe, no new dependencies.acc_coords(lane, j)acc_offset(lane, j)acc_matrix_offset(..., tile_row, tile_col, row_stride)Noneon overflowacc_decode(offset)acc_decodeis exported deliberately: a caller with an existing hand-rolled layout can round-trip it in a test and catch a disagreement cheaply, rather than discovering it as wrong numerics.Verified against hardware, not only against its own tests
A bijection onto the wrong positions would pass every unit test here. So the check that matters ran real
mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32on sm_86 (RTX A2000, CUDA 13.3), gathering A and B from the PTX ISA layouts, scattering the result withacc_coords, and comparing against a host matmul. Inputs are asymmetric and non-repeating so a transposed or half-swapped layout could not coincidentally match:That also validates the A and B fragment layouts indirectly — a wrong gather would have produced a wrong product.
Tests
Bijection via an ownership map (fails on both overlap and gaps), agreement between the numeral and row/col forms, decode as a true inverse over all 128 points, the two-position split of the fragment index, disjointness preserved under strided placement, and overflow rejection. 6 tests,
fmt --checkclean, zero warnings.Scope
The C/D accumulator only. A and B are different numerals and would each need the same treatment. Only sm_86 was available to test on; the formulas come from the PTX ISA rather than from hardware probing, and the layout is part of the instruction's ISA contract, so I would not expect divergence — but it is untested elsewhere.