feat(BACKEND-TENSTORRENT-KEEPQUANT): dot Q4_K weights on device via the W1 decode (#2959) - #3005
Merged
lu-zero merged 2 commits intoSep 6, 2026
Merged
Conversation
…he W1 decode (mudler#2959) W2 of the row: the keep-quant DOT. MatmulBTQuantKernel registers kMatmulBTQuant on the P150 and is reached through vt::MatmulBT's block-weight dispatch (ops.cpp:163) — the entry every model matmul helper already uses, so the test drives the public seam and not a hand-cast op pointer. tt-metal has no packed-weight GEMM primitive, so the composition decodes the blocks through the W1 bit-exact chain (DecodeQ4KBlocksF32, factored out of the decode kernel verbatim — one decode, two consumers, one numerics authority), rounds the decoded weight once to bf16 (RNE, the device round-once convention), and runs the same bf16 tile matmul as kMatmulBT. The M=1 GEMV — the production decode shape — is in the sweep. The second half is the kTENSTORRENT arm of DeviceKeepQuantSupported, admitting exactly {Q4_K}. The P150 is discrete with no CPU fallback tier, so an arm wider than the registered kernel set throws at first forward with the model resident — the exact failure this predicate exists to prevent, and the reason Q5_K/Q6_K/Q8_0 stay on the expand-bf16 residency until their W4 kernels land. The routing test pins the set host-side; a reviewer who widens the arm without widening the kernel reds it. Red-first on both halves, captured before the implementation: the device test's REQUIRE(OpRegistered(kMatmulBTQuant, kTENSTORRENT)) fatalled, and all six unimplemented encodings routed kKeepQuant on TT under the default arm — the too-wide answer this change replaces. Green after: the dot sits inside the analytic bf16 operand-rounding envelope of the decode-based reference (worst bound-ratio 0.53 across M {1,5} x K {256,512}; the bound is computed from the data, 1.05 * 2^-8 * (sum |a_k w_k| + |ref|), not a picked tolerance), full suites 54/54 and 53/53 on the device. Reachability note: the op-level dot is reachable, the LOADER path is not until the W3 e2e battery runs a Q4_K GGUF end to end on the P150; that scope is named here, owned by the row, and listed in the spec's Owed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…chor again (mudler#2959) The spec's Now moves to W2 complete and the backend matrix's KEEPQUANT row states the new position: W1 landed in mudler#2989, W2 (dot + predicate) is on the branch, W3 owns capture-safe residency plus the e2e vehicle battery, W4 owns Q5_K/Q6_K/Q8_0, the int8 lever and the 27B arm. The vehicle GGUF was fetched under the developer's 2026-09-05 session grant and recorded in developer-preferences.md; its pin lands in docs/USAGE.md when the arm first runs (W3), per the trigger the spec body sets. The GDN row's registration citation shifted again — this row's W2 kernel insert moved tenstorrent_ops.cpp's registration block from 6687-6702 to 6755-6770 and the record gate flagged it; the citation now points at the block's new home. Anchor repaired, not the baseline — the same repair W1's record commit made, which is the cost of a keyed record citing line ranges in a file this row keeps extending. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/BACKEND-TENSTORRENT-KEEPQUANT-W2
branch
from
September 6, 2026 07:28
90b7aca to
40c69f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(BACKEND-TENSTORRENT-KEEPQUANT): dot Q4_K weights on device via the W1 decode (#2959)
W2 of the row: the keep-quant DOT. MatmulBTQuantKernel registers
kMatmulBTQuant on the P150 and is reached through vt::MatmulBT's block-weight
dispatch (ops.cpp:163) — the entry every model matmul helper already uses, so
the test drives the public seam and not a hand-cast op pointer. tt-metal has
no packed-weight GEMM primitive, so the composition decodes the blocks
through the W1 bit-exact chain (DecodeQ4KBlocksF32, factored out of the
decode kernel verbatim — one decode, two consumers, one numerics authority),
rounds the decoded weight once to bf16 (RNE, the device round-once
convention), and runs the same bf16 tile matmul as kMatmulBT. The M=1 GEMV —
the production decode shape — is in the sweep.
The second half is the kTENSTORRENT arm of DeviceKeepQuantSupported,
admitting exactly {Q4_K}. The P150 is discrete with no CPU fallback tier, so
an arm wider than the registered kernel set throws at first forward with the
model resident — the exact failure this predicate exists to prevent, and the
reason Q5_K/Q6_K/Q8_0 stay on the expand-bf16 residency until their W4
kernels land. The routing test pins the set host-side; a reviewer who widens
the arm without widening the kernel reds it.
Red-first on both halves, captured before the implementation: the device
test's REQUIRE(OpRegistered(kMatmulBTQuant, kTENSTORRENT)) fatalled, and all
six unimplemented encodings routed kKeepQuant on TT under the default arm —
the too-wide answer this change replaces. Green after: the dot sits inside
the analytic bf16 operand-rounding envelope of the decode-based reference
(worst bound-ratio 0.53 across M {1,5} x K {256,512}; the bound is computed
from the data, 1.05 * 2^-8 * (sum |a_k w_k| + |ref|), not a picked
tolerance), full suites 54/54 and 53/53 on the device.
Reachability note: the op-level dot is reachable, the LOADER path is not
until the W3 e2e battery runs a Q4_K GGUF end to end on the P150; that scope
is named here, owned by the row, and listed in the spec's Owed.
The record commit moves the spec's Now to W2 complete and the backend
matrix's KEEPQUANT row to its new position (W1 landed in #2989, W2 on the
branch, W3 capture-safe residency plus the e2e vehicle battery, W4
Q5_K/Q6_K/Q8_0 with the int8 lever and the 27B arm). The vehicle GGUF was
fetched under the developer's 2026-09-05 session grant and recorded in
developer-preferences.md; its pin lands in docs/USAGE.md when the arm first
runs (W3), per the trigger the spec body sets. The GDN row's registration
citation shifted again with this kernel insert (6687-6702 to 6755-6770) and
is repaired to the same block's new home — the anchor, never the baseline.
Opens against main after #2989 merges (the stacked base row/...-W1 exists
only on the fork, and GitHub resolves a PR base in the upstream repo — the
diff becomes W2-only the moment #2989 lands). Full preflight on this tree:
exit 0 (the five standing skips are the host-inherent ones every prior TT PR
carried).
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]