Skip to content

feat(BACKEND-TENSTORRENT-KEEPQUANT): dot Q4_K weights on device via the W1 decode (#2959) - #3005

Merged
lu-zero merged 2 commits into
mudler:mainfrom
lu-zero:row/BACKEND-TENSTORRENT-KEEPQUANT-W2
Sep 6, 2026
Merged

feat(BACKEND-TENSTORRENT-KEEPQUANT): dot Q4_K weights on device via the W1 decode (#2959)#3005
lu-zero merged 2 commits into
mudler:mainfrom
lu-zero:row/BACKEND-TENSTORRENT-KEEPQUANT-W2

Conversation

@lu-zero

@lu-zero lu-zero commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

feat(BACKEND-TENSTORRENT-KEEPQUANT): dot Q4_K weights on device via the W1 decode (#2959)

W2 of the row: the keep-quant DOT. MatmulBTQuantKernel registers
kMatmulBTQuant on the P150 and is reached through vt::MatmulBT's block-weight
dispatch (ops.cpp:163) — the entry every model matmul helper already uses, so
the test drives the public seam and not a hand-cast op pointer. tt-metal has
no packed-weight GEMM primitive, so the composition decodes the blocks
through the W1 bit-exact chain (DecodeQ4KBlocksF32, factored out of the
decode kernel verbatim — one decode, two consumers, one numerics authority),
rounds the decoded weight once to bf16 (RNE, the device round-once
convention), and runs the same bf16 tile matmul as kMatmulBT. The M=1 GEMV —
the production decode shape — is in the sweep.

The second half is the kTENSTORRENT arm of DeviceKeepQuantSupported,
admitting exactly {Q4_K}. The P150 is discrete with no CPU fallback tier, so
an arm wider than the registered kernel set throws at first forward with the
model resident — the exact failure this predicate exists to prevent, and the
reason Q5_K/Q6_K/Q8_0 stay on the expand-bf16 residency until their W4
kernels land. The routing test pins the set host-side; a reviewer who widens
the arm without widening the kernel reds it.

Red-first on both halves, captured before the implementation: the device
test's REQUIRE(OpRegistered(kMatmulBTQuant, kTENSTORRENT)) fatalled, and all
six unimplemented encodings routed kKeepQuant on TT under the default arm —
the too-wide answer this change replaces. Green after: the dot sits inside
the analytic bf16 operand-rounding envelope of the decode-based reference
(worst bound-ratio 0.53 across M {1,5} x K {256,512}; the bound is computed
from the data, 1.05 * 2^-8 * (sum |a_k w_k| + |ref|), not a picked
tolerance), full suites 54/54 and 53/53 on the device.

Reachability note: the op-level dot is reachable, the LOADER path is not
until the W3 e2e battery runs a Q4_K GGUF end to end on the P150; that scope
is named here, owned by the row, and listed in the spec's Owed.

The record commit moves the spec's Now to W2 complete and the backend
matrix's KEEPQUANT row to its new position (W1 landed in #2989, W2 on the
branch, W3 capture-safe residency plus the e2e vehicle battery, W4
Q5_K/Q6_K/Q8_0 with the int8 lever and the 27B arm). The vehicle GGUF was
fetched under the developer's 2026-09-05 session grant and recorded in
developer-preferences.md; its pin lands in docs/USAGE.md when the arm first
runs (W3), per the trigger the spec body sets. The GDN row's registration
citation shifted again with this kernel insert (6687-6702 to 6755-6770) and
is repaired to the same block's new home — the anchor, never the baseline.

Opens against main after #2989 merges (the stacked base row/...-W1 exists
only on the fork, and GitHub resolves a PR base in the upstream repo — the
diff becomes W2-only the moment #2989 lands). Full preflight on this tree:
exit 0 (the five standing skips are the host-inherent ones every prior TT PR
carried).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]

…he W1 decode (mudler#2959)

W2 of the row: the keep-quant DOT. MatmulBTQuantKernel registers
kMatmulBTQuant on the P150 and is reached through vt::MatmulBT's block-weight
dispatch (ops.cpp:163) — the entry every model matmul helper already uses, so
the test drives the public seam and not a hand-cast op pointer. tt-metal has
no packed-weight GEMM primitive, so the composition decodes the blocks
through the W1 bit-exact chain (DecodeQ4KBlocksF32, factored out of the
decode kernel verbatim — one decode, two consumers, one numerics authority),
rounds the decoded weight once to bf16 (RNE, the device round-once
convention), and runs the same bf16 tile matmul as kMatmulBT. The M=1 GEMV —
the production decode shape — is in the sweep.

The second half is the kTENSTORRENT arm of DeviceKeepQuantSupported,
admitting exactly {Q4_K}. The P150 is discrete with no CPU fallback tier, so
an arm wider than the registered kernel set throws at first forward with the
model resident — the exact failure this predicate exists to prevent, and the
reason Q5_K/Q6_K/Q8_0 stay on the expand-bf16 residency until their W4
kernels land. The routing test pins the set host-side; a reviewer who widens
the arm without widening the kernel reds it.

Red-first on both halves, captured before the implementation: the device
test's REQUIRE(OpRegistered(kMatmulBTQuant, kTENSTORRENT)) fatalled, and all
six unimplemented encodings routed kKeepQuant on TT under the default arm —
the too-wide answer this change replaces. Green after: the dot sits inside
the analytic bf16 operand-rounding envelope of the decode-based reference
(worst bound-ratio 0.53 across M {1,5} x K {256,512}; the bound is computed
from the data, 1.05 * 2^-8 * (sum |a_k w_k| + |ref|), not a picked
tolerance), full suites 54/54 and 53/53 on the device.

Reachability note: the op-level dot is reachable, the LOADER path is not
until the W3 e2e battery runs a Q4_K GGUF end to end on the P150; that scope
is named here, owned by the row, and listed in the spec's Owed.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…chor again (mudler#2959)

The spec's Now moves to W2 complete and the backend matrix's KEEPQUANT row
states the new position: W1 landed in mudler#2989, W2 (dot + predicate) is on the
branch, W3 owns capture-safe residency plus the e2e vehicle battery, W4 owns
Q5_K/Q6_K/Q8_0, the int8 lever and the 27B arm. The vehicle GGUF was fetched
under the developer's 2026-09-05 session grant and recorded in
developer-preferences.md; its pin lands in docs/USAGE.md when the arm first
runs (W3), per the trigger the spec body sets.

The GDN row's registration citation shifted again — this row's W2 kernel
insert moved tenstorrent_ops.cpp's registration block from 6687-6702 to
6755-6770 and the record gate flagged it; the citation now points at the
block's new home. Anchor repaired, not the baseline — the same repair W1's
record commit made, which is the cost of a keyed record citing line ranges
in a file this row keeps extending.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
@lu-zero
lu-zero force-pushed the row/BACKEND-TENSTORRENT-KEEPQUANT-W2 branch from 90b7aca to 40c69f4 Compare September 6, 2026 07:28
@lu-zero
lu-zero merged commit 943ffd0 into mudler:main Sep 6, 2026
23 of 27 checks passed
@lu-zero
lu-zero deleted the row/BACKEND-TENSTORRENT-KEEPQUANT-W2 branch September 6, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant