feat(BACKEND-TENSTORRENT-KEEPQUANT): Q5_K/Q6_K/Q8_0 decode chains and the q4km vehicle gate (#2959) - #3028
Merged
lu-zero merged 5 commits intoSep 6, 2026
Conversation
…t; W3 takes the decode set (mudler#2959) The fetched Q4_K_M vehicle is not a uniform Q4_K artifact: token_embd is Q6_K (tied head), attn_qkv/ssm_out are Q5_K, ssm_alpha/ssm_beta are Q8_0, and ffn_down/attn_v mix Q6_K into Q4_K layers — read from the GGUF tensor headers of the hashed bytes. Under the W2 predicate a first forward refuses by name, which is the predicate working. The spec's W3/W4 split assumed a pure-Q4_K vehicle, so the e2e battery it schedules could never run; this reconciles the record with the evidence instead of re-scoping in a commit body: the Q5_K/Q6_K/Q8_0 decodes and the predicate widening move from W4 into W3, and W4 keeps the int8-dot lever and the 27B arm. The amendment also commits the W3 staging design the survey produced: stage the i32 word form once per weight through the existing persistent-shadow machinery (36 i32 words are exactly 144 packed bytes — zero expansion, residency thesis intact), decode on-core from the resident words every call, CHECK on a mid-capture miss, and prove it with a capture dump x2 byte-identity leg that reds against W2's per-call host round trip. The Q6_K embedding gather-vs-staged-table decision and the teacher-forcing oracle (transformers on the dequantized artifact, never the bf16 checkpoint) are named as implementation-time decisions with their constraints. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…eSiluMul, host-side twin (mudler#2959) The q4km vehicle is mixed-quant (token_embd Q6_K, attn_qkv/ssm_out Q5_K, ssm_alpha/ssm_beta Q8_0), so the decode set the row's falsification moved into W3 lands here: bit-exact DecodeKeepQuantBlocksF32 chains for Q5_K, Q6_K, and Q8_0 beside the W1 Q4_K chain, each pinned against vt::cpu::BlockToFloat across the shape sweep. The weight twin is built host-side, not on device. W2's device-side twin decode OOM-fatalled the vehicle's first forward with a 4,068,474,880 B ttnn::where (addr2line: the signed-zero repair inside DecodeKeepQuantBlocksF32). The kernel now decodes blocks on the host (vt::cpu::BlockToFloat), pre-rounds each f32 through BF16ToF32(F32ToBF16(x)) so the uploaded values are exactly bf16, and uploads once via from_vector<float> — ttnn converts to BFLOAT16 Tile during the upload, the EnsureEmbedTableDevice precedent — memoized per host buffer in DecodedWeightShadow and dropped in UnregisterHostBuffer. vt::MoeSiluMul lands because the GGUF dense MLP's split gate/up path dispatches it (qwen3_5.cpp:7610); this vehicle never dispatches kSiluAndMul, and the e2e ladder's backend-proof list now asserts kMoeSiluMul for the keep-quant arm instead. Capture safety stays structural: EnsureKeepQuantWords refuses a capture-time arrival by name, and the staging counter (KeepQuantCaptureStagingWrites) proves the captured e2e stages zero word uploads — the accessors live in tenstorrent_device.h with inline no-ops for device-agnostic TUs. The i32 word-shadow machinery and kKeepQuantDecode stay registered and tested, but no production path reads them now; the residency reconciliation is recorded owed in the spec. Red-first: per-encoding decode-equality pins, the MoeSiluMul golden case, and the stages-zero-words-during-capture case each reded before their implementation. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…hicle's decode set (mudler#2959) DeviceKeepQuantSupported's Tenstorrent arm admits exactly the encodings src/vt/tenstorrent/tenstorrent_ops.cpp has a registered decode for — now {Q4_K, Q5_K, Q6_K, Q8_0} instead of {Q4_K} — because the vehicle's own histogram pulled Q5_K/Q6_K/Q8_0 from W4 into W3 and the kernels and the predicate widen IN THE SAME CHANGE: admitting an encoding without its kernel throws at first forward with the model resident, the exact failure this predicate exists to prevent. kQ4_0 has no TT arm at all and kQ2_K/kQ3_K stay owed, so they still refuse by name. tests/vllm/test_gguf_keep_quant.cpp pins the widened set and reds any admission past the registered kernel set — the GLM-5.3 W10 lesson, kept as a gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…golden pair (mudler#2959) The qwen3.5 gate gains a second arm: the SAME ladder shape driven through the unsloth Qwen3.5-0.8B Q4_K_M GGUF, checkpoint-gated on VLLM_CPP_QWEN35_Q4KM_GGUF with a loud SKIP when absent. RunGate takes the checkpoint explicitly (the bf16 arm keeps the pinned HF snapshot), and its backend-proof list is arm-aware: the keep-quant vehicle dispatches kMatmulBTQuant and kMoeSiluMul, never kMatmulBT and kSiluAndMul, so the quant path's e2e reach is asserted by the same selections>0/declines==0 proof (first e2e reachability evidence for the W3 decode). The bf16 arm's committed list is unchanged. A capture-staging CHECK (staged == 0 across the captured e2e) turns the mudler#2812 class into a gate failure, and the ladder records the keep-quant capture dump under its own arm name. The dequant dump tool (test_qwen35_gguf_q4km_dequant_dump.cpp) writes the safetensors dir the teacher-forcing oracle reads. Committed goldens (tests/parity/goldens/qwen35_gguf_q4km/): the generic anchor pair (our_ids.npy + neartie_gap_mnats.npy — oracle greedy teacher-forced, 0 token-divergent, max gap 0.125 nats), the Tenstorrent capture pair (our_ids_tenstorrent_capture.npy + neartie_gap_mnats_tenstorrent_capture.npy — 51/256 near-tie divergences, max gap 0.1875 nats, inside the 500-mnat band), the 16 prompt-id files, and the capture dump (md5 eb8fb9894c3e4e05504f769b4119f443, byte-identical across two tt-smi-reset runs). READY adjudication on the P150: 147/147 assertions, 16/16 prompts PASS — 11/16 STRICT token-exact vs oracle per-prompt greedy, 5/16 near-tie-band only, 0 forward-divergent — with 16 ops proven at 0 declines. The oracle is transformers on the DEQUANTIZED artifact, never the bf16 checkpoint (a different model's logits); no manifest.json (the committed golden convention carries pairs + prompts only; the recipe lives in the spec's Evidence section). No eager/ambient pair is owed: the gate keys on DecodeCaptureEnabled() and the eager arm is covered by the op-level eager decode-equality suites. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…w branch (mudler#2959) The spec gains its Evidence section: E1 op-level counts, the capture dump x2 byte-identity leg (md5, reset between), the teacher-forced golden derivation (oracle venv, dequant artifact, per-arm pair numbers), the READY adjudication (16/16 PASS, 11 strict / 5 near-tie, 0 forward-divergent, 0 declines), and the exact run recipe with the vehicle pin (unsloth/Qwen3.5-0.8B-GGUF @ 6ab46149, sha256 bd258782...dc517, 532517120 bytes). ## Now moves to evidence-complete, pending fresh review, preflight, and landing; the row stays ACTIVE until merge. ## Owed records the residency reconciliation the evidence forced into the open: the spec's residency thesis expected per-call on-core decode from the resident i32 word shadow with no bf16 twin, but the device-side twin decode OOM-fatalled the vehicle, so W3 ships a host-side twin (decode + bf16 pre-round + one upload, memoized per host buffer) while the word-shadow machinery and kKeepQuantDecode stay registered, tested, and capture-guarded with no production reader. Decide before W4: keep the twin as shipped, or restore on-core decode where the twin's memory cost is real (the 27B arm). Also recorded: no eager/ambient TT pair owed for this arm (the gate keys on DecodeCaptureEnabled(); the eager arm is covered by the op-level suites), and no manifest.json (golden convention). The backend matrix's GDN registration line anchor moves with the file that grew it (tenstorrent_ops.cpp 6755-6770 -> 6790-6805); no lifecycle change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/BACKEND-TENSTORRENT-KEEPQUANT-W3
branch
from
September 6, 2026 15:08
0bca828 to
9aa143f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(BACKEND-TENSTORRENT-KEEPQUANT): Q5_K/Q6_K/Q8_0 decode chains and the q4km vehicle gate (#2959)
The q4km vehicle is mixed-quant (token_embd Q6_K, attn_qkv/ssm_out Q5_K,
ssm_alpha/ssm_beta Q8_0), so W3 lands the full keep-quant decode set the
row's falsification moved here, together with the first end-to-end
vehicle gate on the arm.
What lands:
beside the W1 Q4_K chain, each pinned against vt::cpu::BlockToFloat
across the shape sweep.
the vehicle's first forward with a 4,068,474,880 B ttnn::where (the
signed-zero repair inside DecodeKeepQuantBlocksF32). The kernel now
decodes blocks on the host, pre-rounds each f32 through
BF16ToF32(F32ToBf16(x)) so the uploaded values are exactly bf16, and
uploads once via from_vector — memoized per host buffer.
registered decode set {Q4_K, Q5_K, Q6_K, Q8_0}. Kernels and predicate
widen in the same change: admitting an encoding without its kernel
throws at first forward with the model resident.
through unsloth Qwen3.5-0.8B Q4_K_M GGUF, checkpoint-gated on
VLLM_CPP_QWEN35_Q4KM_GGUF with a loud SKIP when absent. RunGate's
backend-proof list is arm-aware (the keep-quant vehicle dispatches
kMatmulBTQuant and kMoeSiluMul, never kMatmulBT and kSiluAndMul) — the
first e2e reachability evidence for the decode. A capture-staging CHECK
(staged == 0 across the captured e2e) turns the Bring up TT capture-arm golden pairs for Mistral-7B and Qwen3.5-0.8B after the #1625 capture-default flip #2812 class into a
gate failure.
Evidence: READY 147/147, 16/16 prompts PASS (11 strict, 5 near-tie, max
gap 0.188 nats inside the 500-mnat band, 0 forward-divergent, 0
declines); capture dump x2 byte-identical across a card reset (staging
counter 0); op suites 37/37, 7/7, 260/260, 10333/10333; build 1444/1444;
preflight 155 ok. Run recipe and the vehicle pin (unsloth/Qwen3.5-0.8B-GGUF
@ 6ab46149, sha256 bd258782..., 532517120 bytes) are in the spec.
Owed, recorded in the spec: the residency reconciliation. The spec's
thesis expected per-call on-core decode from the resident i32 word
shadow; W3 ships a host-side twin while the word-shadow machinery and
kKeepQuantDecode stay registered, tested, and capture-guarded with no
production reader. Decide before W4: keep the twin as shipped, or
restore on-core decode where the twin's memory cost is real (the 27B
arm).
Refs #2959
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]