feat(BACKEND-TENSTORRENT-KEEPQUANT): Tenstorrent keep-quant arms, staged 27B - #3044
Merged
lu-zero merged 10 commits intoSep 7, 2026
Merged
Conversation
…metic (mudler#2959) The spec's residency NEEDS_DECISION resolves by arithmetic, not taste. Measured on the pinned 27B artifact (unsloth/Qwen3.8-27B-GGUF @ fe1e2a23, sha256 7e78da5d...fe169): the bf16-twin residency W3 ships is a 0.8B-only shape — non-expert keep-quant twins need 18.47 GiB, the expert tower 32.37 GiB, token_embd 2.37 GiB, against a 32 GB device holding 15.92 GiB packed. W4 promotes the word-shadow on-core decode (landed in W2/W3, bit-exact vs vt::cpu::BlockToFloat, capture-guarded, and read by no production path) to the production matmul residency and narrows the twin to the embedding gather. The vehicle OOM history judged twin CONSTRUCTION (the 4,068,474,880 B ttnn::where), not shadow use. A new ## W4 section carries the two changes as two pull requests (developer decision): W4a the 27B arm (mudler#3030) — residency flip, the kMatmulBTQuantGrouped TT port (the seam and a production ROCm kernel already exist: ops.cpp:220, MatmulBTQuantGroupedKernelRocm, fed by the stacked keep-quant tower qwen3_5_gguf_weights.cpp:1287 — the TT backend registers only kMoeSiluMul), MTP blk.64.* handling, the e2e gate against the pinned llama.cpp b10451 oracle, and the docs/USAGE.md pin; W4b the int8-dot lever (mudler#3031), profile-first, sequenced. The MoE grouped line moves out of ## Owed into the scope; ## Now records the landing state. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…ler#2959) Wave-1's blanket residency flip is falsified on the 0.8B vehicle: the head is TIED, [248320,1024] Q6_K, 254,274,560 elements, and its per-call on-core decode re-materializes the 4,068,474,880 B ttnn::where transient the W3 gate memorized. The flip is correct for projection matmuls and wrong for logits-head-class weights, which decode every step. The amendment records the operator decision: weights above 128M elements keep the memoized bf16 twin (a decode transient costs ~16 B/elem peak against the twin's 2 B/elem permanent, and per-step unpack compute on a head is wasted); weights at or below the threshold consume the per-call shadow decode. On 27B the budget becomes ~16 GiB word shadows + 2.37 GiB embedding twin + 2.37 GiB output-head twin + activations on 32 GB, expert tower packed — it fits with headroom. On the 0.8B vehicle the gate keeps today's semantics. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…the grouped kernel (mudler#2959) The threshold flip's capture leg is the second and decisive falsification. With the threshold in place the vehicle passed focused and suite legs, then the first decode-step capture died: TT_FATAL, trace buffers of 425,754,624 B against a 52,428,800 B trace region (mesh_trace.cpp:81). A weight decoded per call persists as a bf16 tile inside the captured graph, and trace replay re-runs the decode every step — a throughput regression against the twin with no compensating win. The twin-for-all diagnostic re-ran the same gate 16/16 PASS, 147/147 (11 strict / 5 near-tie, max 0.188 nats, 0 forward-divergent), isolating the cause to the flipped weights. No element-count threshold repairs the class, and the four candidate repairs each fail: enlarging the trace region needs ~18 GB on 27B; capture-mode twins leave the word shadow feeding only the eager arm; persistent decode scratch defeats the residency arithmetic; a no-capture fallback breaks the capture model. The flip waves are cancelled and the branch diff is reverted; production dense keep-quant matmuls keep the landed W3 twin. W4a(1) re-anchors on the TT provider for vt::MatmulBTQuantGrouped: the ROCm reference (rocm_grouped_gemm.hip:1456) already reads packed block weights natively for Q8_0/Q4_K/Q6_K, E=1 covers dense and E=N covers the stacked expert tower, and the seam's production consumers (qwen3_5.cpp:6147) reach a registered provider without new wiring. Q5_K is owed on top of the reference. Steady-state 27B budget: ~20.6 GB weights + activations on 32 GB. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…nt provider (mudler#2959) The Tenstorrent backend now serves OpId::kMatmulBTQuantGrouped. The W4a re-anchor names this kernel as the row's production surface: the ROCm reference (rocm_grouped_gemm.hip:1456) reads packed block weights natively, E=1 covers dense, and E=N covers the stacked expert tower. The weight stays packed on device as staged i32 words. Per call the kernel reads the P expert ids from host, decodes only the selected [N,K] row slices through the W3 chain, rounds once to bf16, and runs one 2D tile matmul per group. Results concatenate into a single f32 commit. The registered encoding set is exactly Q4_K and Q8_0; Q5_K, Q6_K, and Q4_0 refuse with a message that names them. DecodeKeepQuantBlocksF32 is split into a slice form over staged words plus a whole-tensor wrapper, and the wrapper keeps the prior behavior. RED first: with no provider both new cases failed at the registration check, and the refusal text "no kernel for op MatmulBTQuantGrouped" was captured verbatim. GREEN: the routing case pins the admitted set and the refusals; the numerics case sweeps seven shapes, both arms, both encodings, and Pa=1 broadcast against the CPU grouped provider inside the W2-ratified analytic bf16 envelope. Worst error ratios are 0.37 to 0.72 of the bound, and the slice decode itself is bit-exact against the analytic dequant on Q8_0's bf16-exact scales. STAGED-UNREACHED: no production entry point reaches this provider yet. The vehicle's expert tower still routes through the W3 twin path, and the dense E=1 arm plus the 27B gate stay with wave-3 wiring tracked in issue mudler#3030 and the spec's wave plan. Capture compatibility is also open: the kernel does EnsureHost on ids per call and builds eager-path constructs, so a captured decode graph cannot include it today; that repair is owed by the same wave. Evidence: focused cases 7277/7277; full backend suite 60 cases / 13,964 assertions (baseline 58 / 6,687 plus exactly these two cases); vehicle gate 16/16 PASS unchanged at 147/147, 11 strict / 5 near-tie, zero forward-divergent. The same change repairs the one record anchor this diff staled: the GDN row's registration citation in backend-matrix.md moves 7244-7260 to 7422-7436, its pre-edit line range. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…uped keep-quant provider (mudler#2959) The kMatmulBTQuantGrouped registered set widens from {Q4_K, Q8_0} to exactly {Q4_K, Q5_K, Q6_K, Q8_0}. The 27B pin (issue mudler#3030) carries 67 Q6_K and 48 Q5_K tensors, so the stacked expert tower is not fully servable without them. Both encodings ride the wave-2a path unchanged: the tower stages as i32 words once, each call decodes only the P selected [N,K] row slices through the W3 chains (DecodeKeepQuantWordsF32), rounds once to bf16, and runs one tile matmul per group. No new decode arithmetic ships in this change. Q5_K has no ROCm grouped reference to mirror (rocm_grouped_gemm.hip admits Q8_0/Q4_K/Q6_K natively), so its decode derives from the W3 dense Q5_K chain, bit-exact vs vt::cpu::BlockToFloat; Q6_K mirrors the ROCm grouped kernel's native dequant through the same W3 chain. RED first: the routing REQUIRE demanded the four-encoding set and failed on the then-current admission (both new encodings refused by name), and the numerics sweep threw at its first Q5_K shape with the refusal text captured verbatim. GREEN: the routing case pins the four-encoding set plus the Q4_0 refusal; the numerics sweep grows from seven to fifteen shapes (per new encoding: E=1 dense, E=N tower, Pa=1 broadcast with non-tile N, and a 27B mirror at K=5120) against the CPU grouped provider inside the ratified analytic bf16 envelope. Worst bound ratios for the new arms: Q5_K 0.20-0.51, Q6_K 0.21-0.47. New slice-decode bit-exact legs compare bf16 bits against BlockToFloat at the routed element; the Q6_K leg constrains scales to strictly positive because a negative scale meeting a zero q makes the true dequant -0, which any dot flattens in the f32 accumulate (the decode itself keeps -0, per the W3 pin). STAGED-UNREACHED: unchanged from wave-2a — no production entry point reaches this provider yet; wave-3 owns the wiring and the 27B gate (issue mudler#3030), and the vehicle's expert tower still routes through the W3 twin path. Evidence: focused grouped cases 2/2, 20,769 assertions; full backend suite 60/60 cases, 27,456 assertions (wave-2a baseline 60 / 13,964 plus exactly these arms); clean full-tree build, 0 warnings; vehicle gate 16/16 PASS unchanged at 147/147, 11 strict / 5 near-tie, max gap 0.188 nats, 0 forward-divergent. The same change repairs the one record anchor this diff staled: the GDN row's registration citation in backend-matrix.md moves 7422-7436 to 7428-7442; the record-anchor ratchet holds at its baseline 33. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…xis and wave-3 shape (mudler#2959) A tensor-name scan of the pinned artifact settles what the dtype-only survey misread: arch qwen35, zero ffn_*_exps / ffn_gate_inp / exp_probs tensors, and blk.0 carries singular ffn_down/ffn_gate/ffn_up. There are no experts on the 27B; the 294 Q4_K tensors are dense ffn/attn weights. The "experts Q4_K x294 -> 9.81 GiB" split in the budget was wrong, and so was "the 27B gate is reachable only behind the dense E=1 arm" for the interim reason given. The corrected axis: the whole keep-quant set beyond the gather class (~14.27 GB packed) cannot exist as twins (~42 GB); the 27B is served packed through the E=1 arm with a chunked slice-decode + accumulate matmul, so the captured graph holds chunk buffers and never a whole-weight tile. Budget ~22-23 GB on 32 GB. Replay re-decodes each step — correct, slower — and throughput is W4b's lever. The E=N expert arm stays staged-owed behind a MoE artifact (30B-A3B class; none on disk; a download needs authority). Wave-2b landed 14e8fe4 (all four encodings, suite 27,456 green, vehicle 147/147, operator-passed, pushed). Wave-3a = E=1 chunked capture-compatible packed dense; wave-3b = 27B wiring + gate + MTP skip + USAGE pin. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
… keep-quant arm (mudler#2959) The grouped provider's E=1 arm now decodes [chunk, K] word ranges — one tile matmul per chunk, partials concatenated in f32 — instead of the whole-[N,K] slice, and it never reads the routing ids (E=1 ids are statically all zero): no EnsureHost, no host readback, every chunk offset a capture-time constant. E=N keeps the wave-2 whole-slice decode and its dynamic-id readback unchanged. Both capture-time failure modes are now measured, not assumed (P150, 2026-09-07, head shape [248320, 1024] Q6_K). Before: the whole-slice arm died in capture demanding 4,068,474,880 B of device DRAM — one 1.02 GiB f32 plane (bank_manager.cpp:462), the grouped arm's wave-1b mechanism. Many small chunks die the other way, on the trace region (mesh_trace.cpp:81): the captured command stream costs ~3.3 MB per chunk (485 chunks = 1,566,662,656 B; 16 = 53,764,096 B; 2-4 capture clean). CHUNK POLICY: bound the chain's largest live tensor — one [chunk, K] f32 plane — at 256 MiB (chunk = 256 MiB / (4 B * K), a whole decode when it fits) with ceil(N / 8) as the command-stream guard. The head decodes as 4 chunks: capture demand 13,484,032 B of the 52,428,800 B region, staging writes during capture 0, capture dumps x2 byte-identical, demand identical across both passes. Envelope: the grouped sweep under forced 3-row chunks reproduces every pre-change E=1 ratio bit-for-bit (vs cpu 0.378085, 0.449989, 0.390167, 0.487656, 0.491194, 0.278552, 0.376211; new E=1 broadcast shape 0.468875), and a dedicated bit-exact leg pins the chunked decode against BlockToFloat at forced chunk boundaries (81 assertions). Full backend suite 63/63 cases, 524,267 assertions. Vehicle battery 16/16 prompts, 147/147 assertions, 0 forward-divergent. Test hooks: KeepQuantChunkRowsOverrideForTest forces the chunk rows; the trace-bound leg reads VT_KEEPQUANT_TEST_CHUNK_ROWS so one build surveyed the demand curve; LastTraceBytesForTest reports the device trace demand. Staged-unreached: the E=1 arm stays unreached from a production entry point — the model wiring (27B + the qwen3_5 head routing) is wave-3b's gate (issue mudler#3030); the E=N capture indirection is staged-owed behind a MoE artifact (spec ## W4). Anchors moved: .agents/backend-matrix.md (KEEPQUANT row lifecycle and evidence to W2a/2b LANDED + W3a ON BRANCH, and the predicate registered- set phrase after wave-2b; owed narrows to the int8 lever, the wave-3b wiring, and the captured e2e vehicle leg) and the GDN row's registration anchor, which this diff's insertions shifted (tenstorrent_ops.cpp 7428-7442 -> 7517-7531; check-agent-record stale back at baseline 28). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…r chain — adopt the upstream trace-region policy The wave-3b-1 vehicle AFTER leg falsified the chunked E=1 arm's capture-safety at model scale: after the two device-level defects it exposed were fixed red-first (a chained ROW_MAJOR activation reaching ttnn::matmul unconverted, per_core_M = 0 at matmul_program_config.cpp:372; and the E=1 arm committing ROW_MAJOR into the output slot so replay's tile-padded view overran the buffer, mesh_tensor_impl.hpp:33 — focused 4/4, suite 66/66, 524,428/524,428 after both), capture still demanded 444,424,192 B of the 52,428,800 B region (mesh_trace.cpp:81). The demand is ~113 keep-quant weights x ~3.5 MB/chain of serialized decode commands — the wave-3a per-chain constant — so chunk sizing cannot pay it down; the cost multiplies by chain count. This reconciles the third amendment's rejected "trace-region enlargement": that rejection sized wave-1b's whole-weight-per-step decode planes (~18 GB); the measured quantity here is the command stream (0.44 GB on 0.8B, to be measured on 27B). The fifth amendment moves the region policy onto the pinned tt-metal's own practice (models/demos/utils/trace_region_sizes.py): dynamic (trace_region_size=0, the upstream default for unconfigured models and deepseek-v3's explicit choice) tried first on the vehicle; a per-model resolved region sized from measured demand is the fallback if the pre-ITEM-5 overlap hazard reproduces, with the fixed 50 MB (the vLLM plugin's generic value, ITEM 5) as that fallback's unspecified-model default. Capture demand stays a measured, reported axis; the perf guard records tokens/s both sides of the switch, and a gate that cannot complete inside the vehicle timeout is a NEEDS_DECISION stop. The twin-absence policy is unchanged. Spec-only record commit; the wave-3b-1 implementation diff stays uncommitted for wave-3b-1c to complete under the amended spec. Gate note: the record-anchor gate on the full worktree reports 29 > 28 solely from the pending wave-3b-1 diff's tenstorrent_ops.cpp shift (the GDN kL2Norm anchor at 7517-7531); this commit's content alone verifies clean at HEAD in a detached worktree (agent record OK), and the anchor repair rides the wave-3b-1 landing commit per its granted ride-along. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…hrough the chunked E=1 arm (mudler#3030) The dense keep-quant matmul IS the wave-3a chunked E=1 grouped arm: the packed i32 words stage once per weight (EnsureKeepQuantWords) and each call slice-decodes and accumulates in capture-safe chunks, so the decoded bf16 TWIN of wave-2/W3 is gone from the matmul path. Reached through vt::MatmulBT's public block-weight dispatch (ops.cpp:163) on all four registered encodings, the entry a GGUF load actually takes. The gather class keeps its embed-table twin on purpose: the vehicle's tied head shares the GGUF tensor with the embedding, so its gather keeps the twin while its matmul stages only the packed words. Two capture-path defects surfaced red-first under the switch and are fixed in the same change: the chained activation entered the grouped arm ROW_MAJOR where the arm tiles, and the output TILE committed before CommitDeviceLogical2D replicated it across the mesh (red logs /tmp/w4a3b1-red.log, /tmp/w4a3b1-red2.log). The device now opens with a DYNAMIC trace region (trace_region_size=0), mirroring the pinned tt-metal's own practice (models/demos/utils/trace_region_sizes.py), with VT_TT_TRACE_REGION_MB opting back into a fixed one (50 MB fallback on an unusable value) when a capture overlaps live DRAM. The old fixed 50 MB sized the region as a constant, and the chunked arm's captured command stream alone measures 444,424,192 B on the 0.8B vehicle (mesh_trace.cpp:81) — a per-chain cost no constant tracks. The vehicle ran the dynamic policy. The switch moves six prompt anchors (51 of 256 anchor cells re-derive), the headline being prompt[2] tok=5: 11 -> 13. Mechanism: the packed chunked head matmul resolves a different tt-metal program config than the twin whole-slice matmul over the same [248320,1024] Q4_K_M weight (per_core_N=19 with out_subblock 2x2 vs per_core_N=71), a different dst accumulation order, so last-ulp logit deltas reach band-edge cells. Evidence: the teacher-forced gap golden shows 0 forward-divergent cells across all 16 prompts and max gap 375 mnats (band 500), and the flipped cell is a 0.0-mnat exact tie under oracle logits with the third candidate 1.375 nats behind. Goldens re-derived through the documented path: VT_DUMP_IDS=1 capture plus the dequantized-oracle recipe, the fresh safetensors dump byte-identical to the committed-era artifact. New red-first residency tests: the warm-matmul pair flips the word/twin probes on all four encodings, and the gather-survivor case guards the embed twin (delete the embed map's insert and it stays red). Cost, accepted for the residency policy: the last-112-step replay window runs 2.95 s/cycle on the twin vs 18.02 s/cycle packed, ~6.1x — the replay re-decodes chunks from packed words instead of matmul-ing a resident bf16 tensor. Full backend suite 66/66 cases, 524,428 assertions. Vehicle battery 147/147 assertions, 16/16 prompts: 10 strict + 6 near-tie (11/5 before), max gap 0.375 nats @ prompt[9] tok=4, 0 forward-divergent. Anchors moved: .agents/backend-matrix.md's GDN registration anchor (tenstorrent_ops.cpp 7517-7531 -> 7495-7509, check-agent-record back at baseline 28) and scripts/env-doc-allowlist.txt, which gains VT_TT_TRACE_REGION_MB. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
… head skip, the 27B denominator goldens, and the chunk-plane cap (mudler#3042) Lands wave-3b-2 as a STAGED SLICE (the coordinator's resolution of the wave's NEEDS_DECISION): the production wiring and its witnesses commit now, and the 27B e2e gate stays UNREACHED, owned by [mudler#3042](mudler#3042), listed under `## Owed` in the row spec. Row BACKEND-TENSTORRENT-KEEPQUANT stays ACTIVE; W4b and the 27B gate are the open scope. Landed: - The production MTP drafter skip. The loader's accounting deliberately passes a declared head — its fifteen `blk.64.*` tensors ARE enumerated as expected — and the trunk-only load then leaves them unread, so `LogQwen3_5GgufMtpHeadSkip` prints the skip loud before any weight byte moves: all fifteen tensors, 289,527,808 B, named in full, suppressed only when speculative method `mtp` is configured (src/vllm/entrypoints/model_loader.cpp, src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp). The skip message carries the denominator fact: the pinned llama.cpp `b10451` oracle ignores the same tensors, so a gate against it is matched work only with this skip loud. - The 16-prompt oracle goldens `tests/parity/goldens/qwen38_gguf_q4km_27b/greedy_ids.npy`, derived from the byte-identical llama.cpp `b10451` denominator harness. The TT-side run against them closes the 27B gate. - The reachability test `tests/vllm/entrypoints/test_gguf_accounting_reach.cpp`, which proves the skip through the PRODUCTION loader accounting rather than a hand-built type. - The checkpoint-gated TEST_CASE in `tests/parity/test_qwen35_paged_engine.cpp`, inert with a loud SKIP until `VLLM_CPP_QWEN38_27B_GGUF` names the artifact — the tree's actual env name, read at that test's :588 and documented as such in `docs/USAGE.md`. - `VT_TT_KEEPQUANT_CHUNK_BYTES` (env-doc allowlisted), the keep-quant chunk-plane budget: default 256 MiB surveyed on the 0.8B vehicle, empty/unset keeps the default, a positive integer is a HARD CAP in bytes that trades command-stream length for live memory — the ceil(N/8) trace term otherwise forces a 606+ MB head plane whatever the budget says, exactly the alloc that died at 27B. - The `docs/USAGE.md` 27B arm entry, with the local sha256 pin and the recorded provenance caveat: at the checked unsloth revision no file matches this artifact by size or hash, so the pin rests on the hash, and the exact upstream file and revision are unresolved. UNREACHED, named per the staged-slice contract: the 27B e2e gate. The OOM evidence, eight runs — failing allocations 1,073,725,440 B and 134,184,960 B; free-at-failure 244 MB -> 46 MB -> 12.7 MB/bank; ~34 GB allocated against the 32 GB device with a 3.7 MB largest free block; batch budget 512 fails identically, so the demand is not activation-sized; three mitigations tried and failed. The residency sits ~11 GB above the ~22-23 GB surveyed design residency. Suspects, named and unmeasured: the 2.5 GB bf16 embed twin, f32 plane transients, possible words double-staging. The next lever is the device-side allocation trace, not another mitigation. mudler#3042 owns it. Ride-along anchor repair, authorized by the coordinator: the certified diff moved three record anchors, and the repair rides this landing commit (the f46d102 precedent) so check-agent-record returns to baseline 28 — engine-matrix.md SPEC-MTP-GGUF (qwen3_5_gguf_weights.cpp:1447 -> :1448) and SPEC-DFLASH-GGUF (:1034 -> :1035), backend-matrix.md BACKEND-TENSTORRENT-GDN (tenstorrent_ops.cpp 7495-7509 -> 7518-7532). Unrelated rows in both keyed records are byte-for-byte equal to their pre-diff state; no other row touched, no new anchor added. Spec amendment: the sixth amendment records the staged-slice landing, the owed gate, and the OOM evidence; the row spec's `## Owed` carries the gate with its owning issue. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(BACKEND-TENSTORRENT-KEEPQUANT): Tenstorrent keep-quant arms, staged 27B
W4a brings the keep-quant arms to the Tenstorrent backend and stages
the 27B checkpoint. The row stays open. #3031 owns the int8-dot
recovery. #3042 owns the owed 27B end-to-end gate.
What changed
each slice per call, applies one bf16 RNE, and runs a per-group tile
matmul. Q4_K, Q5_K, Q6_K, and Q8_0 are registered. The provider
refuses other encodings by name.
graph. The arm is capture-safe: it reads no ids back under capture.
commands against a 52,428,800 B region. The backend opens a dynamic
trace region first. VT_TT_TRACE_REGION_MB opts into a fixed region.
This mirrors the pinned tt-metal trace-region practice.
0.0 mnats. per_core_N 19 versus 71 changes the dst accumulation
order over bit-identical operands. The goldens are re-derived, and
the max gap is 375 mnats against the 500 band.
production loader skips the blk.64 MTP drafter head by name, for
denominator parity with llama.cpp b10451. Sixteen-prompt oracle
goldens ship. VT_TT_KEEPQUANT_CHUNK_BYTES caps the f32 chunk plane.
Eight runs exhausted 32 GB of device DRAM in keep-quant decode, so
27B dense keep-quant arm: decode residency exhausts device DRAM, e2e gate unmet #3042 tracks the gate. The skipped gate exits 77 and is never a
pass.
Packed replay runs 18.02 s/cycle against the twin's 2.95 s/cycle. The
slower path buys correctness. #3031 owes the recovery.
How to verify
Build with
-DVLLM_CPP_TENSTORRENT=ONand the tt-metal prefix path.Set
TT_METAL_RUNTIME_ROOTandLD_LIBRARY_PATHto the pinnedtt-metal build. Reset the board before each device leg.
./build/tests/test_tenstorrent_backendreports 66/66 cases and524,428/524,428 assertions.
./build/tests/test_qwen35_paged_engine -tc='*GGUF Q4_K_M*'withVLLM_CPP_QWEN35_Q4KM_GGUFreports 16/16 prompts PASS: 10 STRICTtoken-exact, 6 near-tie at 0.375 mnats maximum, 0 forward-divergent.
The run exits 77 because the gated 27B case skips loudly.
./build/tests/vllm/entrypoints/test_gguf_accounting_reachreports9 cases and 38 assertions.
skip reds the reach test at 30/38. Deleting the accounting refusal
reds it at 32/38. Silencing the 27B skip flips the vehicle to rc 0,
and the gate command then fails the change.
docs/USAGE.mdpins the checkpoints: the 0.8B vehicle, and the 27Bstaged arm by sha256 with its provenance caveat.
What remains out of scope
and the suspected structural contributors.
#3030 tracks the 27B arm broadly; #3042 now carries its live
residency evidence. Row:
BACKEND-TENSTORRENT-KEEPQUANT.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]