From 3e7599b4f1000408ce834b26512173612f1e0349 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Sun, 6 Sep 2026 20:48:53 +0200 Subject: [PATCH 01/10] =?UTF-8?q?record(BACKEND-TENSTORRENT-KEEPQUANT):=20?= =?UTF-8?q?W4=20scope=20=E2=80=94=20the=20residency=20arithmetic=20(#2959)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The spec's residency NEEDS_DECISION resolves by arithmetic, not taste. Measured on the pinned 27B artifact (unsloth/Qwen3.8-27B-GGUF @ fe1e2a23, sha256 7e78da5d...fe169): the bf16-twin residency W3 ships is a 0.8B-only shape — non-expert keep-quant twins need 18.47 GiB, the expert tower 32.37 GiB, token_embd 2.37 GiB, against a 32 GB device holding 15.92 GiB packed. W4 promotes the word-shadow on-core decode (landed in W2/W3, bit-exact vs vt::cpu::BlockToFloat, capture-guarded, and read by no production path) to the production matmul residency and narrows the twin to the embedding gather. The vehicle OOM history judged twin CONSTRUCTION (the 4,068,474,880 B ttnn::where), not shadow use. A new ## W4 section carries the two changes as two pull requests (developer decision): W4a the 27B arm (#3030) — residency flip, the kMatmulBTQuantGrouped TT port (the seam and a production ROCm kernel already exist: ops.cpp:220, MatmulBTQuantGroupedKernelRocm, fed by the stacked keep-quant tower qwen3_5_gguf_weights.cpp:1287 — the TT backend registers only kMoeSiluMul), MTP blk.64.* handling, the e2e gate against the pinned llama.cpp b10451 oracle, and the docs/USAGE.md pin; W4b the int8-dot lever (#3031), profile-first, sequenced. The MoE grouped line moves out of ## Owed into the scope; ## Now records the landing state. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/specs/tenstorrent-keepquant.md | 56 +++++++++++++++++++------- 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/.agents/specs/tenstorrent-keepquant.md b/.agents/specs/tenstorrent-keepquant.md index 5476f7eaa..0fea5f213 100644 --- a/.agents/specs/tenstorrent-keepquant.md +++ b/.agents/specs/tenstorrent-keepquant.md @@ -274,22 +274,26 @@ to make a failure pass. ## Owed -- MoE grouped keep-quant GEMM (follow-up row; vehicle pickable after W2). +- MoE grouped keep-quant GEMM — MOVED INTO W4 SCOPE (#3030): the seam + exists (`ops.cpp:220`, `weight[E*N,K]`) with a production ROCm kernel + (`MatmulBTQuantGroupedKernelRocm`, native Q8_0/Q4_K/Q6_K) fed by the + stacked keep-quant tower (`qwen3_5_gguf_weights.cpp:1287`); the TT + backend registers only `kMoeSiluMul`. "Vehicle pickable after W2" is + literally true. - Block-decoding n-gram gather ([#2394](https://github.com/mudler/vllm.cpp/issues/2394)). - IQ-family / sub-IQ1_S encodings (unsloth fork formats). -- The int8-dot perf lever; llama.cpp-comparable throughput numbers. +- The int8-dot perf lever (#3031); llama.cpp-comparable throughput numbers. - `docs/USAGE.md` vehicle pin when the arm first runs end to end (the W3 capture leg hashes the local bytes); 27B arm entry at W4. -- Residency reconciliation (NEEDS_DECISION): the spec's residency thesis - expected per-call on-core decode from the resident i32 word shadow and no - bf16 twin; W3 ships a host-side decode + bf16 pre-round + single - `from_vector` upload per weight (`DecodedWeightShadow` memoizes it - per host buffer, dropped in `UnregisterHostBuffer`), because the device-side - twin decode OOM-fatalled the vehicle (4,068,474,880 B `ttnn::where`). The - word-shadow machinery and `kKeepQuantDecode` stay registered, tested, and - capture-guarded, but no production path reads them now. Decide before W4: - keep the twin as the shipped path and retire the shadow, or restore - on-core decode for the 27B arm where the twin's memory cost is real. +- Residency reconciliation (RESOLVED BY ARITHMETIC, 2026-09-06, #3030): + the twin residency is a 0.8B-only shape. Measured on the pinned 27B + artifact: non-expert keep-quant twins need 18.47 GiB, the expert tower + 32.37 GiB, token_embd's twin 2.37 GiB — against a 32 GB device whose + packed total is 15.92 GiB. W4 promotes the word-shadow on-core decode to + the production matmul residency and narrows the twin to the embedding + gather. The vehicle's OOM history (device-side twin construction, + 4,068,474,880 B `ttnn::where`) judged twin CONSTRUCTION, not shadow use; + the shadow path's per-call decode is compute, not allocation. - No eager/ambient TT pair is owed for this arm: the READY gate keys on `DecodeCaptureEnabled()` and adjudicates the capture leg; the eager arm is covered by the op-level eager decode-equality suites (E1). The ladder's @@ -299,6 +303,28 @@ to make a failure pass. carries per-arm `.npy` pairs + `p{i}_prompt.i32` only; the recipe lives in this Evidence section and the landing commit body. +## W4 + +Two changes, two pull requests (developer decision 2026-09-06): + +**W4a — the 27B arm ([#3030](https://github.com/mudler/vllm.cpp/issues/3030)).** +(1) Production residency flip: `MatmulBTQuantKernel` consumes the per-call +on-core decode from the resident word shadow for {Q4_K,Q5_K,Q6_K,Q8_0}; the +twin survives for the embedding gather only. Red-first: the 0.8B vehicle +gate re-run under the flipped residency (the W3 gate stays green — the OOM +it memorized was twin construction). (2) `kMatmulBTQuantGrouped` on +kTENSTORRENT beside the W3 dense decode chains, consuming the existing +stacked tower. (3) MTP `blk.64.*` skip/refuse by name. (4) 27B e2e greedy +near-tie gate, checkpoint-gated opt-in loud-skip (#2811 precedent), goldens +vs the pinned llama.cpp b10451 oracle, 500-mnat band + 0 forward-divergent. +(5) `docs/USAGE.md` pin in the same change. Memory axis recorded: ~16 GiB +word shadows + 2.37 GiB embedding twin + activations on 32 GB. + +**W4b — the int8-dot lever ([#3031](https://github.com/mudler/vllm.cpp/issues/3031)).** +Quantized-domain integer vec_dot behind the same seam; profile-first +attribution; recorded-only throughput floor. Sequenced after W4a, never +bundled. + ## Now `ACTIVE`, 2026-09-06. W1 complete (#2989, open). W2 complete on the row @@ -314,6 +340,6 @@ carried the Q5_K/Q6_K/Q8_0 decodes and the predicate widening before the capture leg and the e2e battery (see the falsification section). W3 EVIDENCE COMPLETE on the row branch (see `## Evidence`): capture dump x2 byte-identity, staging counter 0, READY gate 16/16 PASS (11 strict / 5 -near-tie, 0 forward-divergent), backend proof 0 declines. Pending: fresh -review, preflight, landing. W4 owed: the int8 lever, the 27B arm, and the -residency reconciliation in `## Owed`. +near-tie, 0 forward-divergent), backend proof 0 declines. W3 LANDED +2026-09-06 (025c6ed90..f98b63867, #3028). W4 scope committed (see `## W4`, +issues #3030, #3031); arm implementation not started. From 9026467f1eda653760a570b7396d2813f15a916c Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Sun, 6 Sep 2026 21:48:10 +0200 Subject: [PATCH 02/10] record(BACKEND-TENSTORRENT-KEEPQUANT): W4a(1) is threshold-class (#2959) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wave-1's blanket residency flip is falsified on the 0.8B vehicle: the head is TIED, [248320,1024] Q6_K, 254,274,560 elements, and its per-call on-core decode re-materializes the 4,068,474,880 B ttnn::where transient the W3 gate memorized. The flip is correct for projection matmuls and wrong for logits-head-class weights, which decode every step. The amendment records the operator decision: weights above 128M elements keep the memoized bf16 twin (a decode transient costs ~16 B/elem peak against the twin's 2 B/elem permanent, and per-step unpack compute on a head is wasted); weights at or below the threshold consume the per-call shadow decode. On 27B the budget becomes ~16 GiB word shadows + 2.37 GiB embedding twin + 2.37 GiB output-head twin + activations on 32 GB, expert tower packed — it fits with headroom. On the 0.8B vehicle the gate keeps today's semantics. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/specs/tenstorrent-keepquant.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/.agents/specs/tenstorrent-keepquant.md b/.agents/specs/tenstorrent-keepquant.md index 0fea5f213..003a48b28 100644 --- a/.agents/specs/tenstorrent-keepquant.md +++ b/.agents/specs/tenstorrent-keepquant.md @@ -310,15 +310,24 @@ Two changes, two pull requests (developer decision 2026-09-06): **W4a — the 27B arm ([#3030](https://github.com/mudler/vllm.cpp/issues/3030)).** (1) Production residency flip: `MatmulBTQuantKernel` consumes the per-call on-core decode from the resident word shadow for {Q4_K,Q5_K,Q6_K,Q8_0}; the -twin survives for the embedding gather only. Red-first: the 0.8B vehicle -gate re-run under the flipped residency (the W3 gate stays green — the OOM -it memorized was twin construction). (2) `kMatmulBTQuantGrouped` on +twin survives where the operand is **threshold-class**: weights above 128M +elements keep the memoized bf16 twin (embedding gather, logits-head-class +weights), because a per-call decode transient costs ~16 B/elem peak against +the twin's 2 B/elem permanent, and a head runs every step so per-step unpack +compute is wasted. Wave-1 falsified the blanket flip on the 0.8B vehicle: +the head is TIED ([248320,1024] Q6_K, 254,274,560 elems) and its per-call +decode re-materializes the 4,068,474,880 B `ttnn::where`. Red-first: the +0.8B vehicle gate re-run under the flipped residency (the W3 gate stays +green — the OOM it memorized was twin construction, and under the threshold +it stays so for the head). (2) `kMatmulBTQuantGrouped` on kTENSTORRENT beside the W3 dense decode chains, consuming the existing stacked tower. (3) MTP `blk.64.*` skip/refuse by name. (4) 27B e2e greedy near-tie gate, checkpoint-gated opt-in loud-skip (#2811 precedent), goldens vs the pinned llama.cpp b10451 oracle, 500-mnat band + 0 forward-divergent. -(5) `docs/USAGE.md` pin in the same change. Memory axis recorded: ~16 GiB -word shadows + 2.37 GiB embedding twin + activations on 32 GB. +(5) `docs/USAGE.md` pin in the same change. Memory axis recorded on 27B: +~16 GiB word shadows (all four encodings; 210/34-byte blocks pad to 53/9 +words) + 2.37 GiB embedding twin + 2.37 GiB output-head twin (threshold- +class) + activations on 32 GB; the expert tower stays packed. **W4b — the int8-dot lever ([#3031](https://github.com/mudler/vllm.cpp/issues/3031)).** Quantized-domain integer vec_dot behind the same seam; profile-first @@ -342,4 +351,8 @@ W3 EVIDENCE COMPLETE on the row branch (see `## Evidence`): capture dump x2 byte-identity, staging counter 0, READY gate 16/16 PASS (11 strict / 5 near-tie, 0 forward-divergent), backend proof 0 declines. W3 LANDED 2026-09-06 (025c6ed90..f98b63867, #3028). W4 scope committed (see `## W4`, -issues #3030, #3031); arm implementation not started. +issues #3030, #3031). AMENDED 2026-09-06 (second): wave-1's blanket flip is +falsified on the 0.8B vehicle — the tied Q6_K head's per-call decode +re-materializes the 4 GB `ttnn::where` — so W4a(1) is threshold-class +(twin above 128M elements, shadow decode below); wave-1b implements the +threshold on top of wave-1's uncommitted red-first test, probes, and flip. From a9d6321ea6d3347dca52f75771bd99aaa7770ecb Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Sun, 6 Sep 2026 22:53:38 +0200 Subject: [PATCH 03/10] record(BACKEND-TENSTORRENT-KEEPQUANT): cancel the flip, re-anchor on the grouped kernel (#2959) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The threshold flip's capture leg is the second and decisive falsification. With the threshold in place the vehicle passed focused and suite legs, then the first decode-step capture died: TT_FATAL, trace buffers of 425,754,624 B against a 52,428,800 B trace region (mesh_trace.cpp:81). A weight decoded per call persists as a bf16 tile inside the captured graph, and trace replay re-runs the decode every step — a throughput regression against the twin with no compensating win. The twin-for-all diagnostic re-ran the same gate 16/16 PASS, 147/147 (11 strict / 5 near-tie, max 0.188 nats, 0 forward-divergent), isolating the cause to the flipped weights. No element-count threshold repairs the class, and the four candidate repairs each fail: enlarging the trace region needs ~18 GB on 27B; capture-mode twins leave the word shadow feeding only the eager arm; persistent decode scratch defeats the residency arithmetic; a no-capture fallback breaks the capture model. The flip waves are cancelled and the branch diff is reverted; production dense keep-quant matmuls keep the landed W3 twin. W4a(1) re-anchors on the TT provider for vt::MatmulBTQuantGrouped: the ROCm reference (rocm_grouped_gemm.hip:1456) already reads packed block weights natively for Q8_0/Q4_K/Q6_K, E=1 covers dense and E=N covers the stacked expert tower, and the seam's production consumers (qwen3_5.cpp:6147) reach a registered provider without new wiring. Q5_K is owed on top of the reference. Steady-state 27B budget: ~20.6 GB weights + activations on 32 GB. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/specs/tenstorrent-keepquant.md | 63 ++++++++++++++++---------- 1 file changed, 40 insertions(+), 23 deletions(-) diff --git a/.agents/specs/tenstorrent-keepquant.md b/.agents/specs/tenstorrent-keepquant.md index 003a48b28..6713b1616 100644 --- a/.agents/specs/tenstorrent-keepquant.md +++ b/.agents/specs/tenstorrent-keepquant.md @@ -308,26 +308,39 @@ to make a failure pass. Two changes, two pull requests (developer decision 2026-09-06): **W4a — the 27B arm ([#3030](https://github.com/mudler/vllm.cpp/issues/3030)).** -(1) Production residency flip: `MatmulBTQuantKernel` consumes the per-call -on-core decode from the resident word shadow for {Q4_K,Q5_K,Q6_K,Q8_0}; the -twin survives where the operand is **threshold-class**: weights above 128M -elements keep the memoized bf16 twin (embedding gather, logits-head-class -weights), because a per-call decode transient costs ~16 B/elem peak against -the twin's 2 B/elem permanent, and a head runs every step so per-step unpack -compute is wasted. Wave-1 falsified the blanket flip on the 0.8B vehicle: -the head is TIED ([248320,1024] Q6_K, 254,274,560 elems) and its per-call -decode re-materializes the 4,068,474,880 B `ttnn::where`. Red-first: the -0.8B vehicle gate re-run under the flipped residency (the W3 gate stays -green — the OOM it memorized was twin construction, and under the threshold -it stays so for the head). (2) `kMatmulBTQuantGrouped` on -kTENSTORRENT beside the W3 dense decode chains, consuming the existing -stacked tower. (3) MTP `blk.64.*` skip/refuse by name. (4) 27B e2e greedy +(1) RESIDENCY: the flip waves are CANCELLED 2026-09-06 — two falsifications. +Wave-1's blanket flip OOM'd the vehicle at the tied head's per-call decode +(4,068,474,880 B `ttnn::where`). Wave-1b's threshold flip (twin above 128M +elements, shadow decode below) passed focused + suite (59/59, 6704/6704) +but the vehicle capture demanded 425,754,624 B of trace region against the +52,428,800 B allocated (TT_FATAL, mesh_trace.cpp:81) — a weight decoded per +call persists as a bf16 tile inside the captured graph, and replay re-runs +the decode every step, a throughput regression with no compensating win. +The twin-for-all diagnostic re-ran the same gate 16/16 PASS 147/147 (11 +strict / 5 near-tie, max 0.188 nats, 0 forward-divergent), isolating the +cause to the flipped weights. Rejected repairs: trace-region enlargement +(~18 GB on 27B), capture-mode twins (words would feed only the eager arm — +dead in production), persistent decode scratch (defeats the arithmetic), +no-capture fallback (breaks the capture model). Production dense keep-quant +matmuls KEEP the memoized twin (the landed W3 behavior). Packed-word +residency pays off only where a kernel consumes the words natively: (2) the TT provider for `vt::MatmulBTQuantGrouped` +(ops.cpp:220, weight[E*N,K], expert_ids[P]) ports the ROCm reference's +native packed-word dequant (rocm_grouped_gemm.hip:1456, Q8_0/Q4_K/Q6_K — +extend to Q5_K with the W3 chain; the 27B pin carries 48 Q5_K tensors). +E=1 covers dense; E=N covers experts fed by the stacked tower +(qwen3_5_gguf_weights.cpp:1282). Twins remain for gather-class operands +(embedding; the vehicle's tied head shares the table). Capture +compatibility of the grouped arm is wave-3's committed obligation. (3) MTP `blk.64.*` skip/refuse by name. (4) 27B e2e greedy near-tie gate, checkpoint-gated opt-in loud-skip (#2811 precedent), goldens vs the pinned llama.cpp b10451 oracle, 500-mnat band + 0 forward-divergent. -(5) `docs/USAGE.md` pin in the same change. Memory axis recorded on 27B: -~16 GiB word shadows (all four encodings; 210/34-byte blocks pad to 53/9 -words) + 2.37 GiB embedding twin + 2.37 GiB output-head twin (threshold- -class) + activations on 32 GB; the expert tower stays packed. +(5) `docs/USAGE.md` pin in the same change. Memory axis on 27B at steady +state: 9.81 GiB experts packed + 6.01 GB non-expert packed (both through +the native grouped kernel) + 2.37 GiB embedding twin + 2.37 GiB output +twin ≈ 20.6 GB + activations on 32 GB. Interim (experts alone): non-expert +twins 18.47 GB push the total to ~33 GB, so the 27B gate (wave-3) is +reachable only behind the dense E=1 arm. The 0.8B vehicle carries no +experts: its gate guards the dense twin path and must stay green +throughout. **W4b — the int8-dot lever ([#3031](https://github.com/mudler/vllm.cpp/issues/3031)).** Quantized-domain integer vec_dot behind the same seam; profile-first @@ -351,8 +364,12 @@ W3 EVIDENCE COMPLETE on the row branch (see `## Evidence`): capture dump x2 byte-identity, staging counter 0, READY gate 16/16 PASS (11 strict / 5 near-tie, 0 forward-divergent), backend proof 0 declines. W3 LANDED 2026-09-06 (025c6ed90..f98b63867, #3028). W4 scope committed (see `## W4`, -issues #3030, #3031). AMENDED 2026-09-06 (second): wave-1's blanket flip is -falsified on the 0.8B vehicle — the tied Q6_K head's per-call decode -re-materializes the 4 GB `ttnn::where` — so W4a(1) is threshold-class -(twin above 128M elements, shadow decode below); wave-1b implements the -threshold on top of wave-1's uncommitted red-first test, probes, and flip. +issues #3030, #3031). AMENDED 2026-09-06 (third): the threshold flip's capture leg +falsified the mechanism itself (trace region 425,754,624 B vs 52,428,800 B, +mesh_trace.cpp:81; per-step decode compute at replay; the twin-for-all +diagnostic ran the same gate 147/147 green). The flip waves are cancelled; +the branch diff reverted; dense twins are the shipped behavior. W4a(1) +re-anchors on the TT `kMatmulBTQuantGrouped` provider (wave-2: packed +tower, native in-kernel dequant, E=1 dense + E=N experts, Q4_K/Q8_0 first, +Q5_K extension owed, staged slice owed to wave-3 wiring and the 27B gate). +The 128M threshold dissolves. From 5e43d770778de11d0730724aaff71c8387702a08 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Mon, 7 Sep 2026 00:54:04 +0200 Subject: [PATCH 04/10] feat(BACKEND-TENSTORRENT-KEEPQUANT): register the TT grouped keep-quant provider (#2959) The Tenstorrent backend now serves OpId::kMatmulBTQuantGrouped. The W4a re-anchor names this kernel as the row's production surface: the ROCm reference (rocm_grouped_gemm.hip:1456) reads packed block weights natively, E=1 covers dense, and E=N covers the stacked expert tower. The weight stays packed on device as staged i32 words. Per call the kernel reads the P expert ids from host, decodes only the selected [N,K] row slices through the W3 chain, rounds once to bf16, and runs one 2D tile matmul per group. Results concatenate into a single f32 commit. The registered encoding set is exactly Q4_K and Q8_0; Q5_K, Q6_K, and Q4_0 refuse with a message that names them. DecodeKeepQuantBlocksF32 is split into a slice form over staged words plus a whole-tensor wrapper, and the wrapper keeps the prior behavior. RED first: with no provider both new cases failed at the registration check, and the refusal text "no kernel for op MatmulBTQuantGrouped" was captured verbatim. GREEN: the routing case pins the admitted set and the refusals; the numerics case sweeps seven shapes, both arms, both encodings, and Pa=1 broadcast against the CPU grouped provider inside the W2-ratified analytic bf16 envelope. Worst error ratios are 0.37 to 0.72 of the bound, and the slice decode itself is bit-exact against the analytic dequant on Q8_0's bf16-exact scales. STAGED-UNREACHED: no production entry point reaches this provider yet. The vehicle's expert tower still routes through the W3 twin path, and the dense E=1 arm plus the 27B gate stay with wave-3 wiring tracked in issue #3030 and the spec's wave plan. Capture compatibility is also open: the kernel does EnsureHost on ids per call and builds eager-path constructs, so a captured decode graph cannot include it today; that repair is owed by the same wave. Evidence: focused cases 7277/7277; full backend suite 60 cases / 13,964 assertions (baseline 58 / 6,687 plus exactly these two cases); vehicle gate 16/16 PASS unchanged at 147/147, 11 strict / 5 near-tie, zero forward-divergent. The same change repairs the one record anchor this diff staled: the GDN row's registration citation in backend-matrix.md moves 7244-7260 to 7422-7436, its pre-edit line range. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/backend-matrix.md | 2 +- src/vt/tenstorrent/tenstorrent_ops.cpp | 194 ++++++++++++- tests/vt/test_tenstorrent_backend.cpp | 378 +++++++++++++++++++++++++ 3 files changed, 565 insertions(+), 9 deletions(-) diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index eb7befd2b..938d2555b 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -246,7 +246,7 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | | `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | **Gate PASSED on Blackhole P150 (2026-08-12):** [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap **0.062 nats**, BACKEND PROOF 0 declines (kMatmul selections=256 = untied lm_head on device, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded, same as Qwen3-0.6B TT precedent). Exit SIGSEGV 139 is the known MeshDevice teardown crash, not a gate failure | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | -| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7244-7260](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7244-L7260) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | +| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7422-7436](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7422-L7436) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | | `BACKEND-TENSTORRENT-GDN-DEVICE-PURE` | Child of `BACKEND-TENSTORRENT-GDN` — make the decode-side GDN ops (`CausalConv1dUpdateKernel`, `GdnDecodeKernel`) device-resident so a tt-metal trace capture admits them, unblocking the Qwen3.5-0.8B captured arm, the last capture-blocked family with a committed eager pair (#2907, owed from #2812) | Move the per-call host orchestration on-device: resident inputs from the graph's producer ops, device-side indexed state update (baked slot addressing re-primed by the recapture cadence, or `kGdnStateGather`/`kGdnStateScatter` indirection — decided on recapture-cost measurement), conv two-views coherent on device, token readback outside the captured span | [CausalConv1dUpdateKernel :5013](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5013), [GdnDecodeKernel :5225](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5225) (EnsureHost ×5, ReadIdxHost, UploadTensor ×5+), q35 harness pair selection [test_qwen35_paged_engine.cpp:233](../tests/parity/test_qwen35_paged_engine.cpp#L233), fatal repro `fd_mesh_command_queue.cpp:760` 2/2 | Red-first: the #2812 loud-skip opt-in cells run captured; captured dump ×2 byte-identity with reset between; teacher-forced pair vs the transformers oracle inside the eager band; Qwen3.5 joins `DecodeCaptureDefaultArch`, ambient adjudicates CAPTURED, env=0 eager; tamper + arch-deletion mutations red; gate arm selection [test_qwen35_paged_engine.cpp:301-330](../tests/parity/test_qwen35_paged_engine.cpp#L301-L330); closure [parity-ledger.md:947](parity-ledger.md#L947) | [tenstorrent-gdn-device-pure.md](specs/tenstorrent-gdn-device-pure.md) | `DONE` | `bcade48d6` | | `BACKEND-TENSTORRENT-KEEPQUANT` | Child of `BACKEND-TENSTORRENT` — keep-quant dense dot on TT-Metal ([#2959](https://github.com/mudler/vllm.cpp/issues/2959)): GGUF k-quant arms refuse on the P150 until this row lands — W1 (#2989, open) lands the Q4_K decode, W2 (branch) lands the dot and the `kTENSTORRENT` predicate arm (`gguf_keep_quant.cpp:136-148`) admitting exactly `{Q4_K}` — while the smallest Qwen3.8 artifact that fits the card is the dense 27B Q4_K_M (17.1 GB; bf16 is 53.8 GB) and a quantized Qwen3.5 (0.8B, ~0.5 GB) is the test vehicle | ggml k-quant block formats via our reader, pinned bit-exact vs llama.cpp `b10451` (#2240/#2260 lineage); no vLLM mirror for TT (secondary substrate, deviation by design); substrate surveyed 2026-09-05: tt-metal has NO packed-weight matmul (`ttnn` quantization is per-tensor activation dquant only), so the kernel is ours — resident blocks, on-core decode to bf16 tiles through f32 (decode bit-exact vs `BlockToFloat`), the dot reusing the `kMatmulBT` tile path, device-bf16 band at the residual-golden boundary, the int8 dot as the named next lever | W1 LANDED (#2989): the Q4_K block-decode device path. W2 ON BRANCH: the dot (`MatmulBTQuantKernel`, decode → one bf16 RNE → `kMatmulBT` tile matmul, reached through `vt::MatmulBT`'s block-weight dispatch) + the predicate arm admitting exactly the registered set (never wider — the GLM-5.3 W10 lesson). OWED W3: capture-safe residency + the e2e vehicle. OWED W4: Q5_K/Q6_K/Q8_0, the int8 lever, the 27B arm | W1 LANDED (#2989): decode bit-exact vs `vt::cpu::BlockToFloat` across a shape sweep (red-first, op-level suite pattern). W2 ON BRANCH: the dot vs the decode-based bf16 oracle inside the analytic operand-rounding envelope + the predicate routing set (both red-first). OWED W3: capture dump ×2 byte-identity (#2907 discipline); e2e 16-prompt battery on the P150 vs the bf16 arm's committed pair (STRICT or inside the ≤500-mnat near-tie band) | [tenstorrent-keepquant.md](specs/tenstorrent-keepquant.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-KEEPQUANT` — implementer on the row branch; spec commit on `row/BACKEND-TENSTORRENT-KEEPQUANT`, claim in [.agents/claims/](claims/CLAIM-BACKEND-TENSTORRENT-KEEPQUANT.md) | | `BACKEND-TENSTORRENT-QWEN35` | Child of `BACKEND-TENSTORRENT` — the wiring row: `Qwen3_5ForConditionalGeneration` (dense text GDN hybrid) on the TT allow-list, the op delta the family forward refuses by name (`kGdnPostConv`, `kSigmoidGateBf16`, `kAttnQkNormRopeGate`, `kAttnQkNormRope` — pinned empirically by a W0 refusal sweep), and the first e2e gate. Makes the GDN row's ops production-reached (#1715 stays open until the family runs; GDN row lifecycle moves in the same change this lands) | Substrate: the GDN row's 8 kernels + the 27-op TT registry; e2e mirrors the ratified Mistral TT golden treatment (`VT_DUMP_IDS` bootstrap → `scripts/qwen3-neartie-gap-transformers.py` teacher-forced near-tie gaps → committed device-golden pair, POL-ORACLE deviation); op-level oracle is the CPU f32 arm. Capacity: `Qwen/Qwen3.5-0.8B` bf16 ≈1.6 GB fits (proven envelope Mistral-7B ≈14.5 GB); 27B bf16 ≈53.8 GB and GGUF k-quant arms (no TT kernels) refused by name, owed | `src/vllm/platforms/tenstorrent.cpp` (allow-list, lands last) + `src/vt/tenstorrent/tenstorrent_ops.cpp` (op delta) + `tests/parity/test_qwen35_paged_engine.cpp` (TT arm) | Spec committed 2026-08-23; W0 sweep, W1 op delta, W2 e2e, W3 GDN-reviewer leftovers (d2h counter completeness, `conv_transposed` fast-path check) owed | [tenstorrent-qwen35.md](specs/tenstorrent-qwen35.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-QWEN35` | diff --git a/src/vt/tenstorrent/tenstorrent_ops.cpp b/src/vt/tenstorrent/tenstorrent_ops.cpp index d20cf4b35..45ed9249f 100644 --- a/src/vt/tenstorrent/tenstorrent_ops.cpp +++ b/src/vt/tenstorrent/tenstorrent_ops.cpp @@ -1945,11 +1945,18 @@ ttnn::Tensor Neg0CacheGet(const ttnn::Shape& shape, MeshDevice& device) { return it->second; } -ttnn::Tensor DecodeKeepQuantBlocksF32(const Tensor& packed, DType enc, - int64_t rows, int64_t nb, - MeshDevice& device) { - const uint32_t B = static_cast(rows * nb); - const ttnn::Tensor w = EnsureKeepQuantWords(packed, enc, rows, nb, device); +// Decode `slice_rows` packed rows (nb blocks each) ALREADY staged as the +// resident i32 word tensor w ([slice_rows*nb, wpb]) into the repaired f32 +// {slice_rows, nb*elems} in ROW_MAJOR — the W3 chains, one encoding each, run +// on a word RANGE instead of a whole packed tensor. DecodeKeepQuantBlocksF32 +// below is the whole-tensor form; the grouped keep-quant matmul +// (MatmulBTQuantGroupedKernel) slices the tower's words to the P selected +// [N,K] row-ranges per call and runs THIS. Same chain, same numerics, so the +// W1/W3 bit-exact pins carry over unchanged. +ttnn::Tensor DecodeKeepQuantWordsF32(const ttnn::Tensor& w, DType enc, + int64_t slice_rows, int64_t nb, + MeshDevice& device) { + const uint32_t B = static_cast(slice_rows * nb); // f16 bit pattern (held in INT32) -> f32 value, the integer chain of auto f16_bits_to_f32 = [](ttnn::Tensor t) { @@ -2181,7 +2188,7 @@ ttnn::Tensor DecodeKeepQuantBlocksF32(const Tensor& packed, DType enc, neg0_full(ttnn::Shape({B, 8u, 32u}))); return ttnn::reshape( std::move(y), - ttnn::Shape({static_cast(rows), + ttnn::Shape({static_cast(slice_rows), static_cast(nb) * 256u})); } case DType::kQ6_K: { @@ -2294,7 +2301,7 @@ ttnn::Tensor DecodeKeepQuantBlocksF32(const Tensor& packed, DType enc, neg0_full(ttnn::Shape({B, 16u, 16u}))); return ttnn::reshape( prod, - ttnn::Shape({static_cast(rows), + ttnn::Shape({static_cast(slice_rows), static_cast(nb) * 256u})); } case DType::kQ8_0: { @@ -2324,7 +2331,7 @@ ttnn::Tensor DecodeKeepQuantBlocksF32(const Tensor& packed, DType enc, neg0_full(ttnn::Shape({B, 32u}))); return ttnn::reshape( prod, - ttnn::Shape({static_cast(rows), + ttnn::Shape({static_cast(slice_rows), static_cast(nb) * 32u})); } default: @@ -2334,6 +2341,15 @@ ttnn::Tensor DecodeKeepQuantBlocksF32(const Tensor& packed, DType enc, } } +// The whole-tensor form: stage the packed [rows, nb] tensor once (EnsureKeepQuantWords) +// and run the chains on all of it — the W1/W3 keep-quant decode entry. +ttnn::Tensor DecodeKeepQuantBlocksF32(const Tensor& packed, DType enc, + int64_t rows, int64_t nb, + MeshDevice& device) { + const ttnn::Tensor w = EnsureKeepQuantWords(packed, enc, rows, nb, device); + return DecodeKeepQuantWordsF32(w, enc, rows, nb, device); +} + void KeepQuantDecodeKernel(Queue&, Tensor& out, const Tensor& packed) { TT_OP_TRACE("KeepQuantDecode"); VT_CHECK(packed.rank == 2 && out.rank == 2, @@ -2498,6 +2514,165 @@ void MatmulBTQuantKernel(Queue&, Tensor& out, const Tensor& a, const Tensor& b) CommitDevice2D(out, std::move(dev_c)); } +// kMatmulBTQuantGrouped (KEEPQUANT W4a wave-2, #3030): out[P,N], act[Pa,K] +// (Pa==1 broadcast), weight[E*N,K] PACKED block-quant, expert_ids[P] i32 — +// the expert-batched analog of MatmulBTQuantKernel above, mirroring the ROCm +// reference's native packed-weight grouped GEMM +// (rocm_grouped_gemm.hip MatmulBTQuantGroupedKernelRocm). The tower sits on +// device in its i32 WORD form (EnsureKeepQuantWords — the PACKED residency, +// never a bf16 twin), and each call decodes ONLY the P selected [N,K] +// row-slices (per group p: word rows [e*N*nb, (e+1)*N*nb)) through the +// bit-exact W3 chain on the slice (DecodeKeepQuantWordsF32), rounds the +// decoded weight ONCE to bf16 (the device's round-once convention), and runs +// the kMatmulBT tile matmul per group — the CPU provider's per-group +// structure (cpu_quant_gemm.cpp, the comparison oracle) with the decode +// on-core. The whole tower is NEVER decoded; no twin is built. +// +// REGISTERED SET: exactly {Q4_K, Q8_0} on kTENSTORRENT. Q5_K/Q6_K refuse BY +// NAME (the owed grouped extension — the 27B pin carries 48 Q5_K tensors; +// recorded under the spec's W4 plan); any other encoding refuses the same +// way. Never a silent wrong answer — the ROCm refusal precedent. +// +// NOT capture-safe yet, and staged UNREACHED: the per-call EnsureHost of the +// routing ids and the per-group decode writes are eager-path constructs; +// capture compatibility of the grouped arm is wave-3's committed obligation, +// and no production entry point reaches this kernel yet (the wiring row is +// W4a wave-3, dense E=1 + the 27B gate). +void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, + const Tensor& weight, + const Tensor& expert_ids) { + TT_OP_TRACE("MatmulBTQuantGrouped"); + VT_CHECK(act.rank == 2 && weight.rank == 2 && out.rank == 2, + "tenstorrent kMatmulBTQuantGrouped: rank-2 act/weight/out required"); + const DType enc = weight.dtype; + // vt::Name() emits the lowercase storage name ("q4_0"); the refusal must + // name the ENUM the caller passed, so the k-prefix and capital go on here + // (the MatmulBTQuantKernel convention). + const std::string enc_lower = Name(enc); + const std::string enc_name = + std::string("k") + static_cast(enc_lower[0] - 'a' + 'A') + + enc_lower.substr(1); + VT_CHECK(enc == DType::kQ4_K || enc == DType::kQ8_0, + std::string("tenstorrent kMatmulBTQuantGrouped: ") + enc_name + + " has no GROUPED keep-quant decode on TENSTORRENT; the " + "registered set is kQ4_K/kQ8_0 (BACKEND-TENSTORRENT-" + "KEEPQUANT W4a; the Q5_K/Q6_K grouped extension is owed)"); + const int64_t elems = BlockElems(enc); + VT_CHECK(weight.shape[1] % elems == 0, + std::string("tenstorrent kMatmulBTQuantGrouped: K must be a whole " + "number of ") + + Name(enc) + " blocks (" + std::to_string(elems) + " elems)"); + VT_CHECK(IsFloatDType(act.dtype) && + (out.dtype == DType::kF32 || out.dtype == DType::kBF16), + "tenstorrent kMatmulBTQuantGrouped: float activation, f32/bf16 out"); + VT_CHECK(act.IsContiguous() && weight.IsContiguous() && out.IsContiguous(), + "tenstorrent kMatmulBTQuantGrouped: strided tensors are not " + "supported in W4a wave-2"); + const int64_t P = out.shape[0]; + const int64_t N = out.shape[1]; + const int64_t K = act.shape[1]; + const int64_t Pa = act.shape[0]; + VT_CHECK(Pa == P || Pa == 1, + "tenstorrent kMatmulBTQuantGrouped: act rows must be P (per-expert) " + "or 1 (broadcast)"); + VT_CHECK(weight.shape[1] == K, + "tenstorrent kMatmulBTQuantGrouped: act/weight inner dim mismatch"); + VT_CHECK(weight.shape[0] % N == 0, + "tenstorrent kMatmulBTQuantGrouped: weight rows must be a whole " + "multiple of N"); + VT_CHECK(out.shape[0] == P && out.shape[1] == N, + "tenstorrent kMatmulBTQuantGrouped: out shape mismatch"); + const int64_t E = weight.shape[0] / N; + const int64_t nb = K / elems; + if (P == 0 || N == 0) return; + + MeshDevice& device = SharedMeshDevice(); + // The routing ids are small; the established TT index-tensor contract is + // EnsureHost + a host read (EmbeddingKernel), range-checked like the + // embedding gather. + EnsureHost(expert_ids); + const int32_t* eids = expert_ids.Ptr(); + for (int64_t p = 0; p < P; ++p) + VT_CHECK(eids[p] >= 0 && eids[p] < E, + "tenstorrent kMatmulBTQuantGrouped: expert id out of range (id " + + std::to_string(eids[p]) + ", E " + std::to_string(E) + ")"); + + // Stage the PACKED tower once — the resident i32 word shadow keyed by the + // host weight pointer, served forever after (the dense arm's pattern). The + // per-call decode reads word ROW-RANGES of it; a capture-time miss refuses + // inside EnsureKeepQuantWords, as on the dense arm. + const ttnn::Tensor words = EnsureKeepQuantWords(weight, enc, E * N, nb, device); + + // Activation: the MatmulBTQuantKernel convention — one device-side bf16 + // round of an f32 master, TILE layout, shared by every group. Pa > 1 takes + // the per-group row view from the ROW_MAJOR form (the proven slice domain); + // Pa == 1 IS the single row. + ttnn::Tensor dev_a = EnsureDevice2D(act, device); + if (act.dtype == DType::kF32) + dev_a = ttnn::to_layout( + ttnn::typecast(std::move(dev_a), ttnn::DataType::BFLOAT16), + ttnn::Layout::TILE); + ttnn::Tensor a_rows; + if (Pa > 1) + a_rows = ttnn::to_layout(std::move(dev_a), ttnn::Layout::ROW_MAJOR); + + const uint32_t wpb = static_cast(KeepQuantWordsPerBlock(enc)); + // The selected [N,K] slice for group p: word rows [e*N*nb, (e+1)*N*nb) — + // decode, one bf16 RNE, TILE — the dense dot's exact weight convention. + auto slice_decode = [&](int64_t p) { + const int64_t w0 = eids[p] * N * nb; + ttnn::Tensor sl = ttnn::slice( + words, ttsl::SmallVector{static_cast(w0), 0u}, + ttsl::SmallVector{static_cast(w0 + N * nb), wpb}, + ttsl::SmallVector{1u, 1u}); + ttnn::Tensor wf = DecodeKeepQuantWordsF32(sl, enc, N, nb, device); + return ttnn::to_layout( + ttnn::typecast(std::move(wf), ttnn::DataType::BFLOAT16), + ttnn::Layout::TILE); + }; + + std::vector outs; + outs.reserve(static_cast(P)); + for (int64_t p = 0; p < P; ++p) { + ttnn::Tensor a_p; + if (Pa == 1) { + a_p = dev_a; + } else { + a_p = ttnn::to_layout( + ttnn::slice(a_rows, + ttsl::SmallVector{static_cast(p), 0u}, + ttsl::SmallVector{static_cast(p + 1), + static_cast(K)}, + ttsl::SmallVector{1u, 1u}), + ttnn::Layout::TILE); + } + outs.push_back(ttnn::operations::matmul::matmul( + std::move(a_p), slice_decode(p), /*transpose_a=*/false, + /*transpose_b=*/true)); + } + // Assemble [P,N] and commit ONCE (the slot is per host pointer, so the + // commit must be a single whole-output store). The tile matmul output is + // bf16, committed as the dense arm commits it; the P > 1 assembly goes + // through ROW_MAJOR f32 (the chains' commit form) and lands out.dtype. + ttnn::Tensor assembled; + if (P == 1) { + assembled = std::move(outs[0]); + } else { + std::vector rows_f; + rows_f.reserve(outs.size()); + for (auto& t : outs) + rows_f.push_back(ttnn::to_layout( + ttnn::typecast(std::move(t), ttnn::DataType::FLOAT32), + ttnn::Layout::ROW_MAJOR)); + assembled = ttnn::concat(std::move(rows_f), /*dim=*/0); + if (out.dtype == DType::kBF16) + assembled = + ttnn::typecast(std::move(assembled), ttnn::DataType::BFLOAT16); + } + CommitDeviceLogical2D(out, std::move(assembled), static_cast(P), + static_cast(N)); +} + // Upload a rank-1 affine vector as TILE BFLOAT16 [1, d], caching on the weight's // host buffer slot so RmsNorm/LayerNorm do not re-upload every layer call. // ttnn's TILE-gamma path requires padded height == tile_height (32); from_vector @@ -7213,6 +7388,9 @@ struct Registrar { reinterpret_cast(static_cast(&KeepQuantDecodeKernel))); RegisterOp(OpId::kMatmulBTQuant, DeviceType::kTENSTORRENT, reinterpret_cast(static_cast(&MatmulBTQuantKernel))); + RegisterOp(OpId::kMatmulBTQuantGrouped, DeviceType::kTENSTORRENT, + reinterpret_cast(static_cast( + &MatmulBTQuantGroupedKernel))); RegisterOp(OpId::kLayerNorm, DeviceType::kTENSTORRENT, reinterpret_cast(static_cast(&LayerNormKernel))); RegisterOp(OpId::kRmsNorm, DeviceType::kTENSTORRENT, diff --git a/tests/vt/test_tenstorrent_backend.cpp b/tests/vt/test_tenstorrent_backend.cpp index ce7999f3c..cbe56921b 100644 --- a/tests/vt/test_tenstorrent_backend.cpp +++ b/tests/vt/test_tenstorrent_backend.cpp @@ -5754,3 +5754,381 @@ TEST_CASE("kTENSTORRENT keep-quant decode stages zero words during capture") { backend.Free(mem_out); } } + +// KEEPQUANT W4a wave-2 (#3030): the GROUPED keep-quant GEMM on the P150 — +// vt::MatmulBTQuantGrouped (ops.cpp:220) with the packed [E*N,K] tower resident +// in its i32 word form, the P selected [N,K] row-slices decoded on-core per +// call (the W3 chains on a word slice), and the kMatmulBT tile matmul per +// group. No bf16 twin, never a full-tower decode. This case is the ROUTING +// leg, red-first: before the provider was registered the first REQUIRE reded, +// and the seam refused through GetOp's no-provider path ("no kernel for op +// MatmulBTQuantGrouped ... on device TENSTORRENT") — never a silent wrong +// answer. The registered encoding set is EXACTLY {Q4_K, Q8_0}: Q5_K and Q6_K +// must refuse BY NAME (the owed grouped extension, recorded in the spec's W4 +// plan), so a wrongly-widened kernel reds the refusal legs below. +TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped registers {Q4_K,Q8_0} and refuses the rest") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + REQUIRE(vt::OpRegistered(vt::OpId::kMatmulBTQuantGrouped, + vt::DeviceType::kTENSTORRENT)); + + Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); + Queue q = backend.CreateQueue(); + + // One Q4_K block per row: K = 256. The block generator is the W3 one: + // d/dmin are finite f16 (random f16 bits go NaN and the device's + // NaN-payload canonicalization would mask a real defect as a decode + // failure), payload bytes are random. + auto fill_block = [](uint8_t* blk, vt::DType enc, std::mt19937& rng) { + auto put_f16 = [&](int64_t off, float v) { + const uint16_t bits = vt::F32ToF16(v); + std::memcpy(blk + off, &bits, sizeof(bits)); + }; + if (enc == vt::DType::kQ8_0) { + put_f16(0, 0.1f + 0.2f * static_cast(rng() % 16) / 16.0f); + for (int i = 2; i < 34; ++i) blk[i] = static_cast(rng() & 0xFF); + } else { // Q4_K + put_f16(0, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); + put_f16(2, 0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); + for (int i = 4; i < 144; ++i) blk[i] = static_cast(rng() & 0xFF); + } + }; + + // THE ADMITTED ARMS: one call each, E=2 tower, both encodings. Only + // reachability is pinned here — the numerics have their own sweep below. + for (const vt::DType enc : {vt::DType::kQ4_K, vt::DType::kQ8_0}) { + const int64_t kBlockBytes = vt::BlockBytes(enc); + const int64_t kBlockElems = vt::BlockElems(enc); + constexpr int64_t kE = 2, kN = 8, kP = 3; + const int64_t k = 1 * kBlockElems; + std::mt19937 rng(20260910u); + std::vector packed(kE * kN * (k / kBlockElems) * kBlockBytes); + for (size_t b = 0; b < packed.size() / kBlockBytes; ++b) + fill_block(packed.data() + b * kBlockBytes, enc, rng); + std::vector a_bf(kP * k); + for (auto& v : a_bf) v = vt::F32ToBF16((static_cast(rng() % 401) - 200.0f) / 100.0f); + std::vector ids(kP); + for (auto& e : ids) e = static_cast(rng() % kE); + + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(kP * kN * sizeof(float)); + void* mem_i = backend.Alloc(ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w, packed.data(), packed.size()); + backend.Copy(q, mem_i, ids.data(), ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, k}); + Tensor w_t = Tensor::Contiguous(mem_w, enc, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kE * kN, k}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, kN}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + bool threw = false; + std::string what; + try { + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + } catch (const std::exception& e) { + threw = true; + what = e.what(); + } + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); + const std::string enc_ok = vt::Name(enc); + CHECK_MESSAGE(!threw, "the registered encoding ", enc_ok, + " must answer on the TENSTORRENT grouped arm, threw: ", what); + } + + // THE REFUSE SIDE: Q5_K / Q6_K (the owed grouped extension) and kQ4_0 (no + // TT arm anywhere) must throw naming THEMSELVES and the registered set — + // never fall through to a misread, never silently widen. + for (const vt::DType enc : {vt::DType::kQ5_K, vt::DType::kQ6_K, + vt::DType::kQ4_0}) { + const int64_t kBlockBytes = vt::BlockBytes(enc); + const int64_t kBlockElems = vt::BlockElems(enc); + constexpr int64_t kE = 2, kN = 8, kP = 3; + const int64_t k = 1 * kBlockElems; + std::vector packed(kE * kN * (k / kBlockElems) * kBlockBytes, 0u); + std::vector a_bf(kP * k, 0u); + std::vector ids(kP, 0); + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(kP * kN * sizeof(float)); + void* mem_i = backend.Alloc(ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, k}); + Tensor w_t = Tensor::Contiguous(mem_w, enc, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kE * kN, k}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, kN}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + bool threw = false; + std::string what; + try { + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + } catch (const std::exception& e) { + threw = true; + what = e.what(); + } + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); + CHECK_MESSAGE(threw, "unregistered encoding ", static_cast(enc), + " must refuse on the TENSTORRENT grouped arm, not misread"); + const std::string enc_lower = vt::Name(enc); + const std::string enc_name = std::string("k") + + static_cast(enc_lower[0] - 'a' + 'A') + enc_lower.substr(1); + CHECK_MESSAGE(what.find(enc_name) != std::string::npos, + "the refusal must name the encoding (", enc_name, + "), got: ", what); + CHECK_MESSAGE(what.find("kQ4_K/kQ8_0") != std::string::npos, + "the refusal must name the registered set, got: ", what); + } +} + +// THE NUMERICS LEG (W4a wave-2, red-first): the grouped op against the CPU +// grouped provider (cpu_quant_gemm.cpp MatmulBTQuantGroupedKernel — the SAME +// kMatmulBTQuant integer-dot core once per group) across shapes sweeping P, N, +// K, E: the E=1 dense arm (ids all zero), the E=N expert tower arm, P=1, a +// broadcast activation ([1,K]), a non-tile-multiple N, both registered +// encodings, and a 27B-mirroring slice (K = the Qwen3.8-27B hidden_size, +// N a production-like per-group intermediate). The bar is the W2-ratified +// analytic operand-rounding envelope — bf16-round-once both operands, f32 +// ascending-k accumulate, bound = 1.05 * 2^-8 * (mag + |acc|) — unchanged +// from the dense W2/W3 pin; the reported bound ratios are the wave evidence. +// Decode bit-exactness on the SLICE path is pinned separately below: a Q8_0 +// tower with power-of-two scales and one-hot activations makes every dot term +// exact, so the grouped output must be BIT-EQUAL to the dequantized weight +// element the slice selection picked. +TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped matches the CPU grouped provider inside the ratified envelope") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + REQUIRE(vt::OpRegistered(vt::OpId::kMatmulBTQuantGrouped, + vt::DeviceType::kTENSTORRENT)); + + Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); + Queue q = backend.CreateQueue(); + vt::Queue qcpu{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + auto widen = [](uint16_t u) { + uint32_t bits = static_cast(u) << 16; + float f; + std::memcpy(&f, &bits, 4); + return f; + }; + auto fill_block = [](uint8_t* blk, vt::DType enc, std::mt19937& rng) { + auto put_f16 = [&](int64_t off, float v) { + const uint16_t bits = vt::F32ToF16(v); + std::memcpy(blk + off, &bits, sizeof(bits)); + }; + if (enc == vt::DType::kQ8_0) { + put_f16(0, 0.1f + 0.2f * static_cast(rng() % 16) / 16.0f); + for (int i = 2; i < 34; ++i) blk[i] = static_cast(rng() & 0xFF); + } else { // Q4_K + put_f16(0, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); + put_f16(2, 0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); + for (int i = 4; i < 144; ++i) blk[i] = static_cast(rng() & 0xFF); + } + }; + + struct Shape { + int64_t p, n, k_blocks, e, act_rows; + vt::DType enc; + const char* note; + }; + const Shape shapes[] = { + {3, 8, 1, 1, -1, vt::DType::kQ4_K, "E=1 dense arm, ids all zero"}, + {4, 8, 2, 4, -1, vt::DType::kQ4_K, "E=N expert tower arm, permuted ids"}, + {1, 16, 1, 2, -1, vt::DType::kQ4_K, "P=1"}, + {4, 33, 1, 4, 1, vt::DType::kQ4_K, "broadcast [1,K] act, non-tile N=33"}, + {3, 8, 2, 2, -1, vt::DType::kQ8_0, "Q8_0 tower arm"}, + {4, 33, 1, 4, -1, vt::DType::kQ8_0, "Q8_0, non-tile N=33"}, + {2, 1024, 20, 1, -1, vt::DType::kQ4_K, + "27B mirror: K=5120 (Qwen3.8-27B hidden_size), N=1024 per-group slice"}, + }; + for (const Shape& s : shapes) { + const int64_t kBlockBytes = vt::BlockBytes(s.enc); + const int64_t kBlockElems = vt::BlockElems(s.enc); + const int64_t P = s.p, N = s.n, K = s.k_blocks * kBlockElems, E = s.e; + const int64_t Pa = s.act_rows == -1 ? P : s.act_rows; + const int64_t nb = K / kBlockElems; + std::mt19937 rng(static_cast(20260911u + P * 7 + N * 13 + E * 3)); + + std::vector packed(E * N * nb * kBlockBytes); + for (size_t b = 0; b < packed.size() / kBlockBytes; ++b) + fill_block(packed.data() + b * kBlockBytes, s.enc, rng); + std::vector a_f32(Pa * K); + for (auto& v : a_f32) v = (static_cast(rng() % 401) - 200.0f) / 100.0f; + std::vector a_bf(a_f32.size()); + for (size_t i = 0; i < a_f32.size(); ++i) a_bf[i] = vt::F32ToBF16(a_f32[i]); + std::vector ids(P); + for (auto& e : ids) e = static_cast(rng() % E); + if (E == 1) std::fill(ids.begin(), ids.end(), 0); + + // ---- the CPU grouped provider on the IDENTICAL bytes ---- + std::vector cpu_out(P * N, 0.0f); + { + Tensor at = Tensor::Contiguous(a_bf.data(), vt::DType::kBF16, qcpu.device, {Pa, K}); + Tensor ot = Tensor::Contiguous(cpu_out.data(), vt::DType::kF32, qcpu.device, {P, N}); + Tensor it = Tensor::Contiguous(ids.data(), vt::DType::kI32, qcpu.device, {P}); + Tensor wt = Tensor::Contiguous(packed.data(), vt::DType::kF32, qcpu.device, {E * N, K}); + wt.dtype = s.enc; // block dtype: elementwise strides are inert + vt::MatmulBTQuantGrouped(qcpu, ot, at, wt, it); + } + + // ---- the analytic bf16-operand reference + the W2 bound ---- + std::vector w_f32(E * N * K); + vt::cpu::BlockToFloat(s.enc)(packed.data(), w_f32.data(), E * N * K); + std::vector w_bf(w_f32.size()); + for (size_t i = 0; i < w_f32.size(); ++i) w_bf[i] = vt::F32ToBF16(w_f32[i]); + std::vector ref(P * N), bound(P * N); + for (int64_t p = 0; p < P; ++p) { + const int64_t e = ids[p]; + const int64_t pa = Pa == 1 ? 0 : p; + for (int64_t n = 0; n < N; ++n) { + float acc = 0.0f, mag = 0.0f; + for (int64_t k = 0; k < K; ++k) { + const float prod = widen(a_bf[pa * K + k]) * + widen(w_bf[(e * N + n) * K + k]); + acc += prod; + mag += std::fabs(prod); + } + ref[p * N + n] = acc; + bound[p * N + n] = 1.05f * std::ldexp(1.0f, -8) * (mag + std::fabs(acc)); + } + } + + // ---- the device grouped call ---- + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(P * N * sizeof(float)); + void* mem_i = backend.Alloc(ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w, packed.data(), packed.size()); + backend.Copy(q, mem_i, ids.data(), ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {Pa, K}); + Tensor w_t = Tensor::Contiguous(mem_w, s.enc, + Device{vt::DeviceType::kTENSTORRENT, 0}, {E * N, K}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {P, N}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {P}); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + std::vector tt_out(P * N, 0.0f); + backend.Copy(q, tt_out.data(), mem_o, tt_out.size() * sizeof(float)); + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); + + float worst_cpu = 0.0f, worst_ref = 0.0f; + for (int64_t i = 0; i < P * N; ++i) { + const float d_cpu = std::fabs(tt_out[i] - cpu_out[i]); + const float d_ref = std::fabs(tt_out[i] - ref[i]); + worst_cpu = std::max(worst_cpu, d_cpu / bound[i]); + worst_ref = std::max(worst_ref, d_ref / bound[i]); + CHECK(std::isfinite(tt_out[i])); + CHECK_MESSAGE(d_cpu <= bound[i], + "P=" << P << " N=" << N << " K=" << K << " E=" << E + << " enc=" << static_cast(s.enc) << " i=" << i + << " tt=" << tt_out[i] << " cpu=" << cpu_out[i] + << " bound=" << bound[i]); + CHECK_MESSAGE(d_ref <= bound[i], + "P=" << P << " N=" << N << " K=" << K << " E=" << E + << " enc=" << static_cast(s.enc) << " i=" << i + << " tt=" << tt_out[i] << " ref=" << ref[i] + << " bound=" << bound[i]); + } + MESSAGE("grouped P=", P, " N=", N, " K=", K, " E=", E, " Pa=", Pa, + " enc=", static_cast(s.enc), " (", s.note, + "): worst bound-ratio vs cpu=", worst_cpu, + " vs analytic-bf16-ref=", worst_ref); + } + + // ---- THE SLICE-DECODE BIT-EXACT LEG ---- + // Q8_0, d = +2^-6, qs = full int8: every dequantized element qs*2^-6 is + // exactly representable in bf16, so the on-core decode -> one bf16 RNE -> + // tile matmul (bf16 output) chain is LOSSLESS for one-hot activations, and + // the grouped output must be BIT-EQUAL to the dequant of the row-slice the + // routing picked — any slice-selection defect (wrong expert row-range, + // whole-tower misindex, bf16-twin aliasing) lands on a DIFFERENT grid value + // and cannot hide. The CPU grouped provider is NOT the bit oracle here: its + // dot is the INTEGER-dot core (the activation is quantized to q8 on the CPU + // side), so TT-vs-CPU agreement is the envelope's job — the sweep above. + { + constexpr int64_t kE = 3, kN = 8, kP = 5, kNb = 2; + const int64_t K = kNb * 32; + const uint16_t d_bits = vt::F32ToF16(std::ldexp(1.0f, -6)); + std::mt19937 rng(20260912u); + std::vector packed(kE * kN * kNb * 34); + std::vector qs(kE * kN * kNb * 32); + for (int64_t b = 0; b < kE * kN * kNb; ++b) { + uint8_t* blk = packed.data() + b * 34; + std::memcpy(blk, &d_bits, sizeof(d_bits)); + for (int i = 0; i < 32; ++i) { + const int8_t q = static_cast(rng() % 255 - 127); // [-127,127] + blk[2 + i] = static_cast(q); + qs[b * 32 + i] = q; + } + } + // One-hot activations: row p selects column (p % K); the group may repeat + // (two rows route to the same expert) and the ids are NOT ascending, so + // every selected slice is exercised independently. + std::vector a_bf(kP * K, 0u); + for (int64_t p = 0; p < kP; ++p) a_bf[p * K + (p % K)] = vt::F32ToBF16(1.0f); + std::vector ids = {2, 0, 2, 1, 0}; + + // The true dequant of the selected slice: out[p,n] = w_dec[ids[p], n, + // p%K]. Block b's qs live at qs[b*32 + i]; bf16(w_dec) == w_dec exactly, + // and the device commits the bf16 matmul output widened to f32, so the + // comparison is memcmp over the widened bits. + std::vector dequant(kP * kN); + for (int64_t p = 0; p < kP; ++p) + for (int64_t n = 0; n < kN; ++n) { + const int64_t c = p % K; + const int64_t b = (ids[p] * kN + n) * kNb + c / 32; + dequant[p * kN + n] = + static_cast(qs[b * 32 + c % 32]) * std::ldexp(1.0f, -6); + } + + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(kP * kN * sizeof(float)); + void* mem_i = backend.Alloc(ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w, packed.data(), packed.size()); + backend.Copy(q, mem_i, ids.data(), ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, K}); + Tensor w_t = Tensor::Contiguous(mem_w, vt::DType::kQ8_0, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kE * kN, K}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, kN}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + std::vector tt_out(kP * kN, 0.0f); + backend.Copy(q, tt_out.data(), mem_o, tt_out.size() * sizeof(float)); + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); + for (int64_t i = 0; i < kP * kN; ++i) { + const uint16_t got_bf = vt::F32ToBF16(tt_out[i]); + const uint16_t want_bf = vt::F32ToBF16(dequant[i]); + CHECK_MESSAGE(got_bf == want_bf, + "slice-decode bit-exact: p=" << i / kN << " n=" << i % kN + << " tt=" << tt_out[i] + << " dequant=" << dequant[i]); + } + } +} From 14e8fe471052ed32e7877496fe724e29297088ed Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Mon, 7 Sep 2026 02:14:46 +0200 Subject: [PATCH 05/10] feat(BACKEND-TENSTORRENT-KEEPQUANT): register Q5_K/Q6_K on the TT grouped keep-quant provider (#2959) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The kMatmulBTQuantGrouped registered set widens from {Q4_K, Q8_0} to exactly {Q4_K, Q5_K, Q6_K, Q8_0}. The 27B pin (issue #3030) carries 67 Q6_K and 48 Q5_K tensors, so the stacked expert tower is not fully servable without them. Both encodings ride the wave-2a path unchanged: the tower stages as i32 words once, each call decodes only the P selected [N,K] row slices through the W3 chains (DecodeKeepQuantWordsF32), rounds once to bf16, and runs one tile matmul per group. No new decode arithmetic ships in this change. Q5_K has no ROCm grouped reference to mirror (rocm_grouped_gemm.hip admits Q8_0/Q4_K/Q6_K natively), so its decode derives from the W3 dense Q5_K chain, bit-exact vs vt::cpu::BlockToFloat; Q6_K mirrors the ROCm grouped kernel's native dequant through the same W3 chain. RED first: the routing REQUIRE demanded the four-encoding set and failed on the then-current admission (both new encodings refused by name), and the numerics sweep threw at its first Q5_K shape with the refusal text captured verbatim. GREEN: the routing case pins the four-encoding set plus the Q4_0 refusal; the numerics sweep grows from seven to fifteen shapes (per new encoding: E=1 dense, E=N tower, Pa=1 broadcast with non-tile N, and a 27B mirror at K=5120) against the CPU grouped provider inside the ratified analytic bf16 envelope. Worst bound ratios for the new arms: Q5_K 0.20-0.51, Q6_K 0.21-0.47. New slice-decode bit-exact legs compare bf16 bits against BlockToFloat at the routed element; the Q6_K leg constrains scales to strictly positive because a negative scale meeting a zero q makes the true dequant -0, which any dot flattens in the f32 accumulate (the decode itself keeps -0, per the W3 pin). STAGED-UNREACHED: unchanged from wave-2a — no production entry point reaches this provider yet; wave-3 owns the wiring and the 27B gate (issue #3030), and the vehicle's expert tower still routes through the W3 twin path. Evidence: focused grouped cases 2/2, 20,769 assertions; full backend suite 60/60 cases, 27,456 assertions (wave-2a baseline 60 / 13,964 plus exactly these arms); clean full-tree build, 0 warnings; vehicle gate 16/16 PASS unchanged at 147/147, 11 strict / 5 near-tie, max gap 0.188 nats, 0 forward-divergent. The same change repairs the one record anchor this diff staled: the GDN row's registration citation in backend-matrix.md moves 7422-7436 to 7428-7442; the record-anchor ratchet holds at its baseline 33. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/backend-matrix.md | 2 +- src/vt/tenstorrent/tenstorrent_ops.cpp | 20 ++-- tests/vt/test_tenstorrent_backend.cpp | 153 +++++++++++++++++++++++-- 3 files changed, 155 insertions(+), 20 deletions(-) diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 938d2555b..db1e820a9 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -246,7 +246,7 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | | `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | **Gate PASSED on Blackhole P150 (2026-08-12):** [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap **0.062 nats**, BACKEND PROOF 0 declines (kMatmul selections=256 = untied lm_head on device, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded, same as Qwen3-0.6B TT precedent). Exit SIGSEGV 139 is the known MeshDevice teardown crash, not a gate failure | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | -| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7422-7436](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7422-L7436) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | +| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7428-7442](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7428-L7442) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | | `BACKEND-TENSTORRENT-GDN-DEVICE-PURE` | Child of `BACKEND-TENSTORRENT-GDN` — make the decode-side GDN ops (`CausalConv1dUpdateKernel`, `GdnDecodeKernel`) device-resident so a tt-metal trace capture admits them, unblocking the Qwen3.5-0.8B captured arm, the last capture-blocked family with a committed eager pair (#2907, owed from #2812) | Move the per-call host orchestration on-device: resident inputs from the graph's producer ops, device-side indexed state update (baked slot addressing re-primed by the recapture cadence, or `kGdnStateGather`/`kGdnStateScatter` indirection — decided on recapture-cost measurement), conv two-views coherent on device, token readback outside the captured span | [CausalConv1dUpdateKernel :5013](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5013), [GdnDecodeKernel :5225](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5225) (EnsureHost ×5, ReadIdxHost, UploadTensor ×5+), q35 harness pair selection [test_qwen35_paged_engine.cpp:233](../tests/parity/test_qwen35_paged_engine.cpp#L233), fatal repro `fd_mesh_command_queue.cpp:760` 2/2 | Red-first: the #2812 loud-skip opt-in cells run captured; captured dump ×2 byte-identity with reset between; teacher-forced pair vs the transformers oracle inside the eager band; Qwen3.5 joins `DecodeCaptureDefaultArch`, ambient adjudicates CAPTURED, env=0 eager; tamper + arch-deletion mutations red; gate arm selection [test_qwen35_paged_engine.cpp:301-330](../tests/parity/test_qwen35_paged_engine.cpp#L301-L330); closure [parity-ledger.md:947](parity-ledger.md#L947) | [tenstorrent-gdn-device-pure.md](specs/tenstorrent-gdn-device-pure.md) | `DONE` | `bcade48d6` | | `BACKEND-TENSTORRENT-KEEPQUANT` | Child of `BACKEND-TENSTORRENT` — keep-quant dense dot on TT-Metal ([#2959](https://github.com/mudler/vllm.cpp/issues/2959)): GGUF k-quant arms refuse on the P150 until this row lands — W1 (#2989, open) lands the Q4_K decode, W2 (branch) lands the dot and the `kTENSTORRENT` predicate arm (`gguf_keep_quant.cpp:136-148`) admitting exactly `{Q4_K}` — while the smallest Qwen3.8 artifact that fits the card is the dense 27B Q4_K_M (17.1 GB; bf16 is 53.8 GB) and a quantized Qwen3.5 (0.8B, ~0.5 GB) is the test vehicle | ggml k-quant block formats via our reader, pinned bit-exact vs llama.cpp `b10451` (#2240/#2260 lineage); no vLLM mirror for TT (secondary substrate, deviation by design); substrate surveyed 2026-09-05: tt-metal has NO packed-weight matmul (`ttnn` quantization is per-tensor activation dquant only), so the kernel is ours — resident blocks, on-core decode to bf16 tiles through f32 (decode bit-exact vs `BlockToFloat`), the dot reusing the `kMatmulBT` tile path, device-bf16 band at the residual-golden boundary, the int8 dot as the named next lever | W1 LANDED (#2989): the Q4_K block-decode device path. W2 ON BRANCH: the dot (`MatmulBTQuantKernel`, decode → one bf16 RNE → `kMatmulBT` tile matmul, reached through `vt::MatmulBT`'s block-weight dispatch) + the predicate arm admitting exactly the registered set (never wider — the GLM-5.3 W10 lesson). OWED W3: capture-safe residency + the e2e vehicle. OWED W4: Q5_K/Q6_K/Q8_0, the int8 lever, the 27B arm | W1 LANDED (#2989): decode bit-exact vs `vt::cpu::BlockToFloat` across a shape sweep (red-first, op-level suite pattern). W2 ON BRANCH: the dot vs the decode-based bf16 oracle inside the analytic operand-rounding envelope + the predicate routing set (both red-first). OWED W3: capture dump ×2 byte-identity (#2907 discipline); e2e 16-prompt battery on the P150 vs the bf16 arm's committed pair (STRICT or inside the ≤500-mnat near-tie band) | [tenstorrent-keepquant.md](specs/tenstorrent-keepquant.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-KEEPQUANT` — implementer on the row branch; spec commit on `row/BACKEND-TENSTORRENT-KEEPQUANT`, claim in [.agents/claims/](claims/CLAIM-BACKEND-TENSTORRENT-KEEPQUANT.md) | | `BACKEND-TENSTORRENT-QWEN35` | Child of `BACKEND-TENSTORRENT` — the wiring row: `Qwen3_5ForConditionalGeneration` (dense text GDN hybrid) on the TT allow-list, the op delta the family forward refuses by name (`kGdnPostConv`, `kSigmoidGateBf16`, `kAttnQkNormRopeGate`, `kAttnQkNormRope` — pinned empirically by a W0 refusal sweep), and the first e2e gate. Makes the GDN row's ops production-reached (#1715 stays open until the family runs; GDN row lifecycle moves in the same change this lands) | Substrate: the GDN row's 8 kernels + the 27-op TT registry; e2e mirrors the ratified Mistral TT golden treatment (`VT_DUMP_IDS` bootstrap → `scripts/qwen3-neartie-gap-transformers.py` teacher-forced near-tie gaps → committed device-golden pair, POL-ORACLE deviation); op-level oracle is the CPU f32 arm. Capacity: `Qwen/Qwen3.5-0.8B` bf16 ≈1.6 GB fits (proven envelope Mistral-7B ≈14.5 GB); 27B bf16 ≈53.8 GB and GGUF k-quant arms (no TT kernels) refused by name, owed | `src/vllm/platforms/tenstorrent.cpp` (allow-list, lands last) + `src/vt/tenstorrent/tenstorrent_ops.cpp` (op delta) + `tests/parity/test_qwen35_paged_engine.cpp` (TT arm) | Spec committed 2026-08-23; W0 sweep, W1 op delta, W2 e2e, W3 GDN-reviewer leftovers (d2h counter completeness, `conv_transposed` fast-path check) owed | [tenstorrent-qwen35.md](specs/tenstorrent-qwen35.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-QWEN35` | diff --git a/src/vt/tenstorrent/tenstorrent_ops.cpp b/src/vt/tenstorrent/tenstorrent_ops.cpp index 45ed9249f..552468d77 100644 --- a/src/vt/tenstorrent/tenstorrent_ops.cpp +++ b/src/vt/tenstorrent/tenstorrent_ops.cpp @@ -2528,10 +2528,15 @@ void MatmulBTQuantKernel(Queue&, Tensor& out, const Tensor& a, const Tensor& b) // structure (cpu_quant_gemm.cpp, the comparison oracle) with the decode // on-core. The whole tower is NEVER decoded; no twin is built. // -// REGISTERED SET: exactly {Q4_K, Q8_0} on kTENSTORRENT. Q5_K/Q6_K refuse BY -// NAME (the owed grouped extension — the 27B pin carries 48 Q5_K tensors; -// recorded under the spec's W4 plan); any other encoding refuses the same -// way. Never a silent wrong answer — the ROCm refusal precedent. +// REGISTERED SET: exactly {Q4_K, Q5_K, Q6_K, Q8_0} on kTENSTORRENT (W4a +// wave-2b widened the wave-2 pair: the 27B pin carries 67 Q6_K + 48 Q5_K +// tensors, so the tower is not servable without them). The two K-quant +// decodes are the W3 dense chains verbatim, run on the selected word slice — +// Q6_K mirrors the ROCm grouped kernel's native dequant +// (rocm_grouped_gemm.hip:1456), and Q5_K has NO ROCm grouped reference (the +// ROCm set is Q8_0/Q4_K/Q6_K), so its decode derives from the W3 dense Q5_K +// chain, bit-exact vs vt::cpu::BlockToFloat. Any other encoding refuses BY +// NAME. Never a silent wrong answer — the ROCm refusal precedent. // // NOT capture-safe yet, and staged UNREACHED: the per-call EnsureHost of the // routing ids and the per-group decode writes are eager-path constructs; @@ -2552,11 +2557,12 @@ void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, const std::string enc_name = std::string("k") + static_cast(enc_lower[0] - 'a' + 'A') + enc_lower.substr(1); - VT_CHECK(enc == DType::kQ4_K || enc == DType::kQ8_0, + VT_CHECK(enc == DType::kQ4_K || enc == DType::kQ5_K || + enc == DType::kQ6_K || enc == DType::kQ8_0, std::string("tenstorrent kMatmulBTQuantGrouped: ") + enc_name + " has no GROUPED keep-quant decode on TENSTORRENT; the " - "registered set is kQ4_K/kQ8_0 (BACKEND-TENSTORRENT-" - "KEEPQUANT W4a; the Q5_K/Q6_K grouped extension is owed)"); + "registered set is kQ4_K/kQ5_K/kQ6_K/kQ8_0 (BACKEND-" + "TENSTORRENT-KEEPQUANT W4a)"); const int64_t elems = BlockElems(enc); VT_CHECK(weight.shape[1] % elems == 0, std::string("tenstorrent kMatmulBTQuantGrouped: K must be a whole " diff --git a/tests/vt/test_tenstorrent_backend.cpp b/tests/vt/test_tenstorrent_backend.cpp index cbe56921b..836d9f69a 100644 --- a/tests/vt/test_tenstorrent_backend.cpp +++ b/tests/vt/test_tenstorrent_backend.cpp @@ -5766,7 +5766,7 @@ TEST_CASE("kTENSTORRENT keep-quant decode stages zero words during capture") { // answer. The registered encoding set is EXACTLY {Q4_K, Q8_0}: Q5_K and Q6_K // must refuse BY NAME (the owed grouped extension, recorded in the spec's W4 // plan), so a wrongly-widened kernel reds the refusal legs below. -TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped registers {Q4_K,Q8_0} and refuses the rest") { +TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped registers {Q4_K,Q5_K,Q6_K,Q8_0} and refuses the rest") { if (!TenstorrentPresent()) { MESSAGE("SKIPPED: no Tenstorrent device on this box"); return; @@ -5789,16 +5789,29 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped registers {Q4_K,Q8_0} and refuses if (enc == vt::DType::kQ8_0) { put_f16(0, 0.1f + 0.2f * static_cast(rng() % 16) / 16.0f); for (int i = 2; i < 34; ++i) blk[i] = static_cast(rng() & 0xFF); - } else { // Q4_K + } else if (enc == vt::DType::kQ4_K) { put_f16(0, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); put_f16(2, 0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); for (int i = 4; i < 144; ++i) blk[i] = static_cast(rng() & 0xFF); + } else if (enc == vt::DType::kQ5_K) { + // block_q5_k: f16 d, f16 dmin, u8 scales[12], qh[32], ql[128]. All + // payload bytes are integers the decode reads exactly; only the two + // f16 scales need finite values. + put_f16(0, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); + put_f16(2, 0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); + for (int i = 4; i < 176; ++i) blk[i] = static_cast(rng() & 0xFF); + } else { // Q6_K + // block_q6_k: ql[128], qh[64], i8 scales[16], f16 d. Same reasoning. + for (int i = 0; i < 208; ++i) blk[i] = static_cast(rng() & 0xFF); + put_f16(208, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); } }; - // THE ADMITTED ARMS: one call each, E=2 tower, both encodings. Only + // THE ADMITTED ARMS: one call each, E=2 tower, all four encodings. Only // reachability is pinned here — the numerics have their own sweep below. - for (const vt::DType enc : {vt::DType::kQ4_K, vt::DType::kQ8_0}) { + for (const vt::DType enc : + {vt::DType::kQ4_K, vt::DType::kQ5_K, vt::DType::kQ6_K, + vt::DType::kQ8_0}) { const int64_t kBlockBytes = vt::BlockBytes(enc); const int64_t kBlockElems = vt::BlockElems(enc); constexpr int64_t kE = 2, kN = 8, kP = 3; @@ -5844,11 +5857,10 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped registers {Q4_K,Q8_0} and refuses " must answer on the TENSTORRENT grouped arm, threw: ", what); } - // THE REFUSE SIDE: Q5_K / Q6_K (the owed grouped extension) and kQ4_0 (no - // TT arm anywhere) must throw naming THEMSELVES and the registered set — - // never fall through to a misread, never silently widen. - for (const vt::DType enc : {vt::DType::kQ5_K, vt::DType::kQ6_K, - vt::DType::kQ4_0}) { + // THE REFUSE SIDE: kQ4_0 (no TT arm anywhere) must throw naming ITSELF and + // the four-encoding registered set — never fall through to a misread, never + // silently widen past the set. + for (const vt::DType enc : {vt::DType::kQ4_0}) { const int64_t kBlockBytes = vt::BlockBytes(enc); const int64_t kBlockElems = vt::BlockElems(enc); constexpr int64_t kE = 2, kN = 8, kP = 3; @@ -5888,7 +5900,7 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped registers {Q4_K,Q8_0} and refuses CHECK_MESSAGE(what.find(enc_name) != std::string::npos, "the refusal must name the encoding (", enc_name, "), got: ", what); - CHECK_MESSAGE(what.find("kQ4_K/kQ8_0") != std::string::npos, + CHECK_MESSAGE(what.find("kQ4_K/kQ5_K/kQ6_K/kQ8_0") != std::string::npos, "the refusal must name the registered set, got: ", what); } } @@ -5897,7 +5909,7 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped registers {Q4_K,Q8_0} and refuses // grouped provider (cpu_quant_gemm.cpp MatmulBTQuantGroupedKernel — the SAME // kMatmulBTQuant integer-dot core once per group) across shapes sweeping P, N, // K, E: the E=1 dense arm (ids all zero), the E=N expert tower arm, P=1, a -// broadcast activation ([1,K]), a non-tile-multiple N, both registered +// broadcast activation ([1,K]), a non-tile-multiple N, all four registered // encodings, and a 27B-mirroring slice (K = the Qwen3.8-27B hidden_size, // N a production-like per-group intermediate). The bar is the W2-ratified // analytic operand-rounding envelope — bf16-round-once both operands, f32 @@ -5932,10 +5944,21 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped matches the CPU grouped provider i if (enc == vt::DType::kQ8_0) { put_f16(0, 0.1f + 0.2f * static_cast(rng() % 16) / 16.0f); for (int i = 2; i < 34; ++i) blk[i] = static_cast(rng() & 0xFF); - } else { // Q4_K + } else if (enc == vt::DType::kQ4_K) { put_f16(0, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); put_f16(2, 0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); for (int i = 4; i < 144; ++i) blk[i] = static_cast(rng() & 0xFF); + } else if (enc == vt::DType::kQ5_K) { + // block_q5_k: f16 d, f16 dmin, u8 scales[12], qh[32], ql[128]. The + // payload bytes are integers the decode reads exactly; only the two + // f16 scales need finite values. + put_f16(0, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); + put_f16(2, 0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); + for (int i = 4; i < 176; ++i) blk[i] = static_cast(rng() & 0xFF); + } else { // Q6_K + // block_q6_k: ql[128], qh[64], i8 scales[16], f16 d. Same reasoning. + for (int i = 0; i < 208; ++i) blk[i] = static_cast(rng() & 0xFF); + put_f16(208, 0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); } }; @@ -5953,6 +5976,26 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped matches the CPU grouped provider i {4, 33, 1, 4, -1, vt::DType::kQ8_0, "Q8_0, non-tile N=33"}, {2, 1024, 20, 1, -1, vt::DType::kQ4_K, "27B mirror: K=5120 (Qwen3.8-27B hidden_size), N=1024 per-group slice"}, + // W4a wave-2b: the Q5_K/Q6_K grouped extension. No ROCm grouped + // reference exists for Q5_K (rocm_grouped_gemm.hip admits + // Q8_0/Q4_K/Q6_K); the decode is the W3 dense chain, bit-exact vs + // vt::cpu::BlockToFloat — the same numerics authority for both arms. + {3, 8, 1, 1, -1, vt::DType::kQ5_K, "Q5_K E=1 dense arm, ids all zero"}, + {4, 8, 2, 4, -1, vt::DType::kQ5_K, + "Q5_K E=N expert tower arm, permuted ids"}, + {4, 33, 1, 4, 1, vt::DType::kQ5_K, + "Q5_K broadcast [1,K] act, non-tile N=33"}, + {2, 1024, 20, 1, -1, vt::DType::kQ5_K, + "Q5_K 27B mirror: K=5120, N=1024 per-group slice (the 27B pin carries " + "48 Q5_K tensors)"}, + {3, 8, 1, 1, -1, vt::DType::kQ6_K, "Q6_K E=1 dense arm, ids all zero"}, + {4, 8, 2, 4, -1, vt::DType::kQ6_K, + "Q6_K E=N expert tower arm, permuted ids"}, + {4, 33, 1, 4, 1, vt::DType::kQ6_K, + "Q6_K broadcast [1,K] act, non-tile N=33"}, + {2, 1024, 20, 1, -1, vt::DType::kQ6_K, + "Q6_K 27B mirror: K=5120, N=1024 per-group slice (the 27B pin carries " + "67 Q6_K tensors)"}, }; for (const Shape& s : shapes) { const int64_t kBlockBytes = vt::BlockBytes(s.enc); @@ -6131,4 +6174,90 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped matches the CPU grouped provider i << " dequant=" << dequant[i]); } } + + // ---- THE Q5_K/Q6_K SLICE-DECODE BIT-EXACT LEGS (W4a wave-2b) ---- + // The Q8_0 leg above pins slice selection on a bf16-exact grid. These two + // pin the W3 dense chains — the only Q5_K/Q6_K decode there is: the ROCm + // grouped kernel has no Q5_K arm to mirror, so the decode derives from the + // W3 chain and its bit-exactness vs vt::cpu::BlockToFloat — through the + // SLICE path: with one-hot activations the grouped output is exactly + // bf16(decode(selected row, col)), and decode is W3-pinned bit-exact vs + // BlockToFloat, so the expected bits are bf16(BlockToFloat) at the routed + // element. A slice-selection or staging defect (wrong expert row-range, + // word-lane misindex, a read past the 210-byte block into the pad) lands on + // a different value and cannot hide. + for (const vt::DType enc : {vt::DType::kQ5_K, vt::DType::kQ6_K}) { + constexpr int64_t kE = 3, kN = 8, kP = 5, kNb = 1; + const int64_t elems = vt::BlockElems(enc); // 256 per K-quant block + const int64_t bb = vt::BlockBytes(enc); // 176 (Q5_K) / 210 (Q6_K) + const int64_t K = kNb * elems; + std::mt19937 rng(static_cast(20260913u)); + std::vector packed(kE * kN * kNb * bb); + for (int64_t b = 0; b < kE * kN * kNb; ++b) { + uint8_t* blk = packed.data() + b * bb; + fill_block(blk, enc, rng); + if (enc == vt::DType::kQ6_K) { + // Q6_K dequant y = (d*sc)*(q-32): a NEGATIVE or zero scale meeting a + // zero q makes the true dequant -0, which any dot then flattens to + // +0 in the f32 accumulate (IEEE (+0)+(-0) = +0) — the DECODE keeps + // the -0 (the W3 pin, or_sign repair at the Q6_K arm), but a matmul + // cannot carry it through a sum. The bit-exact leg, exactly like the + // Q8_0 leg above, therefore constrains its data to the -0-free + // class: strictly positive scales make sign(y) = sign(q-32) with + // q = 32 giving +0. + for (int i = 192; i < 208; ++i) + blk[i] = static_cast(1 + rng() % 127); + } + } + // One-hot activations with repeated, non-ascending ids — every selected + // slice exercised independently, exactly as the Q8_0 leg above. + std::vector a_bf(kP * K, 0u); + for (int64_t p = 0; p < kP; ++p) a_bf[p * K + (p % K)] = vt::F32ToBF16(1.0f); + std::vector ids = {2, 0, 2, 1, 0}; + + // The decode oracle: BlockToFloat over the IDENTICAL bytes (the W3 bit + // authority), then the ONE bf16 RNE the device applies after the slice + // decode. Every payload is an exact f32 integer product under finite + // positive scales, so no -0/NaN ambiguity survives the chain. + std::vector w_f32(kE * kN * K); + vt::cpu::BlockToFloat(enc)(packed.data(), w_f32.data(), kE * kN * K); + std::vector dequant(kP * kN); + for (int64_t p = 0; p < kP; ++p) + for (int64_t n = 0; n < kN; ++n) + dequant[p * kN + n] = w_f32[(ids[p] * kN + n) * K + (p % K)]; + + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(kP * kN * sizeof(float)); + void* mem_i = backend.Alloc(ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w, packed.data(), packed.size()); + backend.Copy(q, mem_i, ids.data(), ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, K}); + Tensor w_t = Tensor::Contiguous(mem_w, enc, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kE * kN, K}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP, kN}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + std::vector tt_out(kP * kN, 0.0f); + backend.Copy(q, tt_out.data(), mem_o, tt_out.size() * sizeof(float)); + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); + for (int64_t i = 0; i < kP * kN; ++i) { + const uint16_t got_bf = vt::F32ToBF16(tt_out[i]); + const uint16_t want_bf = vt::F32ToBF16(dequant[i]); + const std::string msg = std::string("slice-decode bit-exact (") + + vt::Name(enc) + "): p=" + + std::to_string(i / kN) + " n=" + + std::to_string(i % kN) + " tt=" + + std::to_string(tt_out[i]) + " dequant=" + + std::to_string(dequant[i]); + CHECK_MESSAGE(got_bf == want_bf, msg); + } + } } From b40907ee2d5efcd20b166d85fb309776d4e4aa6e Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Mon, 7 Sep 2026 02:43:20 +0200 Subject: [PATCH 06/10] =?UTF-8?q?record(BACKEND-TENSTORRENT-KEEPQUANT):=20?= =?UTF-8?q?the=2027B=20is=20dense=20=E2=80=94=20corrected=20axis=20and=20w?= =?UTF-8?q?ave-3=20shape=20(#2959)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A tensor-name scan of the pinned artifact settles what the dtype-only survey misread: arch qwen35, zero ffn_*_exps / ffn_gate_inp / exp_probs tensors, and blk.0 carries singular ffn_down/ffn_gate/ffn_up. There are no experts on the 27B; the 294 Q4_K tensors are dense ffn/attn weights. The "experts Q4_K x294 -> 9.81 GiB" split in the budget was wrong, and so was "the 27B gate is reachable only behind the dense E=1 arm" for the interim reason given. The corrected axis: the whole keep-quant set beyond the gather class (~14.27 GB packed) cannot exist as twins (~42 GB); the 27B is served packed through the E=1 arm with a chunked slice-decode + accumulate matmul, so the captured graph holds chunk buffers and never a whole-weight tile. Budget ~22-23 GB on 32 GB. Replay re-decodes each step — correct, slower — and throughput is W4b's lever. The E=N expert arm stays staged-owed behind a MoE artifact (30B-A3B class; none on disk; a download needs authority). Wave-2b landed 14e8fe471 (all four encodings, suite 27,456 green, vehicle 147/147, operator-passed, pushed). Wave-3a = E=1 chunked capture-compatible packed dense; wave-3b = 27B wiring + gate + MTP skip + USAGE pin. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/specs/tenstorrent-keepquant.md | 32 ++++++++++++++++++-------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/.agents/specs/tenstorrent-keepquant.md b/.agents/specs/tenstorrent-keepquant.md index 6713b1616..ddb42da8e 100644 --- a/.agents/specs/tenstorrent-keepquant.md +++ b/.agents/specs/tenstorrent-keepquant.md @@ -333,14 +333,21 @@ E=1 covers dense; E=N covers experts fed by the stacked tower compatibility of the grouped arm is wave-3's committed obligation. (3) MTP `blk.64.*` skip/refuse by name. (4) 27B e2e greedy near-tie gate, checkpoint-gated opt-in loud-skip (#2811 precedent), goldens vs the pinned llama.cpp b10451 oracle, 500-mnat band + 0 forward-divergent. -(5) `docs/USAGE.md` pin in the same change. Memory axis on 27B at steady -state: 9.81 GiB experts packed + 6.01 GB non-expert packed (both through -the native grouped kernel) + 2.37 GiB embedding twin + 2.37 GiB output -twin ≈ 20.6 GB + activations on 32 GB. Interim (experts alone): non-expert -twins 18.47 GB push the total to ~33 GB, so the 27B gate (wave-3) is -reachable only behind the dense E=1 arm. The 0.8B vehicle carries no -experts: its gate guards the dense twin path and must stay green -throughout. +(5) `docs/USAGE.md` pin in the same change. CORRECTED 2026-09-07 +(tensor-name scan of the pinned artifact): the 27B is DENSE — arch +`qwen35`, zero `ffn_*_exps` / `ffn_gate_inp` / `exp_probs` tensors, and +`blk.0` carries singular `ffn_down`/`ffn_gate`/`ffn_up`. The earlier +"experts Q4_K ×294" classification was a dtype-only misread; those 294 +Q4_K tensors are dense ffn/attn weights. Memory axis on 27B: the keep- +quant set beyond the gather class (~14.27 GB packed) cannot exist as +twins (~42 GB); it is served PACKED through the E=1 arm with a CHUNKED +slice-decode + accumulate matmul, so the captured graph holds chunk +buffers and never a whole-weight tile — ≈ 15.92 GB packed + 2.37 GiB +embedding twin + 2.37 GiB output twin + chunk tiles + activations ≈ +22-23 GB on 32 GB. Replay re-decodes each step: correct, slower; +throughput is W4b's lever. The E=N expert arm serves the family's MoE +models (30B-A3B class) and stays staged-owed: no MoE artifact is on disk +and a ~17-20 GB download needs authority. **W4b — the int8-dot lever ([#3031](https://github.com/mudler/vllm.cpp/issues/3031)).** Quantized-domain integer vec_dot behind the same seam; profile-first @@ -372,4 +379,11 @@ the branch diff reverted; dense twins are the shipped behavior. W4a(1) re-anchors on the TT `kMatmulBTQuantGrouped` provider (wave-2: packed tower, native in-kernel dequant, E=1 dense + E=N experts, Q4_K/Q8_0 first, Q5_K extension owed, staged slice owed to wave-3 wiring and the 27B gate). -The 128M threshold dissolves. +The 128M threshold dissolves. Wave-2b landed 14e8fe471 (registered set +{Q4_K,Q5_K,Q6_K,Q8_0}; suite 27,456 green; vehicle 147/147; operator- +passed; pushed). AMENDED 2026-09-07 (fourth): the 27B is dense +(tensor-name scan — the ×294 "experts" were a dtype-only misread), so +the 27B path is the E=1 arm with chunked slice-decode (capture holds +chunks, never whole weights); the E=N expert arm stays staged-owed +behind a MoE artifact. Wave-3a = E=1 chunked capture-compatible packed +dense; wave-3b = 27B wiring + gate + MTP skip + USAGE pin. From d614aa4f331d31112e04f31ce2a9f37faae1186c Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Mon, 7 Sep 2026 04:55:00 +0200 Subject: [PATCH 07/10] feat(BACKEND-TENSTORRENT-KEEPQUANT): capture-safe chunked E=1 grouped keep-quant arm (#2959) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The grouped provider's E=1 arm now decodes [chunk, K] word ranges — one tile matmul per chunk, partials concatenated in f32 — instead of the whole-[N,K] slice, and it never reads the routing ids (E=1 ids are statically all zero): no EnsureHost, no host readback, every chunk offset a capture-time constant. E=N keeps the wave-2 whole-slice decode and its dynamic-id readback unchanged. Both capture-time failure modes are now measured, not assumed (P150, 2026-09-07, head shape [248320, 1024] Q6_K). Before: the whole-slice arm died in capture demanding 4,068,474,880 B of device DRAM — one 1.02 GiB f32 plane (bank_manager.cpp:462), the grouped arm's wave-1b mechanism. Many small chunks die the other way, on the trace region (mesh_trace.cpp:81): the captured command stream costs ~3.3 MB per chunk (485 chunks = 1,566,662,656 B; 16 = 53,764,096 B; 2-4 capture clean). CHUNK POLICY: bound the chain's largest live tensor — one [chunk, K] f32 plane — at 256 MiB (chunk = 256 MiB / (4 B * K), a whole decode when it fits) with ceil(N / 8) as the command-stream guard. The head decodes as 4 chunks: capture demand 13,484,032 B of the 52,428,800 B region, staging writes during capture 0, capture dumps x2 byte-identical, demand identical across both passes. Envelope: the grouped sweep under forced 3-row chunks reproduces every pre-change E=1 ratio bit-for-bit (vs cpu 0.378085, 0.449989, 0.390167, 0.487656, 0.491194, 0.278552, 0.376211; new E=1 broadcast shape 0.468875), and a dedicated bit-exact leg pins the chunked decode against BlockToFloat at forced chunk boundaries (81 assertions). Full backend suite 63/63 cases, 524,267 assertions. Vehicle battery 16/16 prompts, 147/147 assertions, 0 forward-divergent. Test hooks: KeepQuantChunkRowsOverrideForTest forces the chunk rows; the trace-bound leg reads VT_KEEPQUANT_TEST_CHUNK_ROWS so one build surveyed the demand curve; LastTraceBytesForTest reports the device trace demand. Staged-unreached: the E=1 arm stays unreached from a production entry point — the model wiring (27B + the qwen3_5 head routing) is wave-3b's gate (issue #3030); the E=N capture indirection is staged-owed behind a MoE artifact (spec ## W4). Anchors moved: .agents/backend-matrix.md (KEEPQUANT row lifecycle and evidence to W2a/2b LANDED + W3a ON BRANCH, and the predicate registered- set phrase after wave-2b; owed narrows to the int8 lever, the wave-3b wiring, and the captured e2e vehicle leg) and the GDN row's registration anchor, which this diff's insertions shifted (tenstorrent_ops.cpp 7428-7442 -> 7517-7531; check-agent-record stale back at baseline 28). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/backend-matrix.md | 4 +- src/vt/tenstorrent/tenstorrent_device.h | 17 + src/vt/tenstorrent/tenstorrent_ops.cpp | 136 ++++++- tests/vt/test_tenstorrent_backend.cpp | 482 ++++++++++++++++++++++++ 4 files changed, 622 insertions(+), 17 deletions(-) diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index db1e820a9..25a645a67 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -246,9 +246,9 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | | `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | **Gate PASSED on Blackhole P150 (2026-08-12):** [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap **0.062 nats**, BACKEND PROOF 0 declines (kMatmul selections=256 = untied lm_head on device, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded, same as Qwen3-0.6B TT precedent). Exit SIGSEGV 139 is the known MeshDevice teardown crash, not a gate failure | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | -| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7428-7442](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7428-L7442) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | +| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7517-7531](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7517-L7531) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | | `BACKEND-TENSTORRENT-GDN-DEVICE-PURE` | Child of `BACKEND-TENSTORRENT-GDN` — make the decode-side GDN ops (`CausalConv1dUpdateKernel`, `GdnDecodeKernel`) device-resident so a tt-metal trace capture admits them, unblocking the Qwen3.5-0.8B captured arm, the last capture-blocked family with a committed eager pair (#2907, owed from #2812) | Move the per-call host orchestration on-device: resident inputs from the graph's producer ops, device-side indexed state update (baked slot addressing re-primed by the recapture cadence, or `kGdnStateGather`/`kGdnStateScatter` indirection — decided on recapture-cost measurement), conv two-views coherent on device, token readback outside the captured span | [CausalConv1dUpdateKernel :5013](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5013), [GdnDecodeKernel :5225](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5225) (EnsureHost ×5, ReadIdxHost, UploadTensor ×5+), q35 harness pair selection [test_qwen35_paged_engine.cpp:233](../tests/parity/test_qwen35_paged_engine.cpp#L233), fatal repro `fd_mesh_command_queue.cpp:760` 2/2 | Red-first: the #2812 loud-skip opt-in cells run captured; captured dump ×2 byte-identity with reset between; teacher-forced pair vs the transformers oracle inside the eager band; Qwen3.5 joins `DecodeCaptureDefaultArch`, ambient adjudicates CAPTURED, env=0 eager; tamper + arch-deletion mutations red; gate arm selection [test_qwen35_paged_engine.cpp:301-330](../tests/parity/test_qwen35_paged_engine.cpp#L301-L330); closure [parity-ledger.md:947](parity-ledger.md#L947) | [tenstorrent-gdn-device-pure.md](specs/tenstorrent-gdn-device-pure.md) | `DONE` | `bcade48d6` | -| `BACKEND-TENSTORRENT-KEEPQUANT` | Child of `BACKEND-TENSTORRENT` — keep-quant dense dot on TT-Metal ([#2959](https://github.com/mudler/vllm.cpp/issues/2959)): GGUF k-quant arms refuse on the P150 until this row lands — W1 (#2989, open) lands the Q4_K decode, W2 (branch) lands the dot and the `kTENSTORRENT` predicate arm (`gguf_keep_quant.cpp:136-148`) admitting exactly `{Q4_K}` — while the smallest Qwen3.8 artifact that fits the card is the dense 27B Q4_K_M (17.1 GB; bf16 is 53.8 GB) and a quantized Qwen3.5 (0.8B, ~0.5 GB) is the test vehicle | ggml k-quant block formats via our reader, pinned bit-exact vs llama.cpp `b10451` (#2240/#2260 lineage); no vLLM mirror for TT (secondary substrate, deviation by design); substrate surveyed 2026-09-05: tt-metal has NO packed-weight matmul (`ttnn` quantization is per-tensor activation dquant only), so the kernel is ours — resident blocks, on-core decode to bf16 tiles through f32 (decode bit-exact vs `BlockToFloat`), the dot reusing the `kMatmulBT` tile path, device-bf16 band at the residual-golden boundary, the int8 dot as the named next lever | W1 LANDED (#2989): the Q4_K block-decode device path. W2 ON BRANCH: the dot (`MatmulBTQuantKernel`, decode → one bf16 RNE → `kMatmulBT` tile matmul, reached through `vt::MatmulBT`'s block-weight dispatch) + the predicate arm admitting exactly the registered set (never wider — the GLM-5.3 W10 lesson). OWED W3: capture-safe residency + the e2e vehicle. OWED W4: Q5_K/Q6_K/Q8_0, the int8 lever, the 27B arm | W1 LANDED (#2989): decode bit-exact vs `vt::cpu::BlockToFloat` across a shape sweep (red-first, op-level suite pattern). W2 ON BRANCH: the dot vs the decode-based bf16 oracle inside the analytic operand-rounding envelope + the predicate routing set (both red-first). OWED W3: capture dump ×2 byte-identity (#2907 discipline); e2e 16-prompt battery on the P150 vs the bf16 arm's committed pair (STRICT or inside the ≤500-mnat near-tie band) | [tenstorrent-keepquant.md](specs/tenstorrent-keepquant.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-KEEPQUANT` — implementer on the row branch; spec commit on `row/BACKEND-TENSTORRENT-KEEPQUANT`, claim in [.agents/claims/](claims/CLAIM-BACKEND-TENSTORRENT-KEEPQUANT.md) | +| `BACKEND-TENSTORRENT-KEEPQUANT` | Child of `BACKEND-TENSTORRENT` — keep-quant dense dot on TT-Metal ([#2959](https://github.com/mudler/vllm.cpp/issues/2959)): GGUF k-quant arms refuse on the P150 until this row lands — W1 (#2989, open) lands the Q4_K decode, W2 (branch) lands the dot and the `kTENSTORRENT` predicate arm (`gguf_keep_quant.cpp:136-148`) admitting exactly the registered set `{Q4_K, Q5_K, Q6_K, Q8_0}` after wave-2b — while the smallest Qwen3.8 artifact that fits the card is the dense 27B Q4_K_M (17.1 GB; bf16 is 53.8 GB) and a quantized Qwen3.5 (0.8B, ~0.5 GB) is the test vehicle | ggml k-quant block formats via our reader, pinned bit-exact vs llama.cpp `b10451` (#2240/#2260 lineage); no vLLM mirror for TT (secondary substrate, deviation by design); substrate surveyed 2026-09-05: tt-metal has NO packed-weight matmul (`ttnn` quantization is per-tensor activation dquant only), so the kernel is ours — resident blocks, on-core decode to bf16 tiles through f32 (decode bit-exact vs `BlockToFloat`), the dot reusing the `kMatmulBT` tile path, device-bf16 band at the residual-golden boundary, the int8 dot as the named next lever | W1 LANDED (#2989): the Q4_K block-decode device path. W2a/2b LANDED (5e43d770, 14e8fe471): the grouped provider `kMatmulBTQuantGrouped` — E=1 dense, E=N stacked tower, registered set exactly {Q4_K, Q5_K, Q6_K, Q8_0}. W3a ON BRANCH: the E=1 arm is CHUNKED slice-decode + f32 assembly — capture-safe on the 0.8B head (trace demand 13,484,032 B of the 52,428,800 B region; capture ×2 byte-identity; routing ids never read; staging writes during capture 0; both capture-time failure modes measured and bounded by the CHUNK POLICY); E=N capture indirection staged-owed behind a MoE artifact (spec ## W4). OWED: the int8 lever, the 27B arm + model wiring (wave-3b), the captured e2e vehicle leg | W1 LANDED (#2989): decode bit-exact vs `vt::cpu::BlockToFloat` across a shape sweep (red-first, op-level suite pattern). W2a/2b LANDED: grouped sweep green across the four encodings (E=1 vs cpu 0.279–0.720, vs analytic-ref 0.201–0.514). W3a ON BRANCH (red-first): the head-shape capture leg reds at base in the wave-1b class (whole-tensor device DRAM fatal, bank_manager.cpp:462) and greens with the CHUNK POLICY (trace demand 13,484,032 B, staging writes 0, ×2 byte-identity, deterministic across passes); the chunked-decode bit-exact leg pins decode math unchanged (81 assertions); the sweep under forced 3-row chunks is bit-identical to the whole-slice ratios; full backend suite 63/63 cases, 524,267 assertions; vehicle 16/16, 147/147, 0 forward-divergent. OWED: the captured e2e vehicle leg | [tenstorrent-keepquant.md](specs/tenstorrent-keepquant.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-KEEPQUANT` — implementer on the row branch; spec commit on `row/BACKEND-TENSTORRENT-KEEPQUANT`, claim in [.agents/claims/](claims/CLAIM-BACKEND-TENSTORRENT-KEEPQUANT.md) | | `BACKEND-TENSTORRENT-QWEN35` | Child of `BACKEND-TENSTORRENT` — the wiring row: `Qwen3_5ForConditionalGeneration` (dense text GDN hybrid) on the TT allow-list, the op delta the family forward refuses by name (`kGdnPostConv`, `kSigmoidGateBf16`, `kAttnQkNormRopeGate`, `kAttnQkNormRope` — pinned empirically by a W0 refusal sweep), and the first e2e gate. Makes the GDN row's ops production-reached (#1715 stays open until the family runs; GDN row lifecycle moves in the same change this lands) | Substrate: the GDN row's 8 kernels + the 27-op TT registry; e2e mirrors the ratified Mistral TT golden treatment (`VT_DUMP_IDS` bootstrap → `scripts/qwen3-neartie-gap-transformers.py` teacher-forced near-tie gaps → committed device-golden pair, POL-ORACLE deviation); op-level oracle is the CPU f32 arm. Capacity: `Qwen/Qwen3.5-0.8B` bf16 ≈1.6 GB fits (proven envelope Mistral-7B ≈14.5 GB); 27B bf16 ≈53.8 GB and GGUF k-quant arms (no TT kernels) refused by name, owed | `src/vllm/platforms/tenstorrent.cpp` (allow-list, lands last) + `src/vt/tenstorrent/tenstorrent_ops.cpp` (op delta) + `tests/parity/test_qwen35_paged_engine.cpp` (TT arm) | Spec committed 2026-08-23; W0 sweep, W1 op delta, W2 e2e, W3 GDN-reviewer leftovers (d2h counter completeness, `conv_transposed` fast-path check) owed | [tenstorrent-qwen35.md](specs/tenstorrent-qwen35.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-QWEN35` | | `BACKEND-TENSTORRENT-TRACE-RUNNER` | Child of `BACKEND-TENSTORRENT` — wire the landed graph-capture foundation (#354 / `59568772`) into a capturable forward region. Handoff §8/§9 "not done". **Decision (2026-08-13): NO-GO for pure T=1 decode capture — MEASURED, not assumed.** | CUDA is the ONLY backend with `SupportsGraphCapture()==true` (`cuda_backend.cu:184-240`); Metal/Vulkan `false`. Shared decode-graph framework `Qwen3DenseDecodeGraph` (qwen3.cpp:489, used by Qwen3/Mistral/Llama/InternLM2) gated on `support_static_graph_mode()` which TT does NOT override (base `false`) | `src/vt/tenstorrent/tenstorrent_backend.cpp:70-76` (capture surface, landed) + `tenstorrent_ops.cpp` `Trace*` (landed); NOT wired into a dense forward | **Measured on Blackhole P150 (2026-08-13):** Q1 no host-free region at T=1; Q2 all-device-at-T=1 costs 12.5→10.7 tok/s; Q2b capture attempt with both overrides flipped → ttnn **`TT_FATAL: Reads are not supported during trace capture`** (backtrace through `to_vector`), `0 replays`. The T=1 forward does device→host readbacks that ttnn trace prohibits; flipping the two thresholds is insufficient. Requires a host-free `ForwardLayers` redesign, not a threshold tweak. Next: prefill capture (separate row) must first audit its `to_vector` readbacks | [tenstorrent-trace-runner.md](specs/tenstorrent-trace-runner.md) | `SPIKE` (decision record complete) | `CLAIM-BACKEND-TENSTORRENT-TRACE-RUNNER` | | `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | Child of `BACKEND-TENSTORRENT` — make the per-decode-layer forward region host-free (zero `to_vector` readbacks) so mesh-trace capture can run. Decomposes into R1 RmsNorm+RoPE all-device, R2 QkvSplit+ReshapeAndCache device, R3 PA decode device-resident metadata, R4 capture wire+measure. Prerequisite revealed by the trace-runner spike (capture aborts on `to_vector`). | CUDA decode-graph capture contract (`cuda_backend.cu:184-197`): captured region is async, no host sync, no malloc, fixed ptrs. TT must match: no `to_vector` between Begin/EndCapture | `src/vt/tenstorrent/tenstorrent_ops.cpp` (RmsNorm:1067, PreferDeviceRope:1344, QkvSplit:1460, ReshapeAndCache:1527, PagedAttention:2009) | **R1-R3b MEASURED on P150** (env-gated `VT_TT_HOST_FREE_DECODE`, inert by default; 23/23 TT tests incl. a dedicated default-path inertness guard, M1-mutation-proved): R1 threshold flip landed; R2 device->device copy (`CopyDeviceDeviceIfCapture`, ttnn::copy+empty) landed; R3 program-cache warm (`enable_program_cache` + eager-warm) landed; R3b device zero-fill (`MemsetDeviceIfCapture`) landed. Capture enters the forward and reaches layer ops (CastBf16/RmsNorm fire). Remaining item-5 blocker = per-op enqueue_write; answer = persistent device tensors + before-replay populate (tt-metal vLLM plugin design). Full blocker map + architecture in [tenstorrent-host-free-r1.md](specs/tenstorrent-host-free-r1.md). **R5 (2026-08-21, #1604): host-free decode is the DEFAULT** — `HostFreeDecodeEnabled()` centralizes the polarity (exact `0` = pre-flip opt-out), both device golden pairs re-dumped and re-adjudicated under the new default (Qwen3 max 375 mnats, Mistral max 250 mnats, 0 outside top-K), both paged-engine gates 16/16 green (125/125 + 128/128 assertions), default leg 10.94-11.06 tok/s vs 5.34 opt-out (2.1x, Qwen3-0.6B b1). Capture declined by default: multi-request captured hangs (#1625); TT async scheduling stays off, no `SupportsAsyncSampledTokenReadback` override (#1627) | [tenstorrent-host-free-forward.md](specs/tenstorrent-host-free-forward.md) | `ACTIVE` (R1-R3b + R5 default flip gated on `row/BACKEND-TENSTORRENT-HOST-FREE-1604`; capture declined per #1625, TT async readback owed per #1627) | `CLAIM-BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | diff --git a/src/vt/tenstorrent/tenstorrent_device.h b/src/vt/tenstorrent/tenstorrent_device.h index 9cb69a39a..ea4731759 100644 --- a/src/vt/tenstorrent/tenstorrent_device.h +++ b/src/vt/tenstorrent/tenstorrent_device.h @@ -174,9 +174,26 @@ bool CopyDeviceDeviceIfResident(void* dst, const void* src, size_t bytes); #ifdef VLLM_CPP_TENSTORRENT int64_t KeepQuantCaptureStagingWrites(); void ResetKeepQuantCaptureStagingWritesForTest(); + +// BACKEND-TENSTORRENT-KEEPQUANT W4a wave-3a test hooks (the red/green +// measurement pair for the chunked E=1 arm): +// - LastTraceBytesForTest: the device-reported total live trace-buffer +// bytes as of the last EndCapture — the observable for the wave-1b +// falsification class (a whole-weight tile inside a captured graph +// demanding 425,754,624 B against the 52,428,800 B region). The reading +// accumulates across every live trace, so tests release a graph before +// measuring the next one in isolation. +// - KeepQuantChunkRowsOverrideForTest: forces the E=1 slice-decode chunk +// row count (0 = the production policy default), so the sweep and the +// bit-exact legs execute MANY chunks at shapes whose policy chunk would +// cover the whole [N, K] slice in one pass. +int64_t LastTraceBytesForTest(); +void KeepQuantChunkRowsOverrideForTest(int64_t rows); #else inline int64_t KeepQuantCaptureStagingWrites() { return 0; } inline void ResetKeepQuantCaptureStagingWritesForTest() {} +inline int64_t LastTraceBytesForTest() { return 0; } +inline void KeepQuantChunkRowsOverrideForTest(int64_t) {} #endif // ITEM 5 (rope): driver-side warm hook — populate the persistent device diff --git a/src/vt/tenstorrent/tenstorrent_ops.cpp b/src/vt/tenstorrent/tenstorrent_ops.cpp index 552468d77..a2274d2dc 100644 --- a/src/vt/tenstorrent/tenstorrent_ops.cpp +++ b/src/vt/tenstorrent/tenstorrent_ops.cpp @@ -203,6 +203,14 @@ std::atomic& KeepQuantCaptureStagingWritesCounter() { static std::atomic* c = new std::atomic(0); // never destroyed (#1486) return *c; } + +// W4a wave-3a: the E=1 slice-decode chunk-rows override (0 = production +// policy). Internal linkage; the ForTest setter below the anonymous +// namespace is the external surface (the staging-counter pattern). +std::atomic& KeepQuantChunkRowsOverride() { + static std::atomic v{0}; + return v; +} } // namespace // ITEM 5 (rope): persistent device cos/sin (expanded per head), built OUTSIDE @@ -2538,11 +2546,15 @@ void MatmulBTQuantKernel(Queue&, Tensor& out, const Tensor& a, const Tensor& b) // chain, bit-exact vs vt::cpu::BlockToFloat. Any other encoding refuses BY // NAME. Never a silent wrong answer — the ROCm refusal precedent. // -// NOT capture-safe yet, and staged UNREACHED: the per-call EnsureHost of the -// routing ids and the per-group decode writes are eager-path constructs; -// capture compatibility of the grouped arm is wave-3's committed obligation, -// and no production entry point reaches this kernel yet (the wiring row is -// W4a wave-3, dense E=1 + the 27B gate). +// W4a wave-3a (#3030) split the two arms by capture compatibility: +// - E=1 (dense, the 27B path) is CHUNKED slice-decode + f32 assembly and +// never reads the routing ids (statically all zero) — capture-clean and +// trace-bounded, see the CHUNK POLICY comment in the arm; +// - E=N (experts) keeps the wave-2 whole-slice decode and its dynamic-id +// host readback; its capture indirection is staged-owed behind a MoE +// artifact (spec ## W4), and it stays UNREACHED (no production entry +// point until wave-3b wires the model). +constexpr int64_t kKeepQuantChunkPlaneBytes = 256 << 20; // 256 MiB f32 plane void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, const Tensor& weight, const Tensor& expert_ids) { @@ -2593,15 +2605,6 @@ void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, if (P == 0 || N == 0) return; MeshDevice& device = SharedMeshDevice(); - // The routing ids are small; the established TT index-tensor contract is - // EnsureHost + a host read (EmbeddingKernel), range-checked like the - // embedding gather. - EnsureHost(expert_ids); - const int32_t* eids = expert_ids.Ptr(); - for (int64_t p = 0; p < P; ++p) - VT_CHECK(eids[p] >= 0 && eids[p] < E, - "tenstorrent kMatmulBTQuantGrouped: expert id out of range (id " + - std::to_string(eids[p]) + ", E " + std::to_string(E) + ")"); // Stage the PACKED tower once — the resident i32 word shadow keyed by the // host weight pointer, served forever after (the dense arm's pattern). The @@ -2619,10 +2622,96 @@ void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, ttnn::typecast(std::move(dev_a), ttnn::DataType::BFLOAT16), ttnn::Layout::TILE); ttnn::Tensor a_rows; - if (Pa > 1) + if (E > 1 && Pa > 1) a_rows = ttnn::to_layout(std::move(dev_a), ttnn::Layout::ROW_MAJOR); const uint32_t wpb = static_cast(KeepQuantWordsPerBlock(enc)); + + if (E == 1) { + // == W4a wave-3a (#3030): the DENSE arm — CHUNKED slice-decode + f32 + // assembly, capture-clean. The fourth spec amendment (b40907ee2) makes + // this arm the row's production surface: the whole keep-quant set beyond + // the gather class is served PACKED through E=1, and a captured graph + // must never hold a whole-weight tile. Wave-1b falsified that shape on + // the vehicle; the chunk-count survey below (P150, 2026-09-07) measured + // BOTH capture-time failure modes on the head shape [248320, 1024] Q6_K: + // one whole-weight chunk dies on DEVICE DRAM (bank_manager.cpp:462 — a + // 1.02 GiB f32 decode plane), while many small chunks die on the TRACE + // REGION (mesh_trace.cpp:81): the captured command stream costs ~3.3 MB + // per chunk (485 chunks = 1,566,662,656 B; 16 chunks = 53,764,096 B; + // 2-4 chunks capture clean). CHUNK POLICY: decode [chunk, K] word + // ranges — one tile matmul per chunk — so the live working set is the + // i32 word slice, the chain's f32 planes and the bf16 tile of ONE chunk + // plus the tiny f32 partials, never a whole-weight tile; each chunk's + // decoded tile dies before the next chunk allocates. The budget bounds + // the chain's largest live tensor — one [chunk, K] f32 plane — at + // 256 MiB (chunk = 256 MiB / (4 B . K), and a whole decode for any + // weight whose plane fits), while ceil(N / 8) keeps the command stream + // under the 52,428,800 B trace region for weights large enough to + // chunk. Chunks cover DISJOINT weight rows, so every output element is + // still ONE dot over the full K and chunks concatenate in f32; the + // decode itself is unchanged (the bit-exact leg pins it). Capture-clean: + // E=1 ids are statically all zero — the only in-range expert — so the + // routing ids are never EnsureHosted or read, and every chunk offset is + // a capture-time constant replayed verbatim. + const int64_t chunk_override = + KeepQuantChunkRowsOverride().load(std::memory_order_relaxed); + const int64_t chunk = + chunk_override > 0 + ? std::min(chunk_override, N) + : std::min(N, std::max( + kKeepQuantChunkPlaneBytes / (K * 4), + (N + 7) / 8)); + std::vector partials; + partials.reserve(static_cast((N + chunk - 1) / chunk)); + for (int64_t c0 = 0; c0 < N; c0 += chunk) { + const int64_t c1 = std::min(N, c0 + chunk); + const ttnn::Tensor sl = ttnn::slice( + words, + ttsl::SmallVector{static_cast(c0 * nb), 0u}, + ttsl::SmallVector{static_cast(c1 * nb), wpb}, + ttsl::SmallVector{1u, 1u}); + ttnn::Tensor wf = DecodeKeepQuantWordsF32(sl, enc, c1 - c0, nb, device); + ttnn::Tensor wb = ttnn::to_layout( + ttnn::typecast(std::move(wf), ttnn::DataType::BFLOAT16), + ttnn::Layout::TILE); + ttnn::Tensor part = ttnn::operations::matmul::matmul( + dev_a, std::move(wb), /*transpose_a=*/false, /*transpose_b=*/true); + partials.push_back(ttnn::to_layout( + ttnn::typecast(std::move(part), ttnn::DataType::FLOAT32), + ttnn::Layout::ROW_MAJOR)); + } + ttnn::Tensor assembled = + partials.size() == 1 + ? std::move(partials[0]) + : ttnn::concat(std::move(partials), /*dim=*/1); + if (Pa == 1 && P > 1) { + // Broadcast contract: every output row is the SAME [1, K] activation + // against expert 0 — replicate the assembled row. Bit-identical to the + // wave-2 per-group decode (identical operands, identical programs). + std::vector rows(static_cast(P), assembled); + assembled = ttnn::concat(std::move(rows), /*dim=*/0); + } + if (out.dtype == DType::kBF16) + assembled = + ttnn::typecast(std::move(assembled), ttnn::DataType::BFLOAT16); + CommitDeviceLogical2D(out, std::move(assembled), static_cast(P), + static_cast(N)); + return; + } + + // E=N EXPERT TOWER arm: the wave-2 path unchanged. The routing ids are + // dynamic here; the established TT index-tensor contract is EnsureHost + a + // host read (EmbeddingKernel), range-checked like the embedding gather. + // That host readback is the arm's eager construct, and its capture + // indirection is staged-owed behind a MoE artifact (spec ## W4). + EnsureHost(expert_ids); + const int32_t* eids = expert_ids.Ptr(); + for (int64_t p = 0; p < P; ++p) + VT_CHECK(eids[p] >= 0 && eids[p] < E, + "tenstorrent kMatmulBTQuantGrouped: expert id out of range (id " + + std::to_string(eids[p]) + ", E " + std::to_string(E) + ")"); + // The selected [N,K] slice for group p: word rows [e*N*nb, (e+1)*N*nb) — // decode, one bf16 RNE, TILE — the dense dot's exact weight convention. auto slice_decode = [&](int64_t p) { @@ -7461,6 +7550,18 @@ void ResetKeepQuantCaptureStagingWritesForTest() { KeepQuantCaptureStagingWritesCounter().store(0, std::memory_order_relaxed); } +// W4a wave-3a test hooks — the contract lives in tenstorrent_device.h. +void KeepQuantChunkRowsOverrideForTest(int64_t rows) { + KeepQuantChunkRowsOverride().store(rows, std::memory_order_relaxed); +} +std::atomic& LastTraceBytes() { + static std::atomic v{0}; + return v; +} +int64_t LastTraceBytesForTest() { + return LastTraceBytes().load(std::memory_order_relaxed); +} + // ---- ttnn mesh-trace capture (Backend graph-capture mapping) ---------------- // Process-local single-slot capture + multi-graph handles (opaque MeshTraceId*). // Mirrors the CUDA backend's single-exec_ vs EndCaptureGraph split. @@ -7526,6 +7627,10 @@ void TraceEndCapture() { VT_CHECK(s.capturing, "tenstorrent: TraceEndCapture without Begin"); MeshDevice& device = SharedMeshDevice(); ttnn::operations::trace::end_trace_capture(&device, s.capturing_id, kTraceCq); + // W4a wave-3a: expose the device-reported live trace demand so the + // chunked E=1 arm's fit inside the 50 MiB trace region is a measurement, + // not an assumption (the wave-1b falsification class). + LastTraceBytes() = static_cast(device.get_trace_buffers_size()); // Drop previous single-slot replay if any. if (s.has_replay) { try { @@ -7559,6 +7664,7 @@ void* TraceEndCaptureGraph() { VT_CHECK(s.capturing, "tenstorrent: TraceEndCaptureGraph without Begin"); MeshDevice& device = SharedMeshDevice(); ttnn::operations::trace::end_trace_capture(&device, s.capturing_id, kTraceCq); + LastTraceBytes() = static_cast(device.get_trace_buffers_size()); NoteGraphCaptured(); s.capturing = false; tt_capture_active() = false; diff --git a/tests/vt/test_tenstorrent_backend.cpp b/tests/vt/test_tenstorrent_backend.cpp index 836d9f69a..e235d9809 100644 --- a/tests/vt/test_tenstorrent_backend.cpp +++ b/tests/vt/test_tenstorrent_backend.cpp @@ -5930,6 +5930,14 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped matches the CPU grouped provider i Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); Queue q = backend.CreateQueue(); vt::Queue qcpu{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + // W4a wave-3a: force the E=1 arms through MANY chunks per call so the + // envelope ratios below measure the chunked slice-decode (policy default + // chunks would cover these small N in one pass). Reset before every + // return path — the guard covers the whole case body. + vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(3); + struct ChunkReset { + ~ChunkReset() { vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(0); } + } chunk_reset; auto widen = [](uint16_t u) { uint32_t bits = static_cast(u) << 16; float f; @@ -5969,6 +5977,8 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped matches the CPU grouped provider i }; const Shape shapes[] = { {3, 8, 1, 1, -1, vt::DType::kQ4_K, "E=1 dense arm, ids all zero"}, + {2, 8, 1, 1, 1, vt::DType::kQ4_K, + "E=1 broadcast [1,K] act, P=2 (wave-3a replication)"}, {4, 8, 2, 4, -1, vt::DType::kQ4_K, "E=N expert tower arm, permuted ids"}, {1, 16, 1, 2, -1, vt::DType::kQ4_K, "P=1"}, {4, 33, 1, 4, 1, vt::DType::kQ4_K, "broadcast [1,K] act, non-tile N=33"}, @@ -6261,3 +6271,475 @@ TEST_CASE("kTENSTORRENT kMatmulBTQuantGrouped matches the CPU grouped provider i } } } + +// --------------------------------------------------------------------------- +// KEEPQUANT W4a wave-3a (#3030): the E=1 (dense) grouped arm becomes +// capture-compatible and memory-bounded. Three legs: +// 1. THE TRACE-BOUND CASE (red-first, the wave-1b falsification class): +// a head-shaped [248320, 1024] Q6_K E=1 op CAPTURED whole — before the +// chunked slice-decode the per-call whole-[N,K] decode persists as a +// bf16 tile inside the captured graph and end_trace_capture demands +// 425,754,624 B-class trace demand against the 52,428,800 B region +// (TT_FATAL, mesh_trace.cpp:81). After: bounded chunk tiles, capture +// succeeds, and the device-reported demand is recorded. +// 2. Capture-clean ids: E=1 ids are statically all zero (the only +// in-range expert), so the op must not EnsureHost/read them; the +// E=N tower arm keeps today's range check. +// 3. Bit-exactness: the chunked decode itself must match BlockToFloat — +// decode math unchanged, only the loop bounds move. +// --------------------------------------------------------------------------- + +TEST_CASE("kTENSTORRENT E=1 grouped keep-quant capture survives the 50 MiB trace region") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); + REQUIRE(backend.SupportsGraphCapture()); + REQUIRE(vt::OpRegistered(vt::OpId::kMatmulBTQuantGrouped, + vt::DeviceType::kTENSTORRENT)); + Queue q = backend.CreateQueue(); + // The production chunk policy (no override): this leg measures it. The env + // knob exists so one build can measure the trace-demand curve across chunk + // counts (the wave-3a trace-region survey); it is a test-only lever. + int64_t chunk_rows_override = 0; + if (const char* env_rows = std::getenv("VT_KEEPQUANT_TEST_CHUNK_ROWS")) { + const long long parsed = std::atoll(env_rows); + if (parsed > 0) chunk_rows_override = static_cast(parsed); + } + vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(chunk_rows_override); + MESSAGE("chunk rows override: " << chunk_rows_override); + + struct ChunkReset { + ~ChunkReset() { vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(0); } + } chunk_reset; + + // The 0.8B tied head: [248320, 1024] Q6_K. Decoded once as one tile that + // is 254,274,560 elems ≈ 485 MiB of bf16 — ~9× the 52,428,800 B trace + // region, and the f32 chain planes are ~2× that again. + constexpr int64_t kN = 248320, kK = 1024, kP = 1, kPa = 1; + const int64_t kElems = vt::BlockElems(vt::DType::kQ6_K); // 256 + const int64_t kBB = vt::BlockBytes(vt::DType::kQ6_K); // 210 + const int64_t kNb = kK / kElems; // 4 + + std::mt19937 rng(20260914u); + std::vector packed(static_cast(kN) * kNb * kBB); + for (size_t b = 0; b < packed.size() / static_cast(kBB); ++b) { + uint8_t* blk = packed.data() + b * kBB; + for (int i = 0; i < 208; ++i) blk[i] = static_cast(rng() & 0xFF); + const uint16_t d_bits = + vt::F32ToF16(0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); + std::memcpy(blk + 208, &d_bits, sizeof(d_bits)); + } + std::vector a_f32(static_cast(kPa * kK)); + for (auto& v : a_f32) v = (static_cast(rng() % 401) - 200.0f) / 100.0f; + std::vector a_bf(a_f32.size()); + for (size_t i = 0; i < a_f32.size(); ++i) a_bf[i] = vt::F32ToBF16(a_f32[i]); + std::vector ids(kP, 0); + + // ---- the CPU grouped provider on the IDENTICAL bytes + the W2 bound ---- + std::vector cpu_out(static_cast(kP * kN), 0.0f); + std::vector bound(static_cast(kP * kN), 0.0f); + { + vt::Queue qcpu{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + Tensor at = Tensor::Contiguous(a_bf.data(), vt::DType::kBF16, qcpu.device, + {kPa, kK}); + Tensor ot = Tensor::Contiguous(cpu_out.data(), vt::DType::kF32, qcpu.device, + {kP, kN}); + Tensor it = Tensor::Contiguous(ids.data(), vt::DType::kI32, qcpu.device, {kP}); + Tensor wt = Tensor::Contiguous(packed.data(), vt::DType::kF32, qcpu.device, + {kN, kK}); + wt.dtype = vt::DType::kQ6_K; // block dtype: elementwise strides are inert + vt::MatmulBTQuantGrouped(qcpu, ot, at, wt, it); + + auto widen = [](uint16_t u) { + uint32_t bits = static_cast(u) << 16; + float f; + std::memcpy(&f, &bits, 4); + return f; + }; + std::vector w_f32(static_cast(kN) * kK); + vt::cpu::BlockToFloat(vt::DType::kQ6_K)(packed.data(), w_f32.data(), + kN * kK); + std::vector w_bf(w_f32.size()); + for (size_t i = 0; i < w_f32.size(); ++i) w_bf[i] = vt::F32ToBF16(w_f32[i]); + for (int64_t n = 0; n < kN; ++n) { + float acc = 0.0f, mag = 0.0f; + for (int64_t k = 0; k < kK; ++k) { + const float prod = + widen(a_bf[static_cast(k)]) * + widen(w_bf[static_cast(n * kK + k)]); + acc += prod; + mag += std::fabs(prod); + } + bound[static_cast(n)] = + 1.05f * std::ldexp(1.0f, -8) * (mag + std::fabs(acc)); + } + } + + // ---- the device grouped call, warmed eagerly ---- + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(static_cast(kP * kN) * sizeof(float)); + void* mem_i = backend.Alloc(ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w, packed.data(), packed.size()); + backend.Copy(q, mem_i, ids.data(), ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kPa, kK}); + Tensor w_t = Tensor::Contiguous(mem_w, vt::DType::kQ6_K, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kN, kK}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kP, kN}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + std::vector eager(static_cast(kP * kN), 0.0f); + backend.Copy(q, eager.data(), mem_o, eager.size() * sizeof(float)); + + float worst_cpu = 0.0f; + for (int64_t n = 0; n < kN; ++n) { + const float d = + std::fabs(eager[static_cast(n)] - cpu_out[static_cast(n)]); + worst_cpu = std::max(worst_cpu, d / bound[static_cast(n)]); + CHECK(std::isfinite(eager[static_cast(n)])); + CHECK_MESSAGE(d <= bound[static_cast(n)], + "head-shape envelope: n=" << n << " tt=" + << eager[static_cast(n)] + << " cpu=" + << cpu_out[static_cast(n)] + << " bound=" + << bound[static_cast(n)]); + } + MESSAGE("head-shape [", kN, ",", kK, "] Q6_K E=1: worst bound-ratio vs cpu=", + worst_cpu); + + // ---- capture ×2 byte-identity (the #2907 discipline) ---- + std::vector dumps[2]; + int64_t demand[2] = {0, 0}; + for (int pass = 0; pass < 2; ++pass) { + vt::tenstorrent::ResetKeepQuantCaptureStagingWritesForTest(); + void* graph = nullptr; + std::string what; + bool threw = false; + try { + backend.BeginCapture(q); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + graph = backend.EndCaptureGraph(q); + } catch (const std::exception& ex) { + threw = true; + what = ex.what(); + } + REQUIRE_MESSAGE(!threw, "capture pass " << pass + << " threw (the wave-1b trace " + "region fatal lives here): " + << what); + REQUIRE(graph != nullptr); + demand[pass] = vt::tenstorrent::LastTraceBytesForTest(); + backend.ReplayGraph(q, graph); + dumps[pass].resize(static_cast(kP * kN), 0.0f); + backend.Copy(q, dumps[pass].data(), mem_o, + dumps[pass].size() * sizeof(float)); + backend.DestroyGraph(graph); // after the blocking readback above + MESSAGE("capture pass ", pass, ": device trace demand ", demand[pass], + " B (region 52428800 B), staging writes during capture=", + vt::tenstorrent::KeepQuantCaptureStagingWrites()); + CHECK_MESSAGE(vt::tenstorrent::KeepQuantCaptureStagingWrites() == 0, + "E=1 chunked capture staged ", + vt::tenstorrent::KeepQuantCaptureStagingWrites(), + " word uploads DURING capture (the #2812 class)"); + } + for (int pass = 0; pass < 2; ++pass) { + REQUIRE_MESSAGE(demand[pass] <= 52428800, + "capture pass " << pass << " demanded " << demand[pass] + << " B of trace region against 52428800 B"); + CHECK(std::memcmp(dumps[static_cast(pass)].data(), eager.data(), + eager.size() * sizeof(float)) == 0); + } + CHECK(std::memcmp(dumps[1].data(), dumps[0].data(), + eager.size() * sizeof(float)) == 0); + MESSAGE("capture x2 byte-identity: PASS; trace demand pass0=", demand[0], + " B pass1=", demand[1], " B (region 52428800 B)"); + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); +} + +TEST_CASE("kTENSTORRENT E=1 grouped keep-quant never reads the routing ids; E=N still range-checks") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); + REQUIRE(vt::OpRegistered(vt::OpId::kMatmulBTQuantGrouped, + vt::DeviceType::kTENSTORRENT)); + REQUIRE(backend.SupportsGraphCapture()); + Queue q = backend.CreateQueue(); + vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(0); + struct ChunkReset { + ~ChunkReset() { vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(0); } + } chunk_reset; + + // Q4_K, E=1 (dense [N,K]), P=3, Pa=1 broadcast. The ids bytes are GARBAGE + // (out of range for E=1): the E=1 arm must treat the routing ids as + // statically all zero — the only in-range expert — so it neither + // EnsureHosts nor reads them, which is exactly what makes the arm + // capture-clean. Red-first: with the W2 host readback the eager call + // below CHECK-fails on "expert id out of range". + constexpr int64_t kN = 8, kK = 256, kP = 3, kPa = 1; + const int64_t kBB = vt::BlockBytes(vt::DType::kQ4_K); + std::mt19937 rng(20260915u); + std::vector packed(kN * kBB); + for (int64_t b = 0; b < kN; ++b) { + uint8_t* blk = packed.data() + b * kBB; + const uint16_t d_bits = vt::F32ToF16(0.05f + 0.35f * static_cast(rng() % 64) / 64.0f); + const uint16_t dmin_bits = vt::F32ToF16(0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); + std::memcpy(blk, &d_bits, sizeof(d_bits)); + std::memcpy(blk + 2, &dmin_bits, sizeof(dmin_bits)); + for (int i = 4; i < kBB; ++i) blk[i] = static_cast(rng() & 0xFF); + } + std::vector a_f32(static_cast(kPa * kK)); + for (auto& v : a_f32) v = (static_cast(rng() % 401) - 200.0f) / 100.0f; + std::vector a_bf(a_f32.size()); + for (size_t i = 0; i < a_f32.size(); ++i) a_bf[i] = vt::F32ToBF16(a_f32[i]); + std::vector garbage_ids{7, -5, 999}; // inert for E=1 + + // The oracle: the CPU grouped provider fed ids {0,0,0} (the semantics the + // E=1 arm must implement), inside the W2 analytic bf16-operand bound. + std::vector cpu_out(static_cast(kP * kN), 0.0f); + { + vt::Queue qcpu{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + Tensor at = Tensor::Contiguous(a_bf.data(), vt::DType::kBF16, qcpu.device, + {kPa, kK}); + Tensor ot = Tensor::Contiguous(cpu_out.data(), vt::DType::kF32, qcpu.device, + {kP, kN}); + std::vector zero_ids(kP, 0); + Tensor it = Tensor::Contiguous(zero_ids.data(), vt::DType::kI32, + qcpu.device, {kP}); + Tensor wt = Tensor::Contiguous(packed.data(), vt::DType::kF32, qcpu.device, + {kN, kK}); + wt.dtype = vt::DType::kQ4_K; + vt::MatmulBTQuantGrouped(qcpu, ot, at, wt, it); + } + auto widen = [](uint16_t u) { + uint32_t bits = static_cast(u) << 16; + float f; + std::memcpy(&f, &bits, 4); + return f; + }; + std::vector w_f32(static_cast(kN) * kK); + vt::cpu::BlockToFloat(vt::DType::kQ4_K)(packed.data(), w_f32.data(), kN * kK); + std::vector w_bf(w_f32.size()); + for (size_t i = 0; i < w_f32.size(); ++i) w_bf[i] = vt::F32ToBF16(w_f32[i]); + std::vector bound(static_cast(kP * kN), 0.0f); + for (int64_t n = 0; n < kN; ++n) { + float acc = 0.0f, mag = 0.0f; + for (int64_t k = 0; k < kK; ++k) { + const float prod = widen(a_bf[static_cast(k)]) * + widen(w_bf[static_cast(n * kK + k)]); + acc += prod; + mag += std::fabs(prod); + } + for (int64_t p = 0; p < kP; ++p) + bound[static_cast(p * kN + n)] = + 1.05f * std::ldexp(1.0f, -8) * (mag + std::fabs(acc)); + } + + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(static_cast(kP * kN) * sizeof(float)); + void* mem_i = backend.Alloc(garbage_ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w, packed.data(), packed.size()); + backend.Copy(q, mem_i, garbage_ids.data(), garbage_ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kPa, kK}); + Tensor w_t = Tensor::Contiguous(mem_w, vt::DType::kQ4_K, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kN, kK}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kP, kN}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); // must not read the ids + std::vector eager(static_cast(kP * kN), 0.0f); + backend.Copy(q, eager.data(), mem_o, eager.size() * sizeof(float)); + for (int64_t i = 0; i < kP * kN; ++i) { + const float d = std::fabs(eager[static_cast(i)] - + cpu_out[static_cast(i)]); + CHECK_MESSAGE(d <= bound[static_cast(i)], + "E=1 ids-inert output diverges at " << i << ": tt=" + << eager[static_cast(i)] + << " cpu=" + << cpu_out[static_cast(i)]); + } + + // The same op CAPTURED: no readback may exist to run inside the region. + { + backend.BeginCapture(q); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + backend.EndCapture(q); + backend.Replay(q); + std::vector after(static_cast(kP * kN), 0.0f); + backend.Copy(q, after.data(), mem_o, after.size() * sizeof(float)); + CHECK(std::memcmp(after.data(), eager.data(), eager.size() * sizeof(float)) == 0); + } + + // NO WEAKENING on the tower arm: E=N ids are dynamic and stay range- + // checked on the host (its capture indirection is a later wave). + { + constexpr int64_t kE2 = 4; + std::vector tower(kE2 * kN * kBB); + for (size_t b = 0; b < tower.size() / static_cast(kBB); ++b) { + uint8_t* blk = tower.data() + b * kBB; + const uint16_t d_bits = vt::F32ToF16(0.05f); + const uint16_t dmin_bits = vt::F32ToF16(0.005f); + std::memcpy(blk, &d_bits, sizeof(d_bits)); + std::memcpy(blk + 2, &dmin_bits, sizeof(dmin_bits)); + for (int i = 4; i < kBB; ++i) blk[i] = static_cast(rng() & 0xFF); + } + void* mem_w2 = backend.Alloc(tower.size()); + backend.Copy(q, mem_w2, tower.data(), tower.size()); + Tensor w2 = Tensor::Contiguous(mem_w2, vt::DType::kQ4_K, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kE2 * kN, kK}); + std::vector bad_ids{99, 0, 0}; + void* mem_i2 = backend.Alloc(bad_ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_i2, bad_ids.data(), bad_ids.size() * sizeof(int32_t)); + Tensor i2 = Tensor::Contiguous(mem_i2, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + bool threw = false; + std::string what; + try { + vt::MatmulBTQuantGrouped(q, o_t, a_t, w2, i2); + } catch (const std::exception& ex) { + threw = true; + what = ex.what(); + } + CHECK_MESSAGE(threw, "E=N out-of-range id must still be refused"); + CHECK_MESSAGE(what.find("out of range") != std::string::npos, + "the refusal must name the range check, got: ", what); + backend.Free(mem_w2); + backend.Free(mem_i2); + } + + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); +} + +TEST_CASE("kTENSTORRENT E=1 grouped keep-quant chunked slice-decode is bit-exact vs BlockToFloat") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); + REQUIRE(vt::OpRegistered(vt::OpId::kMatmulBTQuantGrouped, + vt::DeviceType::kTENSTORRENT)); + Queue q = backend.CreateQueue(); + + // One-hot activations on a bf16-exact dequant grid, the wave-2 legs' + // construction at E=1, with the chunk rows FORCED below the N of the case + // so every call executes MANY chunks. Chunks cover DISJOINT weight rows, + // so each output element is one chunk's dot over the full K and the + // expected bits are exactly bf16(BlockToFloat(element)) — a chunk offset + // or loop-bound defect lands on a different grid value and cannot hide. + // (Pre-chunking this case is trivially green — the whole slice is one + // "chunk"; it pins the chunked loop bounds that replace it.) + auto one_hot_leg = [&](vt::DType enc, int64_t chunk_rows, int64_t nb) { + vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(chunk_rows); + struct ChunkReset { + ~ChunkReset() { vt::tenstorrent::KeepQuantChunkRowsOverrideForTest(0); } + } chunk_reset; + constexpr int64_t kN = 8, kP = 5; + const int64_t elems = vt::BlockElems(enc); + const int64_t bb = vt::BlockBytes(enc); + const int64_t kK = nb * elems; + const int64_t chunks = (kN + chunk_rows - 1) / chunk_rows; + std::mt19937 rng(static_cast(20260916u + nb)); + std::vector packed(kN * nb * bb); + const uint16_t d_bits = vt::F32ToF16(std::ldexp(1.0f, -6)); + for (int64_t b = 0; b < kN * nb; ++b) { + uint8_t* blk = packed.data() + b * bb; + if (enc == vt::DType::kQ8_0) { + std::memcpy(blk, &d_bits, sizeof(d_bits)); + for (int i = 0; i < 32; ++i) { + blk[2 + i] = static_cast(rng() % 255 - 127); + } + } else { + // Q6_K: ql[128] @0, qh[64] @128, i8 scales[16] @192, d f16 @208 — + // positive scales keep the dequant in the -0-free class. + for (int i = 0; i < 128; ++i) blk[i] = static_cast(rng() & 0xFF); + for (int i = 128; i < 192; ++i) blk[i] = static_cast(rng() & 0xFF); + for (int i = 192; i < 208; ++i) + blk[i] = static_cast(1 + rng() % 127); + std::memcpy(blk + 208, &d_bits, sizeof(d_bits)); + } + } + std::vector a_bf(static_cast(kP * kK), 0u); + for (int64_t p = 0; p < kP; ++p) + a_bf[static_cast(p * kK + (p % kK))] = vt::F32ToBF16(1.0f); + std::vector ids(kP, 0); // E=1: statically all zero + + // The decode oracle: BlockToFloat over the IDENTICAL bytes, expert 0. + std::vector w_f32(static_cast(kN) * kK); + vt::cpu::BlockToFloat(enc)(packed.data(), w_f32.data(), kN * kK); + std::vector dequant(static_cast(kP * kN)); + for (int64_t p = 0; p < kP; ++p) + for (int64_t n = 0; n < kN; ++n) + dequant[static_cast(p * kN + n)] = + w_f32[static_cast(n * kK + (p % kK))]; + + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_w = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(static_cast(kP * kN) * sizeof(float)); + void* mem_i = backend.Alloc(ids.size() * sizeof(int32_t)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w, packed.data(), packed.size()); + backend.Copy(q, mem_i, ids.data(), ids.size() * sizeof(int32_t)); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kP, kK}); + Tensor w_t = Tensor::Contiguous(mem_w, enc, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kN, kK}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, + {kP, kN}); + Tensor i_t = Tensor::Contiguous(mem_i, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kP}); + vt::MatmulBTQuantGrouped(q, o_t, a_t, w_t, i_t); + std::vector tt_out(static_cast(kP * kN), 0.0f); + backend.Copy(q, tt_out.data(), mem_o, tt_out.size() * sizeof(float)); + backend.Free(mem_a); + backend.Free(mem_w); + backend.Free(mem_o); + backend.Free(mem_i); + for (int64_t i = 0; i < kP * kN; ++i) { + const uint16_t got_bf = vt::F32ToBF16(tt_out[static_cast(i)]); + const uint16_t want_bf = + vt::F32ToBF16(dequant[static_cast(i)]); + CHECK_MESSAGE(got_bf == want_bf, + "chunked slice-decode bit-exact (" << vt::Name(enc) + << ", chunk=" + << chunk_rows + << ", chunks=" + << chunks + << "): i=" << i + << " tt=" << tt_out[static_cast(i)] + << " dequant=" + << dequant[static_cast(i)]); + } + MESSAGE("chunked bit leg ", vt::Name(enc), " chunk=", chunk_rows, + " chunks=", chunks, ": bits match BlockToFloat"); + }; + one_hot_leg(vt::DType::kQ8_0, 2, 2); // K=64, 4 chunks of 2 rows + one_hot_leg(vt::DType::kQ6_K, 3, 1); // K=256, 3 chunks (last of 2) +} From 4c67a394b00c8eaf61a6af325a97193ab75bea30 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Mon, 7 Sep 2026 07:25:55 +0200 Subject: [PATCH 08/10] =?UTF-8?q?record(BACKEND-TENSTORRENT-KEEPQUANT):=20?= =?UTF-8?q?wave-3b-1=20capture=20demand=20is=20per=20chain=20=E2=80=94=20a?= =?UTF-8?q?dopt=20the=20upstream=20trace-region=20policy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The wave-3b-1 vehicle AFTER leg falsified the chunked E=1 arm's capture-safety at model scale: after the two device-level defects it exposed were fixed red-first (a chained ROW_MAJOR activation reaching ttnn::matmul unconverted, per_core_M = 0 at matmul_program_config.cpp:372; and the E=1 arm committing ROW_MAJOR into the output slot so replay's tile-padded view overran the buffer, mesh_tensor_impl.hpp:33 — focused 4/4, suite 66/66, 524,428/524,428 after both), capture still demanded 444,424,192 B of the 52,428,800 B region (mesh_trace.cpp:81). The demand is ~113 keep-quant weights x ~3.5 MB/chain of serialized decode commands — the wave-3a per-chain constant — so chunk sizing cannot pay it down; the cost multiplies by chain count. This reconciles the third amendment's rejected "trace-region enlargement": that rejection sized wave-1b's whole-weight-per-step decode planes (~18 GB); the measured quantity here is the command stream (0.44 GB on 0.8B, to be measured on 27B). The fifth amendment moves the region policy onto the pinned tt-metal's own practice (models/demos/utils/trace_region_sizes.py): dynamic (trace_region_size=0, the upstream default for unconfigured models and deepseek-v3's explicit choice) tried first on the vehicle; a per-model resolved region sized from measured demand is the fallback if the pre-ITEM-5 overlap hazard reproduces, with the fixed 50 MB (the vLLM plugin's generic value, ITEM 5) as that fallback's unspecified-model default. Capture demand stays a measured, reported axis; the perf guard records tokens/s both sides of the switch, and a gate that cannot complete inside the vehicle timeout is a NEEDS_DECISION stop. The twin-absence policy is unchanged. Spec-only record commit; the wave-3b-1 implementation diff stays uncommitted for wave-3b-1c to complete under the amended spec. Gate note: the record-anchor gate on the full worktree reports 29 > 28 solely from the pending wave-3b-1 diff's tenstorrent_ops.cpp shift (the GDN kL2Norm anchor at 7517-7531); this commit's content alone verifies clean at HEAD in a detached worktree (agent record OK), and the anchor repair rides the wave-3b-1 landing commit per its granted ride-along. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/specs/tenstorrent-keepquant.md | 40 +++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/.agents/specs/tenstorrent-keepquant.md b/.agents/specs/tenstorrent-keepquant.md index ddb42da8e..5a0f2c06f 100644 --- a/.agents/specs/tenstorrent-keepquant.md +++ b/.agents/specs/tenstorrent-keepquant.md @@ -349,6 +349,39 @@ throughput is W4b's lever. The E=N expert arm serves the family's MoE models (30B-A3B class) and stays staged-owed: no MoE artifact is on disk and a ~17-20 GB download needs authority. +**AMENDED 2026-09-07 (fifth): the vehicle capture falsified the chunked +arm's capture claim at model scale — demand is per CHAIN, not per chunk.** +Wave-3b-1's vehicle AFTER leg fataled at capture with 444,424,192 B +demanded of the 52,428,800 B region (mesh_trace.cpp:81) after the two +device-level defects it exposed were fixed red-first (a chained ROW_MAJOR +activation reaching `ttnn::matmul` unconverted — `per_core_M = 0` at +`matmul_program_config.cpp:372`; and the E=1 arm committing ROW_MAJOR into +the output slot, so replay's tile-padded view overran the buffer, +`mesh_tensor_impl.hpp:33`; focused 4/4, suite 66/66, 524,428/524,428 after +both fixes). The demand is ~113 keep-quant weights × ~3.5 MB/chain of +serialized decode commands — the wave-3a per-chain constant — so chunk +sizing cannot pay it down; the cost multiplies by chain count. This +reconciles the third amendment's rejected "trace-region enlargement": that +rejection sized wave-1b's whole-weight-per-step decode planes (~18 GB); +the measured quantity here is the command stream (0.44 GB on 0.8B, to be +measured on 27B). RESOLUTION: the region policy mirrors the pinned +tt-metal's own practice (`models/demos/utils/trace_region_sizes.py`): +DYNAMIC (`trace_region_size=0`, the upstream default for unconfigured +models and deepseek-v3's explicit choice) is tried first on the vehicle; +if the pre-ITEM-5 overlap hazard reproduces, the fallback is a per-model +resolved region sized from measured demand (the upstream YAML shape), +documented in `docs/USAGE.md` beside the demand number, with the fixed +50 MB (the vLLM plugin's generic value, ITEM 5) as that fallback's +unspecified-model default. Capture demand stays a MEASURED, reported axis +(the gate message carries `LastTraceBytesForTest()`) — the axis is the +number, not the carve-out. Perf guard: the vehicle leg records tokens/s on +both sides of the switch ("replay re-decodes each step: correct, slower" +is already the accepted state; W4b owes the lever); a gate that cannot +complete inside the vehicle timeout is a NEEDS_DECISION stop, not an +accepted default. The twin-absence policy is unchanged: no whole-weight +resident decoded shadow on the dense path; decode planes stay per-chain +transients. + **W4b — the int8-dot lever ([#3031](https://github.com/mudler/vllm.cpp/issues/3031)).** Quantized-domain integer vec_dot behind the same seam; profile-first attribution; recorded-only throughput floor. Sequenced after W4a, never @@ -386,4 +419,9 @@ passed; pushed). AMENDED 2026-09-07 (fourth): the 27B is dense the 27B path is the E=1 arm with chunked slice-decode (capture holds chunks, never whole weights); the E=N expert arm stays staged-owed behind a MoE artifact. Wave-3a = E=1 chunked capture-compatible packed -dense; wave-3b = 27B wiring + gate + MTP skip + USAGE pin. +dense; wave-3b = 27B wiring + gate + MTP skip + USAGE pin. AMENDED +2026-09-07 (fifth): the vehicle falsified capture-safety at model scale +(444,424,192 B = ~113 chains × ~3.5 MB of decode command stream); the +trace-region policy moves to the pinned tt-metal's dynamic / per-model +practice, demand stays measured and reported, and wave-3b-1c repairs +under it. From f46d102836322ecd0cccdca2ca9cbfb60e3577df Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Mon, 7 Sep 2026 11:12:40 +0200 Subject: [PATCH 09/10] feat(BACKEND-TENSTORRENT-KEEPQUANT): dense keep-quant serves packed through the chunked E=1 arm (#3030) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dense keep-quant matmul IS the wave-3a chunked E=1 grouped arm: the packed i32 words stage once per weight (EnsureKeepQuantWords) and each call slice-decodes and accumulates in capture-safe chunks, so the decoded bf16 TWIN of wave-2/W3 is gone from the matmul path. Reached through vt::MatmulBT's public block-weight dispatch (ops.cpp:163) on all four registered encodings, the entry a GGUF load actually takes. The gather class keeps its embed-table twin on purpose: the vehicle's tied head shares the GGUF tensor with the embedding, so its gather keeps the twin while its matmul stages only the packed words. Two capture-path defects surfaced red-first under the switch and are fixed in the same change: the chained activation entered the grouped arm ROW_MAJOR where the arm tiles, and the output TILE committed before CommitDeviceLogical2D replicated it across the mesh (red logs /tmp/w4a3b1-red.log, /tmp/w4a3b1-red2.log). The device now opens with a DYNAMIC trace region (trace_region_size=0), mirroring the pinned tt-metal's own practice (models/demos/utils/trace_region_sizes.py), with VT_TT_TRACE_REGION_MB opting back into a fixed one (50 MB fallback on an unusable value) when a capture overlaps live DRAM. The old fixed 50 MB sized the region as a constant, and the chunked arm's captured command stream alone measures 444,424,192 B on the 0.8B vehicle (mesh_trace.cpp:81) — a per-chain cost no constant tracks. The vehicle ran the dynamic policy. The switch moves six prompt anchors (51 of 256 anchor cells re-derive), the headline being prompt[2] tok=5: 11 -> 13. Mechanism: the packed chunked head matmul resolves a different tt-metal program config than the twin whole-slice matmul over the same [248320,1024] Q4_K_M weight (per_core_N=19 with out_subblock 2x2 vs per_core_N=71), a different dst accumulation order, so last-ulp logit deltas reach band-edge cells. Evidence: the teacher-forced gap golden shows 0 forward-divergent cells across all 16 prompts and max gap 375 mnats (band 500), and the flipped cell is a 0.0-mnat exact tie under oracle logits with the third candidate 1.375 nats behind. Goldens re-derived through the documented path: VT_DUMP_IDS=1 capture plus the dequantized-oracle recipe, the fresh safetensors dump byte-identical to the committed-era artifact. New red-first residency tests: the warm-matmul pair flips the word/twin probes on all four encodings, and the gather-survivor case guards the embed twin (delete the embed map's insert and it stays red). Cost, accepted for the residency policy: the last-112-step replay window runs 2.95 s/cycle on the twin vs 18.02 s/cycle packed, ~6.1x — the replay re-decodes chunks from packed words instead of matmul-ing a resident bf16 tensor. Full backend suite 66/66 cases, 524,428 assertions. Vehicle battery 147/147 assertions, 16/16 prompts: 10 strict + 6 near-tie (11/5 before), max gap 0.375 nats @ prompt[9] tok=4, 0 forward-divergent. Anchors moved: .agents/backend-matrix.md's GDN registration anchor (tenstorrent_ops.cpp 7517-7531 -> 7495-7509, check-agent-record back at baseline 28) and scripts/env-doc-allowlist.txt, which gains VT_TT_TRACE_REGION_MB. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/backend-matrix.md | 2 +- scripts/env-doc-allowlist.txt | 1 + src/vt/tenstorrent/tenstorrent_device.cpp | 42 ++- src/vt/tenstorrent/tenstorrent_device.h | 19 ++ src/vt/tenstorrent/tenstorrent_ops.cpp | 207 ++++++------ .../neartie_gap_mnats_tenstorrent_capture.npy | Bin 1152 -> 1152 bytes .../our_ids_tenstorrent_capture.i32 | Bin 1024 -> 1024 bytes .../our_ids_tenstorrent_capture.npy | Bin 1152 -> 1152 bytes tests/parity/test_qwen35_paged_engine.cpp | 10 + tests/vt/test_tenstorrent_backend.cpp | 306 ++++++++++++++++++ 10 files changed, 477 insertions(+), 110 deletions(-) diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 25a645a67..adc5cfed9 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -246,7 +246,7 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | | `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | **Gate PASSED on Blackhole P150 (2026-08-12):** [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap **0.062 nats**, BACKEND PROOF 0 declines (kMatmul selections=256 = untied lm_head on device, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded, same as Qwen3-0.6B TT precedent). Exit SIGSEGV 139 is the known MeshDevice teardown crash, not a gate failure | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | -| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7517-7531](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7517-L7531) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | +| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7495-7509](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7495-L7509) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | | `BACKEND-TENSTORRENT-GDN-DEVICE-PURE` | Child of `BACKEND-TENSTORRENT-GDN` — make the decode-side GDN ops (`CausalConv1dUpdateKernel`, `GdnDecodeKernel`) device-resident so a tt-metal trace capture admits them, unblocking the Qwen3.5-0.8B captured arm, the last capture-blocked family with a committed eager pair (#2907, owed from #2812) | Move the per-call host orchestration on-device: resident inputs from the graph's producer ops, device-side indexed state update (baked slot addressing re-primed by the recapture cadence, or `kGdnStateGather`/`kGdnStateScatter` indirection — decided on recapture-cost measurement), conv two-views coherent on device, token readback outside the captured span | [CausalConv1dUpdateKernel :5013](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5013), [GdnDecodeKernel :5225](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5225) (EnsureHost ×5, ReadIdxHost, UploadTensor ×5+), q35 harness pair selection [test_qwen35_paged_engine.cpp:233](../tests/parity/test_qwen35_paged_engine.cpp#L233), fatal repro `fd_mesh_command_queue.cpp:760` 2/2 | Red-first: the #2812 loud-skip opt-in cells run captured; captured dump ×2 byte-identity with reset between; teacher-forced pair vs the transformers oracle inside the eager band; Qwen3.5 joins `DecodeCaptureDefaultArch`, ambient adjudicates CAPTURED, env=0 eager; tamper + arch-deletion mutations red; gate arm selection [test_qwen35_paged_engine.cpp:301-330](../tests/parity/test_qwen35_paged_engine.cpp#L301-L330); closure [parity-ledger.md:947](parity-ledger.md#L947) | [tenstorrent-gdn-device-pure.md](specs/tenstorrent-gdn-device-pure.md) | `DONE` | `bcade48d6` | | `BACKEND-TENSTORRENT-KEEPQUANT` | Child of `BACKEND-TENSTORRENT` — keep-quant dense dot on TT-Metal ([#2959](https://github.com/mudler/vllm.cpp/issues/2959)): GGUF k-quant arms refuse on the P150 until this row lands — W1 (#2989, open) lands the Q4_K decode, W2 (branch) lands the dot and the `kTENSTORRENT` predicate arm (`gguf_keep_quant.cpp:136-148`) admitting exactly the registered set `{Q4_K, Q5_K, Q6_K, Q8_0}` after wave-2b — while the smallest Qwen3.8 artifact that fits the card is the dense 27B Q4_K_M (17.1 GB; bf16 is 53.8 GB) and a quantized Qwen3.5 (0.8B, ~0.5 GB) is the test vehicle | ggml k-quant block formats via our reader, pinned bit-exact vs llama.cpp `b10451` (#2240/#2260 lineage); no vLLM mirror for TT (secondary substrate, deviation by design); substrate surveyed 2026-09-05: tt-metal has NO packed-weight matmul (`ttnn` quantization is per-tensor activation dquant only), so the kernel is ours — resident blocks, on-core decode to bf16 tiles through f32 (decode bit-exact vs `BlockToFloat`), the dot reusing the `kMatmulBT` tile path, device-bf16 band at the residual-golden boundary, the int8 dot as the named next lever | W1 LANDED (#2989): the Q4_K block-decode device path. W2a/2b LANDED (5e43d770, 14e8fe471): the grouped provider `kMatmulBTQuantGrouped` — E=1 dense, E=N stacked tower, registered set exactly {Q4_K, Q5_K, Q6_K, Q8_0}. W3a ON BRANCH: the E=1 arm is CHUNKED slice-decode + f32 assembly — capture-safe on the 0.8B head (trace demand 13,484,032 B of the 52,428,800 B region; capture ×2 byte-identity; routing ids never read; staging writes during capture 0; both capture-time failure modes measured and bounded by the CHUNK POLICY); E=N capture indirection staged-owed behind a MoE artifact (spec ## W4). OWED: the int8 lever, the 27B arm + model wiring (wave-3b), the captured e2e vehicle leg | W1 LANDED (#2989): decode bit-exact vs `vt::cpu::BlockToFloat` across a shape sweep (red-first, op-level suite pattern). W2a/2b LANDED: grouped sweep green across the four encodings (E=1 vs cpu 0.279–0.720, vs analytic-ref 0.201–0.514). W3a ON BRANCH (red-first): the head-shape capture leg reds at base in the wave-1b class (whole-tensor device DRAM fatal, bank_manager.cpp:462) and greens with the CHUNK POLICY (trace demand 13,484,032 B, staging writes 0, ×2 byte-identity, deterministic across passes); the chunked-decode bit-exact leg pins decode math unchanged (81 assertions); the sweep under forced 3-row chunks is bit-identical to the whole-slice ratios; full backend suite 63/63 cases, 524,267 assertions; vehicle 16/16, 147/147, 0 forward-divergent. OWED: the captured e2e vehicle leg | [tenstorrent-keepquant.md](specs/tenstorrent-keepquant.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-KEEPQUANT` — implementer on the row branch; spec commit on `row/BACKEND-TENSTORRENT-KEEPQUANT`, claim in [.agents/claims/](claims/CLAIM-BACKEND-TENSTORRENT-KEEPQUANT.md) | | `BACKEND-TENSTORRENT-QWEN35` | Child of `BACKEND-TENSTORRENT` — the wiring row: `Qwen3_5ForConditionalGeneration` (dense text GDN hybrid) on the TT allow-list, the op delta the family forward refuses by name (`kGdnPostConv`, `kSigmoidGateBf16`, `kAttnQkNormRopeGate`, `kAttnQkNormRope` — pinned empirically by a W0 refusal sweep), and the first e2e gate. Makes the GDN row's ops production-reached (#1715 stays open until the family runs; GDN row lifecycle moves in the same change this lands) | Substrate: the GDN row's 8 kernels + the 27-op TT registry; e2e mirrors the ratified Mistral TT golden treatment (`VT_DUMP_IDS` bootstrap → `scripts/qwen3-neartie-gap-transformers.py` teacher-forced near-tie gaps → committed device-golden pair, POL-ORACLE deviation); op-level oracle is the CPU f32 arm. Capacity: `Qwen/Qwen3.5-0.8B` bf16 ≈1.6 GB fits (proven envelope Mistral-7B ≈14.5 GB); 27B bf16 ≈53.8 GB and GGUF k-quant arms (no TT kernels) refused by name, owed | `src/vllm/platforms/tenstorrent.cpp` (allow-list, lands last) + `src/vt/tenstorrent/tenstorrent_ops.cpp` (op delta) + `tests/parity/test_qwen35_paged_engine.cpp` (TT arm) | Spec committed 2026-08-23; W0 sweep, W1 op delta, W2 e2e, W3 GDN-reviewer leftovers (d2h counter completeness, `conv_transposed` fast-path check) owed | [tenstorrent-qwen35.md](specs/tenstorrent-qwen35.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-QWEN35` | diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index ff438c7ad..6c65a292b 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -194,6 +194,7 @@ VT_TT_NO_ROPE_REFRESH VT_TT_RECAPTURE_EVERY VT_TT_SDPA_TEST VT_TT_SHARD_Q +VT_TT_TRACE_REGION_MB VT_V4_DECODE_GRAPH VT_V4_DEVICE_ATTN VT_V4_DEVICE_GLUE diff --git a/src/vt/tenstorrent/tenstorrent_device.cpp b/src/vt/tenstorrent/tenstorrent_device.cpp index 7a090f1c9..7a93c263f 100644 --- a/src/vt/tenstorrent/tenstorrent_device.cpp +++ b/src/vt/tenstorrent/tenstorrent_device.cpp @@ -1,6 +1,8 @@ // vllm.cpp original (BACKEND-TENSTORRENT W0); no upstream mirror. #include "vt/tenstorrent/tenstorrent_device.h" +#include +#include #include #include @@ -10,6 +12,22 @@ namespace vt::tenstorrent { bool DeviceAvailable() { return tt::tt_metal::GetNumAvailableDevices() > 0; } +namespace { +// ITEM 5 trace-region policy: 0 (dynamic) unless VT_TT_TRACE_REGION_MB opts +// into a fixed region; an unusable value falls back to 50 MB, the vLLM +// plugin's generic fixed size. +constexpr size_t kFallbackTraceRegionBytes = 50 * 1024 * 1024; + +size_t TraceRegionSizeFromEnv() { + const char* mb = std::getenv("VT_TT_TRACE_REGION_MB"); + if (mb == nullptr || *mb == '\0') return 0; // DYNAMIC + char* end = nullptr; + const long parsed = std::strtol(mb, &end, 10); + if (end == mb || *end != '\0' || parsed <= 0) return kFallbackTraceRegionBytes; + return static_cast(parsed) * 1024 * 1024; +} +} // namespace + MeshDevice& SharedMeshDevice() { // DELIBERATE LEAK, not an oversight: a plain `static std::shared_ptr` // reproducibly SEGFAULTS on process exit (tests/vt/test_tenstorrent_backend.cpp, @@ -27,15 +45,27 @@ MeshDevice& SharedMeshDevice() { // OS/kernel driver reclaim the PCIe device's file descriptors and hardware // state on process exit regardless of a userspace close() call, the same // assumption CUDA processes routinely rely on), if not textbook-clean. - // ITEM 5: reserve a dedicated DRAM trace region (50 MB, the tt-metal vLLM - // plugin's value — worker.py:710) so allocations during trace capture can - // never overlap the trace buffer. With the default (0), the trace buffer - // is carved from the general pool and ANY capture-time allocation fatals - // with "Trace buffer ... overlaps with DRAM activity". + // ITEM 5 (amended, wave-3b-1c): open the device with a DYNAMIC trace + // region (`trace_region_size=0`), mirroring the pinned tt-metal's own + // practice (models/demos/utils/trace_region_sizes.py): unconfigured + // (model, SKU) pairs resolve to TRACE_REGION_SIZE_DYNAMIC (0) — the + // runtime then carves trace buffers from the general DRAM pool at + // capture time — and deepseek-v3 sets 0 explicitly. Our previous fixed + // 50 MB was the vLLM plugin's generic value (its worker.py:710), and it + // sized the region as a constant rather than from demand: the chunked + // E=1 keep-quant arm's captured command stream alone measures ~0.44 GB + // on the 0.8B vehicle (444,424,192 B vs the 52,428,800 B region, + // mesh_trace.cpp:81), a per-chain cost no fixed constant tracks. The old + // reason for a fixed region was the 2026-08-09 capture fatal "Trace + // buffer ... overlaps with DRAM activity" on the then-current runtime; + // that hazard still exists in dynamic mode (mesh_trace.cpp:113 detects + // it), so VT_TT_TRACE_REGION_MB= opts back into a fixed region — + // the fallback's default is 50, the ITEM 5 generic value — when a + // capture overlaps live DRAM allocations. static std::shared_ptr* device = new std::shared_ptr( ttnn::open_mesh_device( /*device_id=*/0, /*l1_small_size=*/DEFAULT_L1_SMALL_SIZE, - /*trace_region_size=*/50 * 1024 * 1024)); + /*trace_region_size=*/TraceRegionSizeFromEnv())); return **device; } diff --git a/src/vt/tenstorrent/tenstorrent_device.h b/src/vt/tenstorrent/tenstorrent_device.h index ea4731759..85f19e5ea 100644 --- a/src/vt/tenstorrent/tenstorrent_device.h +++ b/src/vt/tenstorrent/tenstorrent_device.h @@ -189,11 +189,30 @@ void ResetKeepQuantCaptureStagingWritesForTest(); // cover the whole [N, K] slice in one pass. int64_t LastTraceBytesForTest(); void KeepQuantChunkRowsOverrideForTest(int64_t rows); + +// BACKEND-TENSTORRENT-KEEPQUANT W4a wave-3b-1 residency-policy probes, one +// per shadow map, so the twin policy is a measurement and not a claim: +// - KeepQuantWordShadowPresentForTest: the PACKED i32 word shadow for a host +// weight is staged. The dense keep-quant matmul's residency — the chunked +// E=1 arm decodes from it every call. +// - DecodedWeightShadowPresentForTest: a decoded bf16 TWIN for a host weight +// exists. The dense keep-quant matmul must NOT build one — an entry here +// for a matmul weight is the wave-2 twin residency surviving the switch. +// - EmbedTableShadowPresentForTest: the embedding gather's bf16 table twin +// exists — the gather-class survivor the twin policy deliberately keeps +// (the vehicle's tied head shares the table: its GATHER keeps the twin, +// its MATMUL stages only the packed words). +bool KeepQuantWordShadowPresentForTest(const void* host); +bool DecodedWeightShadowPresentForTest(const void* host); +bool EmbedTableShadowPresentForTest(const void* host); #else inline int64_t KeepQuantCaptureStagingWrites() { return 0; } inline void ResetKeepQuantCaptureStagingWritesForTest() {} inline int64_t LastTraceBytesForTest() { return 0; } inline void KeepQuantChunkRowsOverrideForTest(int64_t) {} +inline bool KeepQuantWordShadowPresentForTest(const void*) { return false; } +inline bool DecodedWeightShadowPresentForTest(const void*) { return false; } +inline bool EmbedTableShadowPresentForTest(const void*) { return false; } #endif // ITEM 5 (rope): driver-side warm hook — populate the persistent device diff --git a/src/vt/tenstorrent/tenstorrent_ops.cpp b/src/vt/tenstorrent/tenstorrent_ops.cpp index a2274d2dc..76e0aa755 100644 --- a/src/vt/tenstorrent/tenstorrent_ops.cpp +++ b/src/vt/tenstorrent/tenstorrent_ops.cpp @@ -2385,61 +2385,31 @@ void KeepQuantDecodeKernel(Queue&, Tensor& out, const Tensor& packed) { static_cast(nb * elems)); } -// kMatmulBTQuant (KEEPQUANT W2, generalized W3): `a` is [M,K] float, `b` is -// [N,K] packed keep-quant blocks — out = a @ b^T, reached through vt::MatmulBT's -// block-weight dispatch (ops.cpp:163), the entry every model matmul helper -// already uses. tt-metal has no packed-weight GEMM primitive (row survey), so -// the composition decodes the blocks through the bit-exact f32 chain -// (DecodeKeepQuantBlocksF32 above), rounds the decoded weight ONCE to bf16 -// (RNE, the device's round-once convention), and runs the same bf16 tile -// matmul as kMatmulBT. The activation takes the device's ordinary bf16 tile -// path (an f32 master rounds once on device, the same widen-on-load rule as -// bf16 weights). Numerics sit inside the analytic bf16 operand-rounding -// envelope of the decode-based reference — the tests pin it, not a picked -// tolerance. W3 residency: the decode runs on-core from the per-weight -// resident i32 word shadow (EnsureKeepQuantWords) — the eager pre-capture -// step stages every weight once and the captured replay hits it. -// The decoded bf16 twin of a keep-quant weight, memoized per host weight -// pointer. MatmulBTQuantKernel's first draft decoded EVERY weight on EVERY -// call; the Qwen3.5-0.8B Q4_K_M vehicle's tied head is [248320, 1024] Q6_K, -// so every step re-ran a quarter-billion-element unpack whose f32 planes + -// where intermediates (~1 GB each) churned and fragmented the DRAM banks -// until a large contiguous allocation failed (TT_FATAL Out of Memory during -// the first generate). Weights are immutable after load — the same invariant -// WeightViewShadow documents — so the twin is built once and reused. The -// recycled-address hazard is handled the same way too: a collision must also -// match rows x cols AND the encoding to hit. Capture-safe by construction: -// the eager warm step builds every twin before capture, so the captured graph -// reads one stable tensor per weight and stages nothing per replay. -struct DecodedWeightShadow { - std::optional device; - uint32_t rows = 0, cols = 0; - DType enc = DType::kF32; -}; -std::mutex& DecodedWeightMutex() { - static std::mutex m; - return m; -} -std::map& DecodedWeightShadows() { - static std::map* m = - new std::map(); // never destroyed (#1486) - return *m; -} -// Free path hook: a freed host weight must drop its twin, so a recycled -// address can never alias a stale decode (UnregisterHostBuffer). -void DropDecodedWeightShadow(void* host) { - if (host == nullptr) return; - std::lock_guard g(DecodedWeightMutex()); - DecodedWeightShadows().erase(reinterpret_cast(host)); -} - -void MatmulBTQuantKernel(Queue&, Tensor& out, const Tensor& a, const Tensor& b) { +// kMatmulBTQuant (KEEPQUANT W2, generalized W3, switched W4a wave-3b-1): +// `a` is [M,K] float, `b` is [N,K] packed keep-quant blocks — out = a @ b^T, +// reached through vt::MatmulBT's block-weight dispatch (ops.cpp:163), the +// entry every model matmul helper already uses. W4a wave-3b-1 (#3030): the +// dense keep-quant matmul IS the wave-3a chunked E=1 grouped arm — the +// production dense path consumes PACKED words. The PACKED words stage once +// per weight (EnsureKeepQuantWords) and each call slice-decodes + accumulates +// in capture-safe chunks (256 MiB plane policy, see the E=1 arm); the decoded +// bf16 TWIN of wave-2/W3 is GONE from this path. The fourth spec amendment +// makes this the row's production surface: the whole keep-quant set beyond +// the gather class is served PACKED, and a captured graph must never hold a +// whole-weight tile. The gather class keeps its own embed-table twin +// (EnsureEmbedTableDevice); the vehicle's tied head shares the GGUF tensor +// with the embedding — its GATHER keeps that twin, its MATMUL stages only the +// packed words. Non-keep-quant (bf16/f32) weights never enter this kernel +// (vt::MatmulBT routes them to kMatmulBT), and non-TT devices are untouched. +// Exactly the encodings DeviceKeepQuantSupported admits on kTENSTORRENT +// (gguf_keep_quant.cpp). Refusing here BY NAME keeps an admitted-but- +// unimplemented encoding from reaching the device. +void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, + const Tensor& weight, const Tensor& expert_ids); +void MatmulBTQuantKernel(Queue& q, Tensor& out, const Tensor& a, const Tensor& b) { TT_OP_TRACE("MatmulBTQuant"); VT_CHECK(a.rank == 2 && b.rank == 2 && out.rank == 2, "tenstorrent kMatmulBTQuant: rank-2 a/b/out required"); - // Exactly the encodings DeviceKeepQuantSupported admits on kTENSTORRENT - // (gguf_keep_quant.cpp). Refusing here BY NAME keeps an admitted-but- - // unimplemented encoding from reaching the device. const DType enc = b.dtype; // vt::Name() emits the lowercase storage name ("q4_0"); the refusal must // name the ENUM the caller passed, so the k-prefix and capital go on here. @@ -2469,57 +2439,46 @@ void MatmulBTQuantKernel(Queue&, Tensor& out, const Tensor& a, const Tensor& b) VT_CHECK(a.IsContiguous() && b.IsContiguous() && out.IsContiguous(), "tenstorrent kMatmulBTQuant: strided tensors are not supported in W2"); - MeshDevice& device = SharedMeshDevice(); - // Memoized twin: decode once, reuse across steps (see DecodedWeightShadow). - bool have_twin = false; - ttnn::Tensor w_bf16; - { - std::lock_guard g(DecodedWeightMutex()); - auto it = DecodedWeightShadows().find(reinterpret_cast(b.data)); - if (it != DecodedWeightShadows().end() && it->second.rows == N && - it->second.cols == K && it->second.enc == enc && it->second.device) { - w_bf16 = *it->second.device; - have_twin = true; - } - } - if (!have_twin) { - // Build the twin HOST-side. The first draft built it on device - // (DecodeKeepQuantBlocksF32 + typecast); for the vehicle's [248320,1024] - // Q6_K tied head that materialized ~1 GB f32 planes plus where - // intermediates whose transient churn fragmented the DRAM banks until a - // 4 GB allocation failed (TT_FATAL Out of Memory) — the FIRST decode of - // the head did not fit alongside residency even once, so the twin cache - // alone could not save the run. The host decoder is the SAME chain the - // kKeepQuantDecode test pins bit-exact (vt::cpu::BlockToFloat == - // DecodeKeepQuantBlocksF32), so the f32 bits are unchanged; only where - // they are produced moves. The bf16 round then happens once, host-side, - // with the same RNE convention (vt::F32ToBF16) the device typecast - // applied — pre-rounding to bf16-representable f32 makes the upload - // independent of from_vector's own conversion mode — and from_vector - // lands it as a TILE bf16 tensor, the identical upload the embedding - // table and UploadRows already use. Decoding host-side also frees the - // device from the per-call word-shadow reads for this one-time build. - std::vector w_f32(static_cast(N) * K); - vt::cpu::BlockToFloat(enc)(b.data, w_f32.data(), static_cast(N) * K); - for (size_t i = 0; i < w_f32.size(); ++i) - w_f32[i] = BF16ToF32(F32ToBF16(w_f32[i])); - w_bf16 = ttnn::Tensor::from_vector(w_f32, TileSpecOf(N, K), &device); - std::lock_guard g(DecodedWeightMutex()); - DecodedWeightShadow& s = DecodedWeightShadows()[reinterpret_cast( - b.data)]; - s.device = w_bf16; - s.rows = N; - s.cols = K; - s.enc = enc; - } - ttnn::Tensor dev_a = EnsureDevice2D(a, device); - if (a.dtype == DType::kF32) - dev_a = ttnn::to_layout( - ttnn::typecast(std::move(dev_a), ttnn::DataType::BFLOAT16), - ttnn::Layout::TILE); - ttnn::Tensor dev_c = ttnn::operations::matmul::matmul( - dev_a, w_bf16, /*transpose_a=*/false, /*transpose_b=*/true); - CommitDevice2D(out, std::move(dev_c)); + // W4a wave-3b-1 (#3030): the dispatch. P = M output rows against expert 0; + // the ids are statically all zero and the E=1 arm never reads them (the + // capture contract proven by the garbage-ids leg), so a host zeros tensor + // is all the grouped contract needs. Capture-safe by construction: the + // eager warm step stages every word shadow before capture (a capture-time + // miss refuses inside EnsureKeepQuantWords) and every chunk offset is a + // capture-time constant replayed verbatim. + std::vector zero_ids(static_cast(a.shape[0]), 0); + const Tensor ids = Tensor::Contiguous(zero_ids.data(), DType::kI32, + Device{DeviceType::kCPU, 0}, + {a.shape[0]}); + MatmulBTQuantGroupedKernel(q, out, a, b, ids); +} + +// The decoded bf16 twin map, retained after the wave-3b-1 switch as the +// TWIN-ABSENCE probe's surface and the recycled-address hygiene hook: the +// dense keep-quant matmul no longer inserts (it serves packed words through +// the E=1 grouped arm), so a populated entry for a matmul weight is exactly +// the regression the probe names. Same collision discipline as before: the +// free path drops by host pointer (UnregisterHostBuffer). +struct DecodedWeightShadow { + std::optional device; + uint32_t rows = 0, cols = 0; + DType enc = DType::kF32; +}; +std::mutex& DecodedWeightMutex() { + static std::mutex m; + return m; +} +std::map& DecodedWeightShadows() { + static std::map* m = + new std::map(); // never destroyed (#1486) + return *m; +} +// Free path hook: a freed host weight must drop its twin, so a recycled +// address can never alias a stale decode (UnregisterHostBuffer). +void DropDecodedWeightShadow(void* host) { + if (host == nullptr) return; + std::lock_guard g(DecodedWeightMutex()); + DecodedWeightShadows().erase(reinterpret_cast(host)); } // kMatmulBTQuantGrouped (KEEPQUANT W4a wave-2, #3030): out[P,N], act[Pa,K] @@ -2621,6 +2580,16 @@ void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, dev_a = ttnn::to_layout( ttnn::typecast(std::move(dev_a), ttnn::DataType::BFLOAT16), ttnn::Layout::TILE); + else if (dev_a.layout() != ttnn::Layout::TILE) + // W4a wave-3b-1 (#3030): a bf16 activation can arrive ROW_MAJOR — the + // exact-shape slot hit returns the layout its producer committed (the + // vehicle's silu-mul output feeding the down projection). A ROW_MAJOR + // operand keeps its unpadded logical shape, so ttnn's auto program + // config divides a sub-tile M by the 32-tile height and fatals + // (per_core_M == 0, matmul_program_config.cpp get_mcast_1d_config). + // Convert once, before the chunk loop; zero tile padding adds only + // discarded zero output rows. + dev_a = ttnn::to_layout(std::move(dev_a), ttnn::Layout::TILE); ttnn::Tensor a_rows; if (E > 1 && Pa > 1) a_rows = ttnn::to_layout(std::move(dev_a), ttnn::Layout::ROW_MAJOR); @@ -2695,6 +2664,15 @@ void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, if (out.dtype == DType::kBF16) assembled = ttnn::typecast(std::move(assembled), ttnn::DataType::BFLOAT16); + // Commit form: TILE — the layout the twin path's matmul output carried. + // The ROW_MAJOR assembly above is an internal concat domain only. A + // ROW_MAJOR commit leaves a ROW_MAJOR slot for the next consumer, and + // downstream consumers build tile-padded views over that slot's buffer + // (the vehicle's rope -> paged-KV RAC reshape view exceeded the buffer: + // mesh_tensor_impl.hpp packed-size fatal on the replay step). Values are + // unchanged; only the committed slot's layout lands as the twin's did. + if (assembled.layout() != ttnn::Layout::TILE) + assembled = ttnn::to_layout(std::move(assembled), ttnn::Layout::TILE); CommitDeviceLogical2D(out, std::move(assembled), static_cast(P), static_cast(N)); return; @@ -7562,6 +7540,29 @@ int64_t LastTraceBytesForTest() { return LastTraceBytes().load(std::memory_order_relaxed); } +// W4a wave-3b-1 residency-policy probes — the contract lives in +// tenstorrent_device.h. Each reads its map under its own mutex; a nullptr or +// absent host reads false. Defined OUTSIDE the anonymous namespace (the +// wave-3a hook pattern): a -Werror=unused-function TU would reject an +// anonymous-namespace helper no other file-local code calls. +bool KeepQuantWordShadowPresentForTest(const void* host) { + if (host == nullptr) return false; + std::lock_guard g(KeepQuantWordMutex()); + return KeepQuantWordShadows().find(host) != KeepQuantWordShadows().end(); +} +bool DecodedWeightShadowPresentForTest(const void* host) { + if (host == nullptr) return false; + std::lock_guard g(DecodedWeightMutex()); + return DecodedWeightShadows().find(reinterpret_cast(host)) != + DecodedWeightShadows().end(); +} +bool EmbedTableShadowPresentForTest(const void* host) { + if (host == nullptr) return false; + std::lock_guard g(EmbedTableMutex()); + return EmbedTableShadows().find(reinterpret_cast(host)) != + EmbedTableShadows().end(); +} + // ---- ttnn mesh-trace capture (Backend graph-capture mapping) ---------------- // Process-local single-slot capture + multi-graph handles (opaque MeshTraceId*). // Mirrors the CUDA backend's single-exec_ vs EndCaptureGraph split. diff --git a/tests/parity/goldens/qwen35_gguf_q4km/neartie_gap_mnats_tenstorrent_capture.npy b/tests/parity/goldens/qwen35_gguf_q4km/neartie_gap_mnats_tenstorrent_capture.npy index 1a3f69c0efe16c343b755540e1a681d9c7fdfb5b..245f5fe1034aaf482e61e2d7f35552b734c9620f 100644 GIT binary patch delta 46 ucmZqRY~b9W!N~Y)Vq^Sd2Ns8k6J&st1cWI75^iQHXWaaeF`jYa1b+Z^a}Z(x delta 36 pcmZqRY~b9W!8mb3!sI`U0TU*t3)w71H(HYEzLYR zi78J{m7RgX3djXnG(7}jprj^H8fYj;OotUBzb=CjXz=8JOotpKq2?<9X^>q)l?)8W XfV3i%pC}07D?$1GY!E)s8({SSyCNeF diff --git a/tests/parity/goldens/qwen35_gguf_q4km/our_ids_tenstorrent_capture.npy b/tests/parity/goldens/qwen35_gguf_q4km/our_ids_tenstorrent_capture.npy index d66e4e3e67d22a4026224b6bbf47a0e1660c4919..662868b4a1d158b7fda7ce76e84fee41a1e0ee08 100644 GIT binary patch delta 253 zcmZqRY~b7=!l=c|z`$?}i1V2k7{nPF7-Xdx7`TCa836_cDWD`sOpKj@L7#E59pgSm z{>gtC{S84HegHK*;9+1W1S$Y&aAAS4K^lIsF)&B~u^EsprNh85dvYyPu#hw}MC3Cc z1H(I@46FL&M@)Hoy;2a3dL|4EFxMCY4V>o1z`zK!q>U3|L9qxk!v@vKb&4;1{t#K7=?hk>CG$mRjEiDx!z+eUBf^3=|0x?ih6DSQd6eOm@3Xxxz!3c6i9rGavNvQb>KpJG1P$dJy YF(9o7 K spans a single block + std::mt19937 rng(20260921u); + auto rand_byte = [&rng]() { return static_cast(rng() & 0xFF); }; + auto put_f16 = [](uint8_t* blk, int64_t off, float v) { + const uint16_t bits = vt::F32ToF16(v); + std::memcpy(blk + off, &bits, sizeof(bits)); + }; + + const vt::DType encodings[] = {vt::DType::kQ4_K, vt::DType::kQ5_K, + vt::DType::kQ6_K, vt::DType::kQ8_0}; + for (const vt::DType enc : encodings) { + const int64_t bb = vt::BlockBytes(enc); + const int64_t be = vt::BlockElems(enc); + REQUIRE(be * bb > 0); + const int64_t K = kNb * be; + + // Deterministic packed blocks (the W3 sweep generator): PRNG bytes + // everywhere, then finite f16 scales (d, and dmin where the encoding has + // one) placed at the layout's offsets, so the decode never sees NaN/Inf. + std::vector packed(kN * kNb * bb); + for (int64_t b = 0; b < kN * kNb; ++b) { + uint8_t* blk = packed.data() + b * bb; + for (int i = 0; i < bb; ++i) blk[i] = rand_byte(); + const float d = (0.05f + 0.35f * static_cast(rng() % 64) / 64.0f) * + ((rng() % 2) != 0 ? 1.0f : -1.0f); + if (enc == vt::DType::kQ6_K) { + put_f16(blk, 208, d); + } else { + put_f16(blk, 0, d); + if (enc == vt::DType::kQ4_K || enc == vt::DType::kQ5_K) + put_f16(blk, 2, 0.005f + 0.02f * static_cast(rng() % 32) / 32.0f); + } + } + std::vector a_bf(K); + for (auto& v : a_bf) + v = vt::F32ToBF16((static_cast(rng() % 401) - 200.0f) / 100.0f); + + // Envelope oracle (the W3 dot's shape, M=1): decode -> ONE bf16 RNE -> + // f32 ascending accumulation; 1.05 * 2^-8 bf16-operand bound. + std::vector w_f32(kN * K); + vt::cpu::BlockToFloat(enc)(packed.data(), w_f32.data(), kN * K); + auto widen = [](uint16_t u) { + uint32_t bits = static_cast(u) << 16; + float f; + std::memcpy(&f, &bits, 4); + return f; + }; + std::vector ref(kN), bound(kN); + for (int64_t n = 0; n < kN; ++n) { + float acc = 0.0f, mag = 0.0f; + for (int64_t k = 0; k < K; ++k) { + const float p = widen(a_bf[static_cast(k)]) * + widen(vt::F32ToBF16(w_f32[static_cast(n) * K + k])); + acc += p; + mag += std::fabs(p); + } + ref[static_cast(n)] = acc; + bound[static_cast(n)] = 1.05f * std::ldexp(1.0f, -8) * (mag + std::fabs(acc)); + } + + void* mem_a = backend.Alloc(a_bf.size() * sizeof(uint16_t)); + void* mem_b = backend.Alloc(packed.size()); + void* mem_o = backend.Alloc(kN * sizeof(float)); + backend.Copy(q, mem_a, a_bf.data(), a_bf.size() * sizeof(uint16_t)); + backend.Copy(q, mem_b, packed.data(), packed.size()); + Tensor a_t = Tensor::Contiguous(mem_a, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {1, K}); + Tensor b_t = Tensor::Contiguous(mem_b, enc, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kN, K}); + Tensor o_t = Tensor::Contiguous(mem_o, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {1, kN}); + + // The warm pair: an eager warm matmul + a second call. A twin, if the + // policy still built one, exists after call one; the second call proves + // no lazy build either. Then the residency probes read the policy. + vt::MatmulBT(q, o_t, a_t, b_t); + vt::MatmulBT(q, o_t, a_t, b_t); + + CHECK_FALSE_MESSAGE( + vt::tenstorrent::DecodedWeightShadowPresentForTest(b_t.data), + "TWIN-ABSENCE (" << vt::Name(enc) << "): the dense keep-quant matmul " + "built a decoded bf16 twin — the wave-2 twin " + "residency survived the wave-3b-1 switch"); + REQUIRE_MESSAGE( + vt::tenstorrent::KeepQuantWordShadowPresentForTest(b_t.data), + "PACKED-WORDS (" << vt::Name(enc) << "): the dense keep-quant matmul " + "staged no resident i32 word shadow — the chunked " + "E=1 arm's residency did not engage"); + + std::vector out(kN, 0.0f); + backend.Copy(q, out.data(), mem_o, out.size() * sizeof(float)); + backend.Free(mem_a); + backend.Free(mem_b); + backend.Free(mem_o); + for (int64_t n = 0; n < kN; ++n) { + const float diff = std::fabs(out[static_cast(n)] - + ref[static_cast(n)]); + CHECK(std::isfinite(out[static_cast(n)])); + CHECK_MESSAGE(diff <= bound[static_cast(n)], + vt::Name(enc) << " out[" << n <<"]=" << out[n] + << " ref=" << ref[n] + << " bound=" << bound[n]); + } + MESSAGE("dense keep-quant residency switch ", vt::Name(enc), + ": twin absent, words staged, envelope ok"); + } +} + +// W4a wave-3b-1 (#3030) survivor leg: the GATHER class keeps its twin. The +// embedding table's bf16 device twin (EnsureEmbedTableDevice -> the +// EmbedTableShadows map) is the residency the twin policy deliberately keeps; +// the switch must not have over-removed it. Mutation guard for the same +// commit's switch: delete the embed map's insert and this stays red. +TEST_CASE("kTENSTORRENT embedding gather twin survives the wave-3b-1 switch (gather class)") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + REQUIRE(vt::OpRegistered(vt::OpId::kEmbedding, vt::DeviceType::kTENSTORRENT)); + + Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); + Queue q = backend.CreateQueue(); + + constexpr int64_t Vocab = 17, H = 24, T = 3; + std::vector host_table(Vocab * H); + for (size_t i = 0; i < host_table.size(); ++i) + host_table[i] = static_cast(i % 13) * 0.1f - 0.5f; + const std::vector host_ids = {0, 16, 3}; + + std::vector host_out(T * H, 0.0f); + void* mem_table = backend.Alloc(host_table.size() * sizeof(float)); + void* mem_ids = backend.Alloc(host_ids.size() * sizeof(int32_t)); + void* mem_out = backend.Alloc(host_out.size() * sizeof(float)); + backend.Copy(q, mem_table, host_table.data(), host_table.size() * sizeof(float)); + backend.Copy(q, mem_ids, host_ids.data(), host_ids.size() * sizeof(int32_t)); + + Tensor table = Tensor::Contiguous(mem_table, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {Vocab, H}); + Tensor ids = Tensor::Contiguous(mem_ids, vt::DType::kI32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {T}); + Tensor out = Tensor::Contiguous(mem_out, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {T, H}); + + auto embedding = reinterpret_cast( + vt::GetOp(vt::OpId::kEmbedding, vt::DeviceType::kTENSTORRENT)); + embedding(q, out, table, ids); + embedding(q, out, table, ids); // warm + second call, the twin pair + + REQUIRE_MESSAGE( + vt::tenstorrent::EmbedTableShadowPresentForTest(table.data), + "GATHER-SURVIVOR: the embedding table's device twin is gone after a " + "warm gather pair — the switch over-removed the gather class"); + REQUIRE_FALSE_MESSAGE( + vt::tenstorrent::DecodedWeightShadowPresentForTest(table.data), + "the gather must serve from the embed-table map, never from the " + "matmul twin map"); + + backend.Free(mem_table); + backend.Free(mem_ids); + backend.Free(mem_out); + MESSAGE("embedding gather twin survivor leg: embed twin present, matmul twin map clean"); +} + +// W4a wave-3b-1 (#3030) decode-shape leg: the ROW_MAJOR chained activation. +// The vehicle's MLP down projection consumes the silu-mul output, whose slot +// carries a ROW_MAJOR device staging (CommitDeviceLogical2D binds whatever +// layout the chain produced), and EnsureDevice2D's exact-shape hit hands that +// ROW_MAJOR tensor straight to the matmul. A ROW_MAJOR [M<32, K] operand +// drives ttnn's auto program config to per_core_M = M / 32 == 0 +// (matmul_program_config.cpp get_mcast_1d_config) — the TT_FATAL the vehicle +// AFTER leg hit on the first non-tile-aligned decode shape. RED-first: the +// chain below fatals while the E=1 arm passes the activation through +// untouched; the arm's TILE conversion makes it green and pins the fix. +// Chained through public calls only: an E=1 grouped matmul produces the +// bf16 activation slot state, then vt::MatmulBT consumes it. +TEST_CASE("kTENSTORRENT dense keep-quant matmul converts a ROW_MAJOR chained activation to TILE (decode shape)") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + REQUIRE(vt::OpRegistered(vt::OpId::kMatmulBTQuant, vt::DeviceType::kTENSTORRENT)); + + Backend& backend = vt::GetBackend(vt::DeviceType::kTENSTORRENT); + Queue q = backend.CreateQueue(); + + constexpr int64_t kM = 5; // the vehicle's decode M: < 32, non-tile-aligned + constexpr int64_t kK = 256; // one Q6_K block per row + std::mt19937 rng(20260922u); + auto rand_byte = [&rng]() { return static_cast(rng() & 0xFF); }; + auto put_f16 = [](uint8_t* blk, int64_t off, float v) { + const uint16_t bits = vt::F32ToF16(v); + std::memcpy(blk + off, &bits, sizeof(bits)); + }; + auto pack_q6 = [&rand_byte, &put_f16, &rng](int64_t rows) { + std::vector packed(rows * vt::BlockBytes(vt::DType::kQ6_K)); + for (int64_t b = 0; b < rows; ++b) { + uint8_t* blk = packed.data() + b * vt::BlockBytes(vt::DType::kQ6_K); + for (int i = 0; i < vt::BlockBytes(vt::DType::kQ6_K); ++i) blk[i] = rand_byte(); + const float d = (0.05f + 0.35f * static_cast(rng() % 64) / 64.0f) * + ((rng() % 2) != 0 ? 1.0f : -1.0f); + put_f16(blk, 208, d); + } + return packed; + }; + auto widen = [](uint16_t u) { + uint32_t bits = static_cast(u) << 16; + float f; + std::memcpy(&f, &bits, 4); + return f; + }; + + // Producer: [kM, kK] bf16 activation against a [kK, kK] Q6_K weight. The + // E=1 arm's assembly commits ROW_MAJOR f32 -> typecast bf16, so out1's slot + // ends ROW_MAJOR — the state the vehicle's elementwise chain leaves. + const std::vector w1 = pack_q6(kK); + std::vector a1(kM * kK); + for (auto& v : a1) + v = vt::F32ToBF16((static_cast(rng() % 401) - 200.0f) / 100.0f); + void* mem_a1 = backend.Alloc(a1.size() * sizeof(uint16_t)); + void* mem_w1 = backend.Alloc(w1.size()); + void* mem_o1 = backend.Alloc(static_cast(kM * kK) * sizeof(uint16_t)); + backend.Copy(q, mem_a1, a1.data(), a1.size() * sizeof(uint16_t)); + backend.Copy(q, mem_w1, w1.data(), w1.size()); + Tensor a1_t = Tensor::Contiguous(mem_a1, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kM, kK}); + Tensor w1_t = Tensor::Contiguous(mem_w1, vt::DType::kQ6_K, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kK, kK}); + Tensor o1_t = Tensor::Contiguous(mem_o1, vt::DType::kBF16, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kM, kK}); + vt::MatmulBT(q, o1_t, a1_t, w1_t); + vt::MatmulBT(q, o1_t, a1_t, w1_t); // warm pair; slot state is what matters + + // Consumer: out1 (bf16 [kM, kK], ROW_MAJOR slot) feeds a second keep-quant + // matmul — the vehicle's down-projection shape. Pre-fix this call fatals + // inside ttnn's program-config search. + const std::vector w2 = pack_q6(8); + void* mem_w2 = backend.Alloc(w2.size()); + void* mem_o2 = backend.Alloc(static_cast(kM * 8) * sizeof(float)); + backend.Copy(q, mem_w2, w2.data(), w2.size()); + Tensor w2_t = Tensor::Contiguous(mem_w2, vt::DType::kQ6_K, + Device{vt::DeviceType::kTENSTORRENT, 0}, {8, kK}); + Tensor o2_t = Tensor::Contiguous(mem_o2, vt::DType::kF32, + Device{vt::DeviceType::kTENSTORRENT, 0}, {kM, 8}); + vt::MatmulBT(q, o2_t, o1_t, w2_t); + vt::MatmulBT(q, o2_t, o1_t, w2_t); // second call: no lazy rebuild either + + std::vector o1_bits(kM * kK, 0); + backend.Copy(q, o1_bits.data(), mem_o1, o1_bits.size() * sizeof(uint16_t)); + std::vector out2(kM * 8, 0.0f); + backend.Copy(q, out2.data(), mem_o2, out2.size() * sizeof(float)); + backend.Free(mem_a1); + backend.Free(mem_w1); + backend.Free(mem_o1); + backend.Free(mem_w2); + backend.Free(mem_o2); + + // Envelope oracle for the chained call: the activation operand is EXACTLY + // the bf16 bits call one committed (read back above), one more bf16 RNE on + // the w2 decode, f32 ascending accumulation; 1.05 * 2^-8 bound. + std::vector w2_f32(8 * kK); + vt::cpu::BlockToFloat(vt::DType::kQ6_K)(w2.data(), w2_f32.data(), 8 * kK); + for (int64_t m = 0; m < kM; ++m) { + for (int64_t n = 0; n < 8; ++n) { + float acc = 0.0f, mag = 0.0f; + for (int64_t k = 0; k < kK; ++k) { + const float p = + widen(o1_bits[static_cast(m) * kK + k]) * + widen(vt::F32ToBF16(w2_f32[static_cast(n) * kK + k])); + acc += p; + mag += std::fabs(p); + } + const float bound = 1.05f * std::ldexp(1.0f, -8) * (mag + std::fabs(acc)); + const float got = out2[static_cast(m) * 8 + n]; + CHECK(std::isfinite(got)); + CHECK_MESSAGE(std::fabs(got - acc) <= bound, + "ROW-MAJOR chained activation: m=" << m << " n=" << n + << " got=" << got + << " ref=" << acc + << " bound=" << bound); + } + } + MESSAGE("ROW_MAJOR chained activation leg: TILE conversion engaged, envelope ok"); +} + From 35f393f6d1b9b75b04d7c5b95ae19277ee78d425 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Mon, 7 Sep 2026 16:33:58 +0200 Subject: [PATCH 10/10] =?UTF-8?q?feat(BACKEND-TENSTORRENT-KEEPQUANT):=20wa?= =?UTF-8?q?ve-3b-2=20staged=20slice=20=E2=80=94=20the=20MTP=20head=20skip,?= =?UTF-8?q?=20the=2027B=20denominator=20goldens,=20and=20the=20chunk-plane?= =?UTF-8?q?=20cap=20(#3042)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lands wave-3b-2 as a STAGED SLICE (the coordinator's resolution of the wave's NEEDS_DECISION): the production wiring and its witnesses commit now, and the 27B e2e gate stays UNREACHED, owned by [#3042](https://github.com/mudler/vllm.cpp/issues/3042), listed under `## Owed` in the row spec. Row BACKEND-TENSTORRENT-KEEPQUANT stays ACTIVE; W4b and the 27B gate are the open scope. Landed: - The production MTP drafter skip. The loader's accounting deliberately passes a declared head — its fifteen `blk.64.*` tensors ARE enumerated as expected — and the trunk-only load then leaves them unread, so `LogQwen3_5GgufMtpHeadSkip` prints the skip loud before any weight byte moves: all fifteen tensors, 289,527,808 B, named in full, suppressed only when speculative method `mtp` is configured (src/vllm/entrypoints/model_loader.cpp, src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp). The skip message carries the denominator fact: the pinned llama.cpp `b10451` oracle ignores the same tensors, so a gate against it is matched work only with this skip loud. - The 16-prompt oracle goldens `tests/parity/goldens/qwen38_gguf_q4km_27b/greedy_ids.npy`, derived from the byte-identical llama.cpp `b10451` denominator harness. The TT-side run against them closes the 27B gate. - The reachability test `tests/vllm/entrypoints/test_gguf_accounting_reach.cpp`, which proves the skip through the PRODUCTION loader accounting rather than a hand-built type. - The checkpoint-gated TEST_CASE in `tests/parity/test_qwen35_paged_engine.cpp`, inert with a loud SKIP until `VLLM_CPP_QWEN38_27B_GGUF` names the artifact — the tree's actual env name, read at that test's :588 and documented as such in `docs/USAGE.md`. - `VT_TT_KEEPQUANT_CHUNK_BYTES` (env-doc allowlisted), the keep-quant chunk-plane budget: default 256 MiB surveyed on the 0.8B vehicle, empty/unset keeps the default, a positive integer is a HARD CAP in bytes that trades command-stream length for live memory — the ceil(N/8) trace term otherwise forces a 606+ MB head plane whatever the budget says, exactly the alloc that died at 27B. - The `docs/USAGE.md` 27B arm entry, with the local sha256 pin and the recorded provenance caveat: at the checked unsloth revision no file matches this artifact by size or hash, so the pin rests on the hash, and the exact upstream file and revision are unresolved. UNREACHED, named per the staged-slice contract: the 27B e2e gate. The OOM evidence, eight runs — failing allocations 1,073,725,440 B and 134,184,960 B; free-at-failure 244 MB -> 46 MB -> 12.7 MB/bank; ~34 GB allocated against the 32 GB device with a 3.7 MB largest free block; batch budget 512 fails identically, so the demand is not activation-sized; three mitigations tried and failed. The residency sits ~11 GB above the ~22-23 GB surveyed design residency. Suspects, named and unmeasured: the 2.5 GB bf16 embed twin, f32 plane transients, possible words double-staging. The next lever is the device-side allocation trace, not another mitigation. #3042 owns it. Ride-along anchor repair, authorized by the coordinator: the certified diff moved three record anchors, and the repair rides this landing commit (the f46d10283 precedent) so check-agent-record returns to baseline 28 — engine-matrix.md SPEC-MTP-GGUF (qwen3_5_gguf_weights.cpp:1447 -> :1448) and SPEC-DFLASH-GGUF (:1034 -> :1035), backend-matrix.md BACKEND-TENSTORRENT-GDN (tenstorrent_ops.cpp 7495-7509 -> 7518-7532). Unrelated rows in both keyed records are byte-for-byte equal to their pre-diff state; no other row touched, no new anchor added. Spec amendment: the sixth amendment records the staged-slice landing, the owed gate, and the OOM evidence; the row spec's `## Owed` carries the gate with its owning issue. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki] --- .agents/backend-matrix.md | 2 +- .agents/engine-matrix.md | 4 +- .agents/specs/tenstorrent-keepquant.md | 56 ++++++- docs/USAGE.md | 1 + .../models/qwen3_5_gguf_weights.h | 38 +++++ scripts/env-doc-allowlist.txt | 1 + src/vllm/entrypoints/model_loader.cpp | 13 ++ .../models/qwen3_5_gguf_weights.cpp | 62 +++++++ src/vt/tenstorrent/tenstorrent_ops.cpp | 27 ++- .../qwen38_gguf_q4km_27b/greedy_ids.npy | Bin 0 -> 1152 bytes tests/parity/test_qwen35_paged_engine.cpp | 25 +++ .../test_gguf_accounting_reach.cpp | 156 ++++++++++++++++++ 12 files changed, 378 insertions(+), 7 deletions(-) create mode 100644 tests/parity/goldens/qwen38_gguf_q4km_27b/greedy_ids.npy diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index adc5cfed9..25959ddd0 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -246,7 +246,7 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | | `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | **Gate PASSED on Blackhole P150 (2026-08-12):** [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap **0.062 nats**, BACKEND PROOF 0 declines (kMatmul selections=256 = untied lm_head on device, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded, same as Qwen3-0.6B TT precedent). Exit SIGSEGV 139 is the known MeshDevice teardown crash, not a gate failure | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | -| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7495-7509](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7495-L7509) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | +| `BACKEND-TENSTORRENT-GDN` | Child of `BACKEND-TENSTORRENT` — the GDN linear-attention op chain as native TT kernels, the hard prerequisite for the Qwen3.5/3.8 family (#1715): `kGdnPrefill`, `kGdnDecode`, `kL2Norm`, `kRmsNormGated`, `kCausalConv1dFwd`/`kCausalConv1dUpdate`, `kGdnStateGather`/`kGdnStateScatter`. The P150 is discrete, so an op miss refuses by name — the ops must land before any `Qwen3_5*` arch registration. Correctness oracle is our own CPU f32 arm (residual-golden precedent); no vLLM mirror exists for TT | Substrate: pinned tt-metal `ttnn::transformer::chunk_gated_delta_rule` (FLA chunked GDN forward, on-core recurrent state, `initial_state`/`final_state`) behind a varlen+state-permute adapter for `kGdnPrefill`; decode = rank-1 update composed from ttnn matmul+eltwise with a device shadow keyed by host pointer (`PagedKvShadow` pattern); contracts at `src/vt/ops.cpp:1823-2500`, CPU reference `src/vt/cpu/cpu_ops.cpp:1537-1740` | GDN kernels [GdnPrefillKernel :4220](../src/vt/tenstorrent/tenstorrent_ops.cpp#L4220) + [GdnDecodeKernel :5049](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5049), registered [tenstorrent_ops.cpp:7518-7532](../src/vt/tenstorrent/tenstorrent_ops.cpp#L7518-L7532) (`kL2Norm`..`kGdnStateScatter`); op-level cases vs the CPU f32 oracle (T-sweep, indexed-`state_idx` forms, prefill↔decode state round-trip) | [test_tenstorrent_backend.cpp:1749-3340](../tests/vt/test_tenstorrent_backend.cpp#L1749-L3340): L2Norm, RmsNormGated, CausalConv1dFwd/Update, GdnPrefill, GdnDecode, prefill↔decode round-trip, StateGather/Scatter, edge shapes — every op family vs the CPU f32 oracle. W1 `34fde3502` (prefill set) + W2 `c85af0aaf` (decode+state-I/O set) landed, both fresh-review PASS; production-reached via the `Qwen3_5*` wiring row (BACKEND-TENSTORRENT-QWEN35: allow-list, e2e sacred pair 16/16 STRICT both legs); capture compatibility unmeasured behind #1625; closure [parity-ledger.md#L946](parity-ledger.md#L946) | [tenstorrent-gdn.md](specs/tenstorrent-gdn.md) | `DONE` | `c85af0aaf` | | `BACKEND-TENSTORRENT-GDN-DEVICE-PURE` | Child of `BACKEND-TENSTORRENT-GDN` — make the decode-side GDN ops (`CausalConv1dUpdateKernel`, `GdnDecodeKernel`) device-resident so a tt-metal trace capture admits them, unblocking the Qwen3.5-0.8B captured arm, the last capture-blocked family with a committed eager pair (#2907, owed from #2812) | Move the per-call host orchestration on-device: resident inputs from the graph's producer ops, device-side indexed state update (baked slot addressing re-primed by the recapture cadence, or `kGdnStateGather`/`kGdnStateScatter` indirection — decided on recapture-cost measurement), conv two-views coherent on device, token readback outside the captured span | [CausalConv1dUpdateKernel :5013](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5013), [GdnDecodeKernel :5225](../src/vt/tenstorrent/tenstorrent_ops.cpp#L5225) (EnsureHost ×5, ReadIdxHost, UploadTensor ×5+), q35 harness pair selection [test_qwen35_paged_engine.cpp:233](../tests/parity/test_qwen35_paged_engine.cpp#L233), fatal repro `fd_mesh_command_queue.cpp:760` 2/2 | Red-first: the #2812 loud-skip opt-in cells run captured; captured dump ×2 byte-identity with reset between; teacher-forced pair vs the transformers oracle inside the eager band; Qwen3.5 joins `DecodeCaptureDefaultArch`, ambient adjudicates CAPTURED, env=0 eager; tamper + arch-deletion mutations red; gate arm selection [test_qwen35_paged_engine.cpp:301-330](../tests/parity/test_qwen35_paged_engine.cpp#L301-L330); closure [parity-ledger.md:947](parity-ledger.md#L947) | [tenstorrent-gdn-device-pure.md](specs/tenstorrent-gdn-device-pure.md) | `DONE` | `bcade48d6` | | `BACKEND-TENSTORRENT-KEEPQUANT` | Child of `BACKEND-TENSTORRENT` — keep-quant dense dot on TT-Metal ([#2959](https://github.com/mudler/vllm.cpp/issues/2959)): GGUF k-quant arms refuse on the P150 until this row lands — W1 (#2989, open) lands the Q4_K decode, W2 (branch) lands the dot and the `kTENSTORRENT` predicate arm (`gguf_keep_quant.cpp:136-148`) admitting exactly the registered set `{Q4_K, Q5_K, Q6_K, Q8_0}` after wave-2b — while the smallest Qwen3.8 artifact that fits the card is the dense 27B Q4_K_M (17.1 GB; bf16 is 53.8 GB) and a quantized Qwen3.5 (0.8B, ~0.5 GB) is the test vehicle | ggml k-quant block formats via our reader, pinned bit-exact vs llama.cpp `b10451` (#2240/#2260 lineage); no vLLM mirror for TT (secondary substrate, deviation by design); substrate surveyed 2026-09-05: tt-metal has NO packed-weight matmul (`ttnn` quantization is per-tensor activation dquant only), so the kernel is ours — resident blocks, on-core decode to bf16 tiles through f32 (decode bit-exact vs `BlockToFloat`), the dot reusing the `kMatmulBT` tile path, device-bf16 band at the residual-golden boundary, the int8 dot as the named next lever | W1 LANDED (#2989): the Q4_K block-decode device path. W2a/2b LANDED (5e43d770, 14e8fe471): the grouped provider `kMatmulBTQuantGrouped` — E=1 dense, E=N stacked tower, registered set exactly {Q4_K, Q5_K, Q6_K, Q8_0}. W3a ON BRANCH: the E=1 arm is CHUNKED slice-decode + f32 assembly — capture-safe on the 0.8B head (trace demand 13,484,032 B of the 52,428,800 B region; capture ×2 byte-identity; routing ids never read; staging writes during capture 0; both capture-time failure modes measured and bounded by the CHUNK POLICY); E=N capture indirection staged-owed behind a MoE artifact (spec ## W4). OWED: the int8 lever, the 27B arm + model wiring (wave-3b), the captured e2e vehicle leg | W1 LANDED (#2989): decode bit-exact vs `vt::cpu::BlockToFloat` across a shape sweep (red-first, op-level suite pattern). W2a/2b LANDED: grouped sweep green across the four encodings (E=1 vs cpu 0.279–0.720, vs analytic-ref 0.201–0.514). W3a ON BRANCH (red-first): the head-shape capture leg reds at base in the wave-1b class (whole-tensor device DRAM fatal, bank_manager.cpp:462) and greens with the CHUNK POLICY (trace demand 13,484,032 B, staging writes 0, ×2 byte-identity, deterministic across passes); the chunked-decode bit-exact leg pins decode math unchanged (81 assertions); the sweep under forced 3-row chunks is bit-identical to the whole-slice ratios; full backend suite 63/63 cases, 524,267 assertions; vehicle 16/16, 147/147, 0 forward-divergent. OWED: the captured e2e vehicle leg | [tenstorrent-keepquant.md](specs/tenstorrent-keepquant.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-KEEPQUANT` — implementer on the row branch; spec commit on `row/BACKEND-TENSTORRENT-KEEPQUANT`, claim in [.agents/claims/](claims/CLAIM-BACKEND-TENSTORRENT-KEEPQUANT.md) | | `BACKEND-TENSTORRENT-QWEN35` | Child of `BACKEND-TENSTORRENT` — the wiring row: `Qwen3_5ForConditionalGeneration` (dense text GDN hybrid) on the TT allow-list, the op delta the family forward refuses by name (`kGdnPostConv`, `kSigmoidGateBf16`, `kAttnQkNormRopeGate`, `kAttnQkNormRope` — pinned empirically by a W0 refusal sweep), and the first e2e gate. Makes the GDN row's ops production-reached (#1715 stays open until the family runs; GDN row lifecycle moves in the same change this lands) | Substrate: the GDN row's 8 kernels + the 27-op TT registry; e2e mirrors the ratified Mistral TT golden treatment (`VT_DUMP_IDS` bootstrap → `scripts/qwen3-neartie-gap-transformers.py` teacher-forced near-tie gaps → committed device-golden pair, POL-ORACLE deviation); op-level oracle is the CPU f32 arm. Capacity: `Qwen/Qwen3.5-0.8B` bf16 ≈1.6 GB fits (proven envelope Mistral-7B ≈14.5 GB); 27B bf16 ≈53.8 GB and GGUF k-quant arms (no TT kernels) refused by name, owed | `src/vllm/platforms/tenstorrent.cpp` (allow-list, lands last) + `src/vt/tenstorrent/tenstorrent_ops.cpp` (op delta) + `tests/parity/test_qwen35_paged_engine.cpp` (TT arm) | Spec committed 2026-08-23; W0 sweep, W1 op delta, W2 e2e, W3 GDN-reviewer leftovers (d2h counter completeness, `conv_transposed` fast-path check) owed | [tenstorrent-qwen35.md](specs/tenstorrent-qwen35.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-QWEN35` | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index f88284157..a3fa5c29c 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -172,8 +172,8 @@ lifecycle are unchanged. |---|---|---|---|---|---|---|---|---| | `SPEC-MTP` | Qwen3.6 MTP heads, k=1 first. **M-mtp-0 CLOSED 2026-07-24: the standalone draft head is oracle-parity-proven on BOTH checkpoints** (27B dense + 35B MoE, k=1, vLLM 0.25.0 executable @ pin `e24d1b24`) - argmax exact on 26/26 unambiguous rows each; the one remaining row per checkpoint is an EXACT oracle top1==top2 tie where vLLM's own `argmax` and `topk` disagree and our pick is a tied maximum; logits within the whole-model bound (atol 0.05 + rtol 0.05), 0/216 out-of-tol on both; shared lm_head isolated is bit-exact on the 35B NVFP4 head. **I2 scheduler-half LANDED (2026-07-24)**: host-side spec plumbing + the FROZEN spec-metadata ABI (spec §2.7) - `SpeculativeConfig`, `DraftTokenIds`, `Request::spec_token_ids`/`NumTokensWithSpec`, populated `scheduled_spec_decode_tokens`, `Scheduler::update_draft_token_ids`, `take_draft_token_ids` seam, `EngineCore::post_step`, `InputBatch::num_accepted_tokens`/`update_req_spec_token_ids`; DEFAULT-OFF and INERT (no `SpeculativeConfig` => `num_lookahead_tokens == 0`). **I3 verify-half LANDED (2026-07-24)**: greedy rejection sampler + per-request logits expansion (see `SPEC-REJECTION`, now `ACTIVE`). **I4 GDN-half LANDED (2026-07-24)**: the GDN speculative slot path + bit-exact state rollback, the piece BOTH GDN-hybrid gate checkpoints need (see `SPEC-GDN-SEGMENTS`, now `ACTIVE`). **I5a GDN LAYER ROUTING + runner spec-metadata upload LANDED (2026-07-24, `CLAIM-SPEC-MTP-I5A`)**: `GdnBlockPaged` now routes a pure-spec batch through `vt::GdnSpecDecode`/`vt::CausalConv1dSpecUpdate` and the runner uploads I4's six spec device tensors — first sub-increment of the scoped M-mtp-1 (I5a GDN wiring → I5b prepare_prefill → I5c MTP paged propose → I5d config+runner-loop+the 27B token gate, spec §5). DEFAULT-OFF INERT, bit-exact vs the I4 ops, no e2e loop yet. **I5b `prepare_prefill_inputs` LANDED (2026-07-24, `CLAIM-SPEC-MTP-I5B`, recorded under `SPEC-REJECTION`)**: the drafter prefill input-prep host routine (shift-splice + `query_len -= num_rejected` + last-token index / metadata) — second scoped M-mtp-1 sub-increment, DEFAULT-OFF INERT, unit-gated RED-first, additive. **I5d CONFIG + RUNNER LOOP LANDED, PARTIAL (2026-07-25, `CLAIM-SPEC-MTP-I5D`)**: `--speculative-config` JSON parse -> `EngineParams::speculative_config`; `LoadedEngine` resolution (`ResolveSpecConfig`/`ResolveMtp`, widened KV `MakeQwen3_5KVCacheSpec(num_spec>0)`, `BuildMtpDraft`, forced sync scheduling, `MakeScheduler(spec)`, `EngineCore(check_for_draft=true)`); the full runner verify/propose loop (draft splice, hidden-tap capture, GDN builder spec-overload feed, k+1 GDN state-slot remap + widened conv cache + draft-KV alloc, `MtpProposePrefill` post-sampling, `take_draft_token_ids`, acceptance telemetry). CUDA `-Werror` 0 warnings, cutlass-ON banner. SPEC-OFF BYTE-IDENTICAL (all gated on `spec_on()`): SACRED 27B 235/235, 35B 315/315, Coder 138/138 + unit test_runner 257 / test_mtp_speculator 169 / test_gdn_metadata_builder 483 / test_ops_gdn 3630 ALL PASS. **The three-way 27B token gate is NOT yet passing** (`tests/parity/test_qwen27_spec_decode.cpp` RUNS the loop + MEASURES the blocker): the spec-ON engine throws on the FIRST prefill step at `gdn_state_gather: working/cache row shapes must match` (`src/vt/ops.cpp:1773`) — I4's spec conv rollback needs the conv row widened to `(K-1)+num_spec` but the non-spec GDN conv ops assume `(K-1)`. Closing needs widened-cache-aware non-spec GDN conv ops + the MIXED `GdnBlockPaged` split/merge. Row LEFT `GATING` at I5e. **I5e LANDED 2026-07-25 (`CLAIM-SPEC-MTP-I5E`) — `SPEC-MTP` LEAVES `GATING`.** Made the non-spec GDN conv ops widened-cache-aware (mirror vLLM `state_len=KERNEL_WIDTH-1` + physical `stride_conv_state_tok`; leading `(K-1)` sub-window; byte-identical at `num_spec==0`, contiguous fast path kept) AND RCA'd the resulting 0-acceptance dead-drafter to the async input-combine overwriting the verify batch's draft position with the committed token (forced off under spec, nullopt-guarded). **THREE-WAY 27B GATE PASSES** (single-request greedy): our-ON == vLLM `--speculative-config mtp` greedy == our-OFF token-for-token; **acceptance 16/16 drafts accepted**, ~16 target steps saved. Spec-OFF SACRED byte-identical (27B 235/235, 35B 315/315, Coder 138/138), `test_ops_gdn` 3678, compute-sanitizer 0 on the spec step. NOT `DONE`: MIXED `GdnBlockPaged` split/merge (concurrency) + throughput A/B are I6. **I6 LANDED 2026-07-25 (`CLAIM-SPEC-MTP-I6`), `benchmark_binding=true` — the §5 c1 THROUGHPUT GATE, first spec-decode speed number:** OURS spec-ON (`examples/vllm-bench` + an additive `--speculative-config` flag, production config) vs pinned vLLM 0.25.0 spec-ON (graphed `vllm serve --speculative-config mtp` + `vllm bench serve`, `enforce_eager=False`/`FULL_AND_PIECEWISE`/inductor; MTP confirmed `Resolved architecture: Qwen3_5MTP`), SAME `{"method":"mtp","num_speculative_tokens":1}`, 27B `~/bench/q36-27b-nvfp4-vllm`, c1, greedy, 8 real prompts x 256 out, prose + code, idle box one-engine-at-a-time under one `flock`, 3 reps (cold TTFT discarded), token-identity re-confirmed FIRST (`test_qwen27_spec_decode` PASS 16/16). RESULT — **ours AT/ABOVE vLLM on EVERY measured axis** (prose / code): TPOT 66.2/62.95 vs 69.1/65.3 ms (ours ~1.04x faster), output tput 15.10/15.72 vs 14.43/15.13 tok/s (+4.6%/+3.9%), ITL 121.6/121.1 vs 123.2 ms, TTFT(warm) 131/131 vs 151.5/181 ms, acceptance ours 0.85/0.92 vs vLLM 0.838 overall (within noise, live drafter both), peak RSS 28.4 GB ON / 24.8 GB OFF (both inside the 119 GiB pool). Spec helps both (ours 1.52x/1.59x, vLLM 1.51x/1.60x TPOT); ours already ~4% faster spec-OFF. STAYS `ACTIVE`: the c>1 mixed spec+non-spec `GdnBlockPaged` split/merge is still refused (needs a row `IndexSelect`/`IndexCopy` vt op) + owes a c>1 A/B, and no user-facing supported `--speculative-config` on the OpenAI server yet (bench flag example-only/additive). Raw logs dgx `~/work/mtp-bench-i6/{results,vresults}`. **I7 LANDED 2026-07-25 (`CLAIM-SPEC-MTP-I7`, `benchmark_binding=true`) — the MIXED spec+non-spec GDN batch (concurrency), the server/CLI `--speculative-config`, and the c>1 A/B — implementation COMPLETE + at vLLM parity; STAYS `ACTIVE` for one honest reason (below), NOT a lag.** New row op `vt::IndexSelect`/`vt::IndexCopy` (CUDA==CPU bit-exact at GDN widths, RED-first); `GdnBlockPagedMixedSpec` split/merge (mirror `qwen_gdn_linear_attn.py:1329-1576`) proven MODEL-INDEPENDENTLY bit-exact (mixed == pure spec + pure prefill, 27B/35B, `test_qwen3_5_gdn_spec_routing`, RED-first by a broken merge); compute-sanitizer 0 on the mixed step + op; server (I5d) + CLI (ABI v6) `--speculative-config`. **c>1 A/B (both spec-ON, same config):** ours ON-PAR-OR-ABOVE vLLM at c2/c4/c8 (output tput within ~+/-2%, ours +1.6%/+2.5% c2, +0.9%/+1.7% c4, +0.9%/-1.1% c8 within noise, prose/code; both ~1.5x spec speedup — does NOT go neutral; acceptance 0.84-0.92 vs vLLM 0.835). **Why STAYS `ACTIVE` (honest, not a lag):** the DONE criterion's strict `token-exact at c>1` clause is a proven MODEL impossibility — the 27B greedy is bf16-batch-nondeterministic (spec-OFF max_seqs 4-vs-1 differs 2/3 short prompts, NO spec involved), affecting vLLM identically, so exact c>1 token identity cannot be met by any correct implementation; c>1 correctness is instead established by the model-independent bit-exact split/merge proof + acceptance parity (near-tie-distributional-gate), with token-exact strict at c1 (I6). No missing work, no lever — the DONE final call is deferred to the user given this criterion ambiguity. SACRED spec-OFF byte-identical 27B 235/235, 35B 315/315, Coder 138/138; CUDA `-Werror` 0 warnings. Raw logs dgx `~/work/mixed-batch/{cN_results,cN_vresults}`. **I8 — `SPEC-MTP` → `DONE` 2026-07-26 (`CLAIM-SPEC-MTP-DONE`, records-only, ZERO code):** the user RATIFIED the deferred c>1 criterion — at concurrency > 1 the DONE bar is the near-tie-distributional form (ours ∈ vLLM's batch-nondeterministic set) + the SPEED delta, NOT strict token-exact (a proven bf16-batch-nondeterminism MODEL impossibility that affects vLLM identically). Both I6-owed DONE items are therefore CLOSED: (1) the MIXED spec+non-spec `GdnBlockPaged` split/merge (I7, model-independently bit-exact + compute-sanitizer 0) with the c2-c8 A/B on-par-or-above vLLM, and (2) the server + CLI + C-ABI(v6) `--speculative-config` flag (I5d/I7, `examples/server/main.cpp`+`examples/cli/main.cpp`+`src/capi/vllm_c.cpp`). MTP k=1 spec-decode is COMPLETE and gated: 27B three-way token-exact at c1 (I5e), c1 above vLLM on every axis (I6), c2-c8 on-par-or-above (I7), spec-OFF byte-identical SACRED (27B 235/235, 35B 315/315, Coder 138/138). This transition is byte-identical BY CONSTRUCTION (`git diff --stat` = records only; ZERO `src/`/`include/`/`examples/` touched, so the I5d/I6/I7 GPU gates stand on this exact code). Tracked follow-ons: the 35B `Qwen3_5MoeMTP` full e2e token gate (M-mtp-2) is now **CLOSED — `DONE` 2026-07-26 (`CLAIM-SPEC-MTP-M-MTP-2`)**: three-way token-exact 16/16 vs the live vLLM 0.25.0 oracle (spec-ON AND spec-OFF), acceptance 16/16 both sides, c1 spec-ON 1.19x TPOT / +16.3% output-tput vs spec-OFF (0.908) — `MODEL-SPEC-qwen3-5-mtp-qwen3-5-moe-mtp` `GATING`→`DONE`, so MTP is `DONE` on BOTH gate models. Remaining spec-decode follow-on: `SPEC-DFLASH` (oracle-BLOCKED, vllm#40898) | T1 | `vllm/v1/worker/gpu/spec_decode/mtp/speculator.py:12`; `vllm/model_executor/models/qwen3_5_mtp.py:63,129-165,192-301`; **I5d** `vllm/engine/arg_utils.py` (`--speculative-config`); `vllm/v1/worker/gpu/model_runner.py:1455-1489` | `include/vllm/config/speculative.h`; `include/vllm/v1/core/sched/scheduler.h`; `src/vllm/v1/core/sched/scheduler.cpp`; `include/vllm/v1/worker/gpu/input_batch.h`; `include/vllm/model_executor/models/qwen3_5_mtp.h:23,58`; `src/vllm/model_executor/models/qwen3_5_mtp.cpp:271`; `src/vllm/model_executor/models/qwen3_5.cpp:3336,3359`; **I5d** `src/vllm/config/speculative.cpp`; `src/vllm/entrypoints/model_loader.cpp` (`ResolveSpecConfig`/`MakeKVCacheMaybeSpec`/ctor wiring); `src/vllm/v1/worker/gpu/runner.cpp` (splice/tap/GDN spec feed/`propose_drafts`/`take_draft_token_ids`/spec-slot remap/draft-KV alloc); `examples/server/main.cpp` | `tests/vllm/v1/test_scheduler.cpp:1135,1238,1272,1316`; `tests/vllm/v1/worker/test_input_batch.cpp`; `tests/vllm/v1/spec_decode/test_mtp_speculator.cpp:201,225,263,299,331` (7/7 cases, 141 assertions); oracle runner `tests/parity/test_op_parity.cpp:1373` + focused case `:1914` (20/20 assertions, both checkpoints, `VLLM_MTP_REQUIRE_CHECKPOINTS=1`); goldens `tests/parity/goldens/qwen3_5_mtp_head_{27b,35b}/`; dump `tools/parity/dump_qwen3_5_mtp.py:144`; **I5d** `tests/parity/test_qwen27_spec_decode.cpp` (three-way gate, RUNS + measures the RCA blocker); **I6** `examples/bench/{main.cpp,bench_core.h}` (additive `--speculative-config` bench flag + acceptance telemetry); **I7** `tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp` (mixed == pure spec + prefill bit-exact), `tests/parity/test_qwen27_spec_decode_concurrent.cpp`, `tests/vt/test_ops_gdn.cpp` (IndexSelect/IndexCopy); DONE closure [ledger](parity-ledger.md#L714) | [mtp-spec-decode.md](specs/mtp-spec-decode.md) | `DONE` | `72f9fb1` | | `SPEC-MTP-K-GT-1` | **MTP speculation DEPTH (`num_speculative_tokens` > 1).** Ports the autoregressive multi-step propose the k=1 early exit sits in front of, so a configured depth is SERVED instead of silently degraded. Before it, `--num-speculative-tokens 3` reserved KV for 3, captured the verify shape at T=4 and stashed ONE draft per request, with no error and no log; a refusal by name landed first and this row removed it in the same flow. `MtpProposeDrafts` runs the prefill, the k=1 early exit, then `prepare_decode_inputs` and the k-1 single-token draft decode steps over the draft's own paged KV, with `update_draft_inputs` recording each step and feeding it forward. Greedy plus accept-if-equal makes the emitted sequence INDEPENDENT of k, so a token-identity gate cannot see a clamped drafter and every depth assertion needs a positive witness beside the identity. The per-depth counters were the FIRST witness and a fresh review proved them BLIND: they report the LENGTH of the emitted draft list, so a propose that runs one forward and pads all k columns satisfies them, and acceptance is zero at every depth on the CPU model, so no acceptance figure separates the arms either. TWO witnesses survive, because one does not cover both failures. `spec_mtp_draft_decode_forwards() == spec_mtp_propose_calls() * (k - 1)`, counted after each draft decode forward RETURNS and guarded by a non-zero call count, catches a propose that SHORT-CIRCUITS or CLAMPS. A third fresh review then proved it does NOT catch PADDING, since a loop that runs every forward and then discards what it sampled increments it honestly. `spec_mtp_proposals_with_varied_drafts()`, read at the CONSUMER on the array the propose delivered, catches exactly that. NEITHER shows per-column provenance, and neither does a non-zero acceptance count AT DEPTH, which a padded row earns whenever the target repeats a token. The owed DGX gate closes it with a per-depth acceptance RATE against a PADDED CONTROL. The CPU tier therefore proves k drafts are PROPOSED and VERIFIED, never ACCEPTED at depth. DEFAULT unchanged at k=1 (both checkpoints' `n_predict`). **NO speed number at any k>1**: the GPU was held by another session for the whole flow, so the DGX three-way at k=2..4 on the 27B and 35B and the matched-k throughput A/B are OWED, as is the bf16 GDN-state arm (the CPU gate runs the f32 arm because `vt::CausalConv1dSpecUpdate` rejects bf16 off CUDA). Also owed and filed: [#1020](https://github.com/mudler/vllm.cpp/issues/1020), a step whose ACTUAL draft count differs from the configured k leaves the captured verify graph silently. | T1 | `vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py:129-274,335-371,374-419,426-471,597-671,674-771` @ `555967922`; `vllm/config/speculative.py:967-991` | [`src/vllm/v1/worker/gpu/spec_decode/mtp/speculator.cpp`](../src/vllm/v1/worker/gpu/spec_decode/mtp/speculator.cpp) (`MtpProposeDrafts`); [`prepare_decode_inputs.cpp`](../src/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_decode_inputs.cpp); `Qwen3_5MTPModel::GatherHiddenRows` ([qwen3_5.cpp](../src/vllm/model_executor/models/qwen3_5.cpp)); `GPUModelRunner::propose_drafts` + the per-depth counters ([runner.cpp](../src/vllm/v1/worker/gpu/runner.cpp), [runner.h](../include/vllm/v1/worker/gpu/runner.h)); the in-memory `mtp_weights` seam ([model_loader.h](../include/vllm/entrypoints/model_loader.h)) | [`test_mtp_depth`](../tests/vllm/v1/spec_decode/test_mtp_depth.cpp) 5/5, 63 assertions (k=1,2,3,4 through `LoadedEngine`, greedy tokens identical to spec-OFF, each arm witnessed BOTH by the draft decode forwards the propose RAN and by whether the DELIVERED draft row varied with depth; neither witness shows per-column provenance, which is owed to the DGX gate); [`test_prepare_decode_inputs`](../tests/vllm/v1/spec_decode/test_prepare_decode_inputs.cpp) 8/8, 33 (both kernel ports + both `max_model_len` clamps, 5 mutations caught); [`test_speculative_mtp_depth`](../tests/vllm/config/test_speculative_mtp_depth.cpp) 4/4, 20; full CPU suite ctest 493 passed / 0 failed / 2 skipped of 495 (the two skips checkpoint-gated and unrelated) | [mtp-k-gt-1.md](specs/mtp-k-gt-1.md) | `ACTIVE` | `CLAIM-SPEC-MTP-K-GT-1` ([#81](https://github.com/mudler/vllm.cpp/issues/81)) | -| `SPEC-MTP-GGUF` | MTP speculative decoding from a GGUF TARGET. Today `FromModelDir` refuses `mtp`+GGUF outright (`src/vllm/entrypoints/model_loader.cpp:717-723`) on the original spike's assumption that GGUF exports carry no `mtp.*` ([mtp-spec-decode.md](specs/mtp-spec-decode.md):979-980, "until we re-export GGUFs with the head"). That is stale: llama.cpp's Qwen3.5 converter DOES emit the head, under layer-indexed `nextn` naming, and our own `HfConfigFromGguf` ALREADY reads `nextn_predict_layers` (it just discards the value into the trunk layer count). Gap is a `TensorResolver` over `GgufFile` mapping `mtp.*` onto `blk.{L+i}.nextn.*` with dequant-to-bf16, one config field, and narrowing the rejection to `dflash`. `ngram`+GGUF already works and is untouched. Qwen3.5/3.6 only (the widened spec KV path serves no other arch). NO ABI change | T2 | llama.cpp (the producer contract; vLLM has no GGUF MTP path) `conversion/qwen.py:535-604` `_Qwen35MtpMixin` (the authoritative `mtp.*`->`nextn` remapper + `add_nextn_predict_layers`); `gguf-py/gguf/constants.py:129,910-917,1494-1501`; `gguf-py/gguf/tensor_mapping.py` `NEXTN_*` | **G1-G3 LANDED 2026-07-28.** `HfConfigFromGguf` republishes the head depth `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:598` (`c.raw["mtp_num_hidden_layers"] = nextn`, previously read then discarded); the head loader `LoadQwen3_5MTPFromGguf` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1447` (+ decl `include/vllm/model_executor/models/qwen3_5_gguf_weights.h:143`) reusing the TRUNK helpers `OwnNormMinus1`/`OwnMatmulWeight`/`OwnBf16`/`LoadAttnGguf`/`LoadMoeGguf` so the head inherits the GGUF (w+1) norm storage, quantization/residency routing and torch [N,K] shapes; `NumMtpLayers`/`UsesDedicatedEmbeddings` exported out of the anon namespace `include/vllm/model_executor/models/qwen3_5_mtp.h`; rejection narrowed to dflash + a head-less-GGUF check `src/vllm/entrypoints/model_loader.cpp` and the head attached in the GGUF branch; **G4 GREEN + `CPU-SPEC-DIVERGENCE` FIXED 2026-07-28**: root cause `src/vllm/model_executor/models/qwen3_5.cpp:3616` sized the GDN state gather/scatter row by `(Kw-1)` while the speculative persistent row is `(Kw-1)+num_spec`, so `GatherRows`/`ScatterRows` mis-strode the slot AND every channel past the first, corrupting post-prefill recurrent state. Fix = `CopyStateRowsStrided` (same TU) used by `GatherStateF32`/`ScatterStateF32` when `cache.shape[2] != work.shape[2]`; the contiguous helpers are kept when the widths agree, so every non-spec path is byte-identical by construction. CPU-only in effect (the fp16/bf16 arm routes through the `GdnStateGather`/`Scatter` ops, so CUDA was never exposed; no GPU result affected) | `tests/vllm/models/test_qwen3_5_gguf_mtp.cpp:109,146,156,184` **4 cases, and the split is the 2026-08-21 repair** ([#1454](https://github.com/mudler/vllm.cpp/issues/1454)): the file used to be the env-gated pair ALONE, each opening on a bare `return`, so with `VLLM_MTP_GGUF_MODEL` unset it reported `test cases: 2 \| 2 passed`, **`assertions: 0`**, `Status: SUCCESS!`, exit 0 - which is every CI run of this repository, the variable being set nowhere in `.github/workflows/`. The `18 assertions` this cell used to record was the LIVE count and was never once reached in CI. Now `:109` and `:146` are **HERMETIC** (KV-only synthetic GGUFs, no weight bytes, 18 assertions on any machine) and pin the arithmetic the old file only NAMED in a comment above `CHECK(c.num_hidden_layers > 0)`: `num_hidden_layers + mtp_num_hidden_layers == block_count` over 65/1, 25/1 and 28/3 - the third arm separating `- nextn` from `- 1` - plus the head-less arm, where the key is NOT published and `NumMtpLayers` answering 1 for an absent key is exactly why the invariant cannot be written with that helper alone. `:156` and `:184` stay env-gated on `VLLM_MTP_GGUF_MODEL` (so CI stays asset-free) and now SKIP LOUDLY with a `MESSAGE` naming the variable, as `tests/vllm/entrypoints/test_gguf_mmproj_reach.cpp` does; `:156` re-derives the same invariant from the file's OWN `block_count` kv. Unset: **4 cases / 18 assertions / `Status: SUCCESS!` / rc 0**. Live on `Qwen3.8-27B-Q4_K_M.gguf` (`block_count` 65, `nextn_predict_layers` 1): **4 cases / 38 assertions / `Status: SUCCESS!` / rc 0**. Mutation-proved on the production line `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:889`, both compiling clean and both restored against a pre-taken sha256: `= block_count` (drop the subtraction) 3/4 cases, 9/18 red, exit 1; `= block_count - 1` (the wrong constant) 2/4 cases, 5/18 red, exit 1. The SAME mutations left the PREVIOUS file at 2/2 cases, 0 assertions, `SUCCESS!`, exit 0. Correctness of the production line is unchanged and was never in question (`1a4db5c3c`, `493327b4e`); this was a test defect. Live-arm content unchanged: depth reaches config.raw; fc is [H,2H] verbatim; 3 norms [H]; head block is full-attention. **RED-first BEHAVIOURAL** (reverting only the G1 line fails both cases 2/2). Trunk inertness: `test_gguf` 103, `test_gguf_qwen36_loader` 99, `test_gguf_keep_quant` 5958, `test_gguf_dequant` 215, `test_capi` 33/232 all unchanged; `tests/parity/test_qwen35_gguf_spec_decode.cpp:74,139` - spec-ON == spec-OFF token-exact with 13 proposed/11 accepted, plus an `ngram` regression guard (widens the cache, never runs the spec conv update) that was token-exact throughout and pinned the widening as innocent. Regression sweep all unchanged: ops_gdn 1825, gdn_metadata_builder 483, gdn_prefill_conv 28, gdn_spec_routing 12, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, gguf_dequant 215, llm_engine 196, input_batch 163, runner 257, capi 232 **GPU CLOSE-OUT + DEVICE-DELTA ATTRIBUTION 2026-07-28 (`G5`-`G7`), ledger [parity-ledger.md#L800](parity-ledger.md#L800).** The GPU end-to-end gate re-run on a from-scratch RELEASE-TARGET build (`-DVLLM_CPP_CUDA_ARCHITECTURES=121a`, build dir DELETED first; arch VERIFIED by `build-cuda/CMakeFiles/vllm.dir/flags.make` `--generate-code=arch=compute_121a,code=[compute_121a,sm_121a]` and by `cuobjdump -lelf` 20 cubins ALL `sm_121a` zero sm_75, NOT by `CMakeCache.txt`, whose `CMAKE_CUDA_ARCHITECTURES:STRING=75` is the `enable_language(CUDA)` compiler-probe default shadowed by the normal variable at `CMakeLists.txt:186` - the prior wrong-arch conclusion was that decoy): dgx.casa GB10 under `flock $HOME/gpu.lock`, 35B A3B NVFP4 GGUF, **2/2 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, 13 proposed / 11 accepted, 90.2 GiB peak RSS, 8m01s; re-run on the EXACT committed source **3/3 cases, 10/10 assertions, exit 0**, 7m25s, the new probe case SKIPping and adding zero assertions. **The CPU-vs-GPU token delta is a MEASURED near-tie, not a defect** (it was never this row's bar - spec-ON == spec-OFF WITHIN a device is): NEW double-gated spec-OFF-only probe `tests/parity/test_qwen35_gguf_spec_decode.cpp:217` (asset + `VLLM_MTP_GGUF_PROBE=1`, 20 alternatives per position, 484/484 assertions per arm, GPU then `CUDA_VISIBLE_DEVICES=` in one `flock` series) shows both arms picking `11751` at position 0 and forking at position 1 on a BIT-IDENTICAL prefix: GPU rank1 `13` -0.773180 over rank2 `11` -0.847055 (margin 0.0739 nats), CPU rank1 `11` -0.765499 over rank2 `13` -0.830374 (margin 0.0649 nats). Each device's pick is the other's rank 2, both ~7x inside the ratified 0.5-nat band, and the cross-device disagreement on the SAME token (0.057 and 0.082 nats) EXCEEDS the margin being decided, so rounding settles it; the 24 texts look unrelated only because positions 2+ cascade off that one coin flip. Margin sweep over all 24 positions: **GGUF GPU and GGUF CPU carry ZERO exact ties**, minimum margins 0.0482 and 0.0649 nats, and both arms reproduced their sequence across every run. **Gate 4 MET on the safetensors sibling of the same quantization run** (`FromModelDir` takes it unchanged): acceptance 12 proposed / 11 accepted vs the GGUF's 13 / 11. That arm, however, FAILS spec-ON == spec-OFF at concurrency 1 and does not reproduce its own spec-OFF sequence run to run, and the probe attributes both to THREE EXACT ties (positions 7, 10, 16, bit-identical logprobs) produced by its 1/16-grid quantized-GEMM logits - which EXONERATES the GGUF arm and opens a recorded, not-root-caused `SPEC-MTP` item on the safetensors NVFP4 path, not on this row. Gate 3 is NOT APPLICABLE twice over: no F16/F32 head-carrying export exists, and the only same-weights sibling is not token-stable against itself. **EVIDENCE RE-ANCHORED 2026-07-29 to a PRODUCTION-CONFIGURED build, because every GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`** (the defect `CLAIM-27B-GATE-RCA` proved, which runs the emulation fp4 GEMM + hand GDN kernels). Re-run from a clean `git archive` tree of `main` `3f34534d`, build proven correct three ways (configure log has ZERO `CUTLASS not found` and prints `CUTLASS found ... sm120a NVFP4 cutlass GEMM` + `FlashAttention-2 ... ENABLED for arch(es) [121a]` + the vendored `sm_121a` Triton-AOT lines with `MANIFEST hashes OK`; `cuobjdump -lelf` 40 cubins ALL `sm_121a`, zero `sm_75`; SACRED `test_qwen27_paged_engine` **235/235 exit 0**, and the build precondition proven to FIRE by recompiling only that TU without the two defines against the same `libvllm.a`, which throws and exits 1 with 0 assertions). **The row PASSES UNCHANGED:** `tests/parity/test_qwen35_gguf_spec_decode` **3/3 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, **13 proposed / 11 accepted (identical to the recorded number)**, 90.26 GiB, 7m13.59s; loader gate 19 assertions on the Qwen3.5-2B and 18 on the 35B A3B, unchanged. **ONE recorded finding is RETRACTED by the re-measurement: the CPU-vs-GPU token delta was a BUILD artifact, not a device near-tie cascade.** On the production build both devices emit the SAME 24 tokens; the probe shows GPU rank1 `11` -0.763897 over rank2 `13` -0.824083 where the defective build had rank1 `13` -0.773180 over rank2 `11` -0.847055, while the CPU arm is bit-identical to the earlier measurement (CUTLASS and Triton are CUDA-only). Zero exact ties in either arm, min margins 0.060186 GPU / 0.064875 CPU, 484/484 assertions per arm. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md) | [specs/gguf-mtp-spec-decode.md](specs/gguf-mtp-spec-decode.md) | `DONE` | `edf91449` | -| `SPEC-DFLASH-GGUF` | DFlash speculative decoding from GGUF, two axes: (A) GGUF DRAFT + safetensors target, (B) GGUF target too. llama.cpp master carries a full `dflash` GGUF contract (arch string `dflash`, tensors `fc`/`enc.output_norm`/`output_norm`/`blk.N.*`, KVs `dflash.target_layers` + `dflash.target_hidden_size`); the arch is ABSENT from checkouts older than ~2026-07, so a stale tree reads as "no contract exists". The GGUF tensor set omits `token_embd`/`output` because the draft SHARES the target's embed+lm_head, which is exactly what `LoadDflashDraft` already does. Blockers are in the loader, not the model: `MakeDflashDraftConfig` reads `draft_dir/config.json` (a GGUF has none), `ResolveDflashDraftDir` probes for `config.json` so it cannot see a `.gguf`, and `LoadDflashDraft` is typed on `std::vector` for the shared bf16 head (the axis-B blocker). Axis A independently shippable. NO ABI change | T2 | llama.cpp `origin/master` @ 2026-07-28 (tag era `b10158`): `gguf-py/gguf/constants.py:547,1151,4350`; `gguf-py/gguf/tensor_mapping.py:1297-1305` (`ENC_OUTPUT_NORM`<-`model.hidden_norm`, `FC`<-`model.fc`); `conversion/qwen.py:351` (mask token via the standard tokenizer KV); `convert_hf_to_gguf.py --target-model-dir` | **GD1-GD7 LANDED 2026-07-28 (BOTH AXES COMPLETE and PROVEN end to end on GB10)**: `MakeDflashGgufConfig` + `LoadQwen3DFlashFromGguf` `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp:88,227` (+ header), `IsDflashGgufDraft` + the `.gguf` branch in `ResolveDflashDraftDir`/`LoadDflashDraft` `src/vllm/entrypoints/model_loader.cpp:121,222`. Goes through the `TensorResolver` seam (unlike `SPEC-MTP-GGUF`) because dflash norms are RAW, so the existing `LoadQwen3DFlash` qkv/gate_up concatenation is reused unchanged. **`GD4` defect FIXED** (`model_loader.cpp:238-249`): the GGUF branch left `config.vocab_size` 0 - correct for `MakeDflashGgufConfig` (the DFLASH arch has no vocab KV and no `token_embd`) but fatal for the forward, which sizes the shared embedding view as `{config.vocab_size, H}`, so the first propose threw `cuda embedding: empty table (vocab 0)`. Now back-filled from the target's `embed_tokens` rows (the condition is on the VALUE, not the draft source, so it generalizes to a GGUF target). Load-level green had hidden it; only GENERATING found it. **GD5-GD7 = axis B**: `SharedHeadSource` `src/vllm/entrypoints/model_loader.cpp` re-expresses the shared bf16 `embed_tokens`+`lm_head` seam as a SOURCE and re-types `LoadDflashDraft`'s second parameter - THAT TYPE was the whole axis-B blocker - with the GGUF arm `LoadGgufSharedEmbedAndHeadBf16` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1034` reusing the trunk loader's tied-embedding rule and sidecar-aware dequant instead of restating them; the shared-head load moved into ONE common tail so all four (draft format x target container) combinations run identical code; the `dflash` half of the GGUF-branch rejection `model_loader.cpp` is deleted (the `mtp` half untouched) and the draft load is wired into the GGUF branch | `tests/vllm/models/test_qwen3_dflash_gguf.cpp:36,84` 2 cases / 47 assertions against the REAL published Qwen3.6-27B DFlash draft (env-gated `VLLM_DFLASH_GGUF_MODEL`, CI asset-free): the +1 target-layer offset undone against the KV read back from the same file, block_size/mask_token present, vocab_size left 0, layer_types cover every block, fc `[H, H*num_taps]` with `nk` SET, qkv/gate_up row-concat shapes, embed/lm_head left EMPTY for the target. **RED-first BEHAVIOURAL** (dropping the `-1` fails the offset checks). **`GD4` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp:343` (second case, draft source env-driven via `VLLM_DFLASH_DRAFT`/`_B`; asset-gated, CI-inert): on dgx GB10 sm_121a against the Qwen3.6-27B NVFP4 safetensors target, the Q4_K_M GGUF draft and the bf16 z-lab safetensors draft produce **token-for-token IDENTICAL** DFlash-ON continuations with **IDENTICAL** accepted/proposed (20/80 on a 24-token prompt, 42/96 on a 48-token prompt), spec-OFF self-reproducible 3/3 and 0 exact ties (min margin 0.197/0.400 nats). Regression: gguf_mtp 19, qwen35_gguf_spec_decode 10, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, ops_gdn 1825, llm_engine 196, capi 232, runner 257 all unchanged. **`GD5` unit gate** `tests/vllm/test_gguf_qwen36_loader.cpp` 3 new synthetic-GGUF cases (6 cases / 286 assertions total, CPU and the dgx CUDA build): the untied head really comes from `output.weight` and not the embedding (distinct fill values), the tied fallback aliases it onto `token_embd`, the `nk` flags separate the gather table from the MatmulBT weight, a file with no `token_embd` is refused. 3-mutant battery, 3 caught (`nk` flipped, head forced to the embedding, tied forced false). **`GD7` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp` third case (targets env-driven via `VLLM_DFLASH_TARGET_B`; asset-gated, CI-inert): on dgx GB10 sm_121a the Qwen3.6-27B NVFP4 **GGUF** target + `Q4_K_M` GGUF draft loads, takes the shared head from the GGUF, generates, and its DFlash-ON continuation is **token-for-token IDENTICAL to that same target's spec-OFF** (24/24, the STRICT form) with acceptance ALIVE at 14/160; 1 case / 15 assertions, exit 0. **The spike's highest risk is EMPTY on this asset, proven not assumed**: the 27B NVFP4 GGUF stores `token_embd`/`output` as ggml BF16, byte-identical to the safetensors sibling (2,542,796,800 bytes each, ZERO differing), so B1's shared-head read is verbatim, not a dequant. Acceptance IS lower than the safetensors-target arm and is NOT chargeable to the head: the two containers diverge at index 4 with NO speculation, because `QUANT-GGUF-NVFP4` is dequant-only so the GGUF target computes in bf16 while the safetensors target runs the true W4A4 kernels. **RE-MEASURED 2026-07-29 on a PRODUCTION-CONFIGURED build (`CLAIM-GGUF-SPEC-REVERIFY`), because every GD4/GD7 GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`.** Build proven correct three ways (see the `SPEC-MTP-GGUF` row; SACRED 27B **235/235**, `cuobjdump` 40 cubins all `sm_121a`). **AXIS B HOLDS EXACTLY**: `test_qwen27_dflash_spec_decode -tc="dflash axis-B*"` **15/15 assertions, exit 0**, GGUF-target DFlash-ON token-identical to that target's own spec-OFF 24/24, acceptance **14/160 unchanged**, cross-target spec-OFF divergence still at index 4, 81.01 GiB peak RSS, 6m53.08s. **AXIS A WAS RED ON THE 48-TOKEN PROMPT (reproducibly, 3 of 3 runs) AND IS NOW CLOSED.** The RED was real: cross-format TOKEN identity held on both prompts, but the exact accept-count half of bar (a) failed (`arm_a.proposed == arm_b.proposed` / `arm_a.accepted == arm_b.accepted`) because the Q4_K_M draft measured **46/112** against the bf16 z-lab draft's **47/96** (one extra 16-wide propose block, one fewer acceptance, zero token difference), 15/17, exit 1; the 24-token prompt stayed green at 17/17 with both drafts at 15/144. **`GD9` 2026-07-29 root-caused it IN WEIGHT SPACE as ordinary `Q4_K_M` cost, category (a), not a defect in our GGUF draft path - and the bar's own premise ("Same weights, two containers") was false for the asset it was pointed at.** The publishing repo also carries an UNQUANTIZED `BF16` GGUF (3,471,497,440 B) beside `Q8_0`/`Q6_K`/`Q5_K`/`Q4_K_M`, which the spec had recorded as nonexistent; that retired the `NOT APPLICABLE` on gate 2. CPU gate `tests/vllm/models/test_qwen3_dflash_gguf.cpp` third case (asset-gated `VLLM_DFLASH_GGUF_BF16_MODEL` + `VLLM_DFLASH_ST_DIR`): `LoadQwen3DFlashFromGguf(BF16)` is **BYTE-IDENTICAL to `LoadQwen3DFlash(z-lab shards)` on all 58 tensors, 302/302 assertions, exit 0**, and FUNCTIONALLY RED against the `Q4_K_M` file (21/302 red, exactly the 21 quantized matmul tensors), so not a vacuous pass. Supporting: our `DequantGgufRowToBf16` is bit-equal to `gguf-py`'s `gguf.quants.dequantize` on the real `fc.weight` (Q4_K), `blk.0.attn_q.weight` (Q4_K) and `blk.2.ffn_down.weight` (Q6_K), zero differing bf16 values; the ladder's mean relative weight error is monotone and uniform with NO outlier tensor (BF16 0, Q8_0 5.6e-3, Q6_K 1.85e-2, Q5_K 3.85e-2, Q4_K_M 7.6e-2); the only numeric config delta is `rms_norm_eps` at 2.5e-9 relative. Also landed: an off-by-default `VT_SPEC_TRACE=1` per-block propose/accept trace in `GPUModelRunner::sample_tokens_with_rejection` (`src/vllm/v1/worker/gpu/runner.cpp`). **`GD10` 2026-07-29 CONFIRMED IT END TO END ON GB10 and closed gates 3 and 5.** Build proven production-configured three ways (configure log 0 `CUTLASS not found`; `cuobjdump -lelf` 40 cubins ALL `sm_121a` zero `sm_75` on both binaries; SACRED `test_qwen27_paged_engine` **235/235, exit 0**, 31.34s, 23.67 GiB). The **`BF16` GGUF draft reads EXACTLY 47/96**, the safetensors draft's own number, at 48 tokens on the discriminating prompt - reproduced 2 of 2 - plus 27/64 = 27/64 at 24 tokens and 15/144 = 15/144 on the second prompt, tokens IDENTICAL throughout, 17/17 exit 0 each time; the `Q4_K_M` arm reads 46/112 on the SAME binary in the SAME `flock` series. Restoring only the draft's numeric precision restores the count, so quantization is the whole cause and nothing structural survives. Bar (a) is consequently SPLIT rather than relaxed (`tests/parity/test_qwen27_dflash_spec_decode.cpp`): tokens stay EXACT unconditionally; accept counts are EXACT on a cross-FORMAT arm and BANDED (`abs(d_accepted) <= 2`, `abs(d_proposed) <= k*2`) on a cross-QUANTIZATION one, with the arm chosen by `IsQuantizedGgufDraft` reading the draft file's ggml types (`GgmlTraits().block_elems > 1`) rather than by a flag. The band is derived, not picked: measured `d_accepted` is 0, 0, -1, so the bound is that maximum plus one quantum; and `d_proposed = -k * d_accepted` EXACTLY once the token streams match (confirmed at -1 / +16), so the proposed bound follows. **Mutation-proved non-vacuous**: rebuilt at band 0 the `Q4_K_M` arm is 15/17 exit 1 while the `BF16` arm stays 17/17 exit 0 on the exact branch. **AXIS B BROADENED from ONE prompt to THREE**, strict form green on all: "The capital of France is" IDENTICAL 14/160 (15/15), "Write a Python function that reverses a string:" IDENTICAL 24/64 (15/15), "Photosynthesis is the process by which" IDENTICAL 15/128 (9/9), all exit 0, ~6m30-6m52 and ~81 GiB peak RSS each. The second prompt REFINES the recorded acceptance claim: the safetensors-target arm is ALSO 24/64 there with the two containers' DFlash-ON streams IDENTICAL, so the GGUF target's lower acceptance is prompt-dependent (their spec-OFF streams diverge at index 4 on the first prompt, index 16 on the second) and not a standing penalty; the cause remains `QUANT-GGUF-NVFP4` being dequant-only, with the shared head excluded by a byte comparison. Gates 1-5 and 7 MET; gate 6 (speed) `PENDING` BY DESIGN and not owed - a DFlash-ON throughput A/B between the two target containers is not a fair comparison until a native NVFP4 GGUF GEMM exists. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md#L845) | [specs/gguf-dflash-draft.md](specs/gguf-dflash-draft.md) | `DONE` | `c62f2fa3` | +| `SPEC-MTP-GGUF` | MTP speculative decoding from a GGUF TARGET. Today `FromModelDir` refuses `mtp`+GGUF outright (`src/vllm/entrypoints/model_loader.cpp:717-723`) on the original spike's assumption that GGUF exports carry no `mtp.*` ([mtp-spec-decode.md](specs/mtp-spec-decode.md):979-980, "until we re-export GGUFs with the head"). That is stale: llama.cpp's Qwen3.5 converter DOES emit the head, under layer-indexed `nextn` naming, and our own `HfConfigFromGguf` ALREADY reads `nextn_predict_layers` (it just discards the value into the trunk layer count). Gap is a `TensorResolver` over `GgufFile` mapping `mtp.*` onto `blk.{L+i}.nextn.*` with dequant-to-bf16, one config field, and narrowing the rejection to `dflash`. `ngram`+GGUF already works and is untouched. Qwen3.5/3.6 only (the widened spec KV path serves no other arch). NO ABI change | T2 | llama.cpp (the producer contract; vLLM has no GGUF MTP path) `conversion/qwen.py:535-604` `_Qwen35MtpMixin` (the authoritative `mtp.*`->`nextn` remapper + `add_nextn_predict_layers`); `gguf-py/gguf/constants.py:129,910-917,1494-1501`; `gguf-py/gguf/tensor_mapping.py` `NEXTN_*` | **G1-G3 LANDED 2026-07-28.** `HfConfigFromGguf` republishes the head depth `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:598` (`c.raw["mtp_num_hidden_layers"] = nextn`, previously read then discarded); the head loader `LoadQwen3_5MTPFromGguf` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1448` (+ decl `include/vllm/model_executor/models/qwen3_5_gguf_weights.h:143`) reusing the TRUNK helpers `OwnNormMinus1`/`OwnMatmulWeight`/`OwnBf16`/`LoadAttnGguf`/`LoadMoeGguf` so the head inherits the GGUF (w+1) norm storage, quantization/residency routing and torch [N,K] shapes; `NumMtpLayers`/`UsesDedicatedEmbeddings` exported out of the anon namespace `include/vllm/model_executor/models/qwen3_5_mtp.h`; rejection narrowed to dflash + a head-less-GGUF check `src/vllm/entrypoints/model_loader.cpp` and the head attached in the GGUF branch; **G4 GREEN + `CPU-SPEC-DIVERGENCE` FIXED 2026-07-28**: root cause `src/vllm/model_executor/models/qwen3_5.cpp:3616` sized the GDN state gather/scatter row by `(Kw-1)` while the speculative persistent row is `(Kw-1)+num_spec`, so `GatherRows`/`ScatterRows` mis-strode the slot AND every channel past the first, corrupting post-prefill recurrent state. Fix = `CopyStateRowsStrided` (same TU) used by `GatherStateF32`/`ScatterStateF32` when `cache.shape[2] != work.shape[2]`; the contiguous helpers are kept when the widths agree, so every non-spec path is byte-identical by construction. CPU-only in effect (the fp16/bf16 arm routes through the `GdnStateGather`/`Scatter` ops, so CUDA was never exposed; no GPU result affected) | `tests/vllm/models/test_qwen3_5_gguf_mtp.cpp:109,146,156,184` **4 cases, and the split is the 2026-08-21 repair** ([#1454](https://github.com/mudler/vllm.cpp/issues/1454)): the file used to be the env-gated pair ALONE, each opening on a bare `return`, so with `VLLM_MTP_GGUF_MODEL` unset it reported `test cases: 2 \| 2 passed`, **`assertions: 0`**, `Status: SUCCESS!`, exit 0 - which is every CI run of this repository, the variable being set nowhere in `.github/workflows/`. The `18 assertions` this cell used to record was the LIVE count and was never once reached in CI. Now `:109` and `:146` are **HERMETIC** (KV-only synthetic GGUFs, no weight bytes, 18 assertions on any machine) and pin the arithmetic the old file only NAMED in a comment above `CHECK(c.num_hidden_layers > 0)`: `num_hidden_layers + mtp_num_hidden_layers == block_count` over 65/1, 25/1 and 28/3 - the third arm separating `- nextn` from `- 1` - plus the head-less arm, where the key is NOT published and `NumMtpLayers` answering 1 for an absent key is exactly why the invariant cannot be written with that helper alone. `:156` and `:184` stay env-gated on `VLLM_MTP_GGUF_MODEL` (so CI stays asset-free) and now SKIP LOUDLY with a `MESSAGE` naming the variable, as `tests/vllm/entrypoints/test_gguf_mmproj_reach.cpp` does; `:156` re-derives the same invariant from the file's OWN `block_count` kv. Unset: **4 cases / 18 assertions / `Status: SUCCESS!` / rc 0**. Live on `Qwen3.8-27B-Q4_K_M.gguf` (`block_count` 65, `nextn_predict_layers` 1): **4 cases / 38 assertions / `Status: SUCCESS!` / rc 0**. Mutation-proved on the production line `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:889`, both compiling clean and both restored against a pre-taken sha256: `= block_count` (drop the subtraction) 3/4 cases, 9/18 red, exit 1; `= block_count - 1` (the wrong constant) 2/4 cases, 5/18 red, exit 1. The SAME mutations left the PREVIOUS file at 2/2 cases, 0 assertions, `SUCCESS!`, exit 0. Correctness of the production line is unchanged and was never in question (`1a4db5c3c`, `493327b4e`); this was a test defect. Live-arm content unchanged: depth reaches config.raw; fc is [H,2H] verbatim; 3 norms [H]; head block is full-attention. **RED-first BEHAVIOURAL** (reverting only the G1 line fails both cases 2/2). Trunk inertness: `test_gguf` 103, `test_gguf_qwen36_loader` 99, `test_gguf_keep_quant` 5958, `test_gguf_dequant` 215, `test_capi` 33/232 all unchanged; `tests/parity/test_qwen35_gguf_spec_decode.cpp:74,139` - spec-ON == spec-OFF token-exact with 13 proposed/11 accepted, plus an `ngram` regression guard (widens the cache, never runs the spec conv update) that was token-exact throughout and pinned the widening as innocent. Regression sweep all unchanged: ops_gdn 1825, gdn_metadata_builder 483, gdn_prefill_conv 28, gdn_spec_routing 12, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, gguf_dequant 215, llm_engine 196, input_batch 163, runner 257, capi 232 **GPU CLOSE-OUT + DEVICE-DELTA ATTRIBUTION 2026-07-28 (`G5`-`G7`), ledger [parity-ledger.md#L800](parity-ledger.md#L800).** The GPU end-to-end gate re-run on a from-scratch RELEASE-TARGET build (`-DVLLM_CPP_CUDA_ARCHITECTURES=121a`, build dir DELETED first; arch VERIFIED by `build-cuda/CMakeFiles/vllm.dir/flags.make` `--generate-code=arch=compute_121a,code=[compute_121a,sm_121a]` and by `cuobjdump -lelf` 20 cubins ALL `sm_121a` zero sm_75, NOT by `CMakeCache.txt`, whose `CMAKE_CUDA_ARCHITECTURES:STRING=75` is the `enable_language(CUDA)` compiler-probe default shadowed by the normal variable at `CMakeLists.txt:186` - the prior wrong-arch conclusion was that decoy): dgx.casa GB10 under `flock $HOME/gpu.lock`, 35B A3B NVFP4 GGUF, **2/2 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, 13 proposed / 11 accepted, 90.2 GiB peak RSS, 8m01s; re-run on the EXACT committed source **3/3 cases, 10/10 assertions, exit 0**, 7m25s, the new probe case SKIPping and adding zero assertions. **The CPU-vs-GPU token delta is a MEASURED near-tie, not a defect** (it was never this row's bar - spec-ON == spec-OFF WITHIN a device is): NEW double-gated spec-OFF-only probe `tests/parity/test_qwen35_gguf_spec_decode.cpp:217` (asset + `VLLM_MTP_GGUF_PROBE=1`, 20 alternatives per position, 484/484 assertions per arm, GPU then `CUDA_VISIBLE_DEVICES=` in one `flock` series) shows both arms picking `11751` at position 0 and forking at position 1 on a BIT-IDENTICAL prefix: GPU rank1 `13` -0.773180 over rank2 `11` -0.847055 (margin 0.0739 nats), CPU rank1 `11` -0.765499 over rank2 `13` -0.830374 (margin 0.0649 nats). Each device's pick is the other's rank 2, both ~7x inside the ratified 0.5-nat band, and the cross-device disagreement on the SAME token (0.057 and 0.082 nats) EXCEEDS the margin being decided, so rounding settles it; the 24 texts look unrelated only because positions 2+ cascade off that one coin flip. Margin sweep over all 24 positions: **GGUF GPU and GGUF CPU carry ZERO exact ties**, minimum margins 0.0482 and 0.0649 nats, and both arms reproduced their sequence across every run. **Gate 4 MET on the safetensors sibling of the same quantization run** (`FromModelDir` takes it unchanged): acceptance 12 proposed / 11 accepted vs the GGUF's 13 / 11. That arm, however, FAILS spec-ON == spec-OFF at concurrency 1 and does not reproduce its own spec-OFF sequence run to run, and the probe attributes both to THREE EXACT ties (positions 7, 10, 16, bit-identical logprobs) produced by its 1/16-grid quantized-GEMM logits - which EXONERATES the GGUF arm and opens a recorded, not-root-caused `SPEC-MTP` item on the safetensors NVFP4 path, not on this row. Gate 3 is NOT APPLICABLE twice over: no F16/F32 head-carrying export exists, and the only same-weights sibling is not token-stable against itself. **EVIDENCE RE-ANCHORED 2026-07-29 to a PRODUCTION-CONFIGURED build, because every GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`** (the defect `CLAIM-27B-GATE-RCA` proved, which runs the emulation fp4 GEMM + hand GDN kernels). Re-run from a clean `git archive` tree of `main` `3f34534d`, build proven correct three ways (configure log has ZERO `CUTLASS not found` and prints `CUTLASS found ... sm120a NVFP4 cutlass GEMM` + `FlashAttention-2 ... ENABLED for arch(es) [121a]` + the vendored `sm_121a` Triton-AOT lines with `MANIFEST hashes OK`; `cuobjdump -lelf` 40 cubins ALL `sm_121a`, zero `sm_75`; SACRED `test_qwen27_paged_engine` **235/235 exit 0**, and the build precondition proven to FIRE by recompiling only that TU without the two defines against the same `libvllm.a`, which throws and exits 1 with 0 assertions). **The row PASSES UNCHANGED:** `tests/parity/test_qwen35_gguf_spec_decode` **3/3 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, **13 proposed / 11 accepted (identical to the recorded number)**, 90.26 GiB, 7m13.59s; loader gate 19 assertions on the Qwen3.5-2B and 18 on the 35B A3B, unchanged. **ONE recorded finding is RETRACTED by the re-measurement: the CPU-vs-GPU token delta was a BUILD artifact, not a device near-tie cascade.** On the production build both devices emit the SAME 24 tokens; the probe shows GPU rank1 `11` -0.763897 over rank2 `13` -0.824083 where the defective build had rank1 `13` -0.773180 over rank2 `11` -0.847055, while the CPU arm is bit-identical to the earlier measurement (CUTLASS and Triton are CUDA-only). Zero exact ties in either arm, min margins 0.060186 GPU / 0.064875 CPU, 484/484 assertions per arm. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md) | [specs/gguf-mtp-spec-decode.md](specs/gguf-mtp-spec-decode.md) | `DONE` | `edf91449` | +| `SPEC-DFLASH-GGUF` | DFlash speculative decoding from GGUF, two axes: (A) GGUF DRAFT + safetensors target, (B) GGUF target too. llama.cpp master carries a full `dflash` GGUF contract (arch string `dflash`, tensors `fc`/`enc.output_norm`/`output_norm`/`blk.N.*`, KVs `dflash.target_layers` + `dflash.target_hidden_size`); the arch is ABSENT from checkouts older than ~2026-07, so a stale tree reads as "no contract exists". The GGUF tensor set omits `token_embd`/`output` because the draft SHARES the target's embed+lm_head, which is exactly what `LoadDflashDraft` already does. Blockers are in the loader, not the model: `MakeDflashDraftConfig` reads `draft_dir/config.json` (a GGUF has none), `ResolveDflashDraftDir` probes for `config.json` so it cannot see a `.gguf`, and `LoadDflashDraft` is typed on `std::vector` for the shared bf16 head (the axis-B blocker). Axis A independently shippable. NO ABI change | T2 | llama.cpp `origin/master` @ 2026-07-28 (tag era `b10158`): `gguf-py/gguf/constants.py:547,1151,4350`; `gguf-py/gguf/tensor_mapping.py:1297-1305` (`ENC_OUTPUT_NORM`<-`model.hidden_norm`, `FC`<-`model.fc`); `conversion/qwen.py:351` (mask token via the standard tokenizer KV); `convert_hf_to_gguf.py --target-model-dir` | **GD1-GD7 LANDED 2026-07-28 (BOTH AXES COMPLETE and PROVEN end to end on GB10)**: `MakeDflashGgufConfig` + `LoadQwen3DFlashFromGguf` `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp:88,227` (+ header), `IsDflashGgufDraft` + the `.gguf` branch in `ResolveDflashDraftDir`/`LoadDflashDraft` `src/vllm/entrypoints/model_loader.cpp:121,222`. Goes through the `TensorResolver` seam (unlike `SPEC-MTP-GGUF`) because dflash norms are RAW, so the existing `LoadQwen3DFlash` qkv/gate_up concatenation is reused unchanged. **`GD4` defect FIXED** (`model_loader.cpp:238-249`): the GGUF branch left `config.vocab_size` 0 - correct for `MakeDflashGgufConfig` (the DFLASH arch has no vocab KV and no `token_embd`) but fatal for the forward, which sizes the shared embedding view as `{config.vocab_size, H}`, so the first propose threw `cuda embedding: empty table (vocab 0)`. Now back-filled from the target's `embed_tokens` rows (the condition is on the VALUE, not the draft source, so it generalizes to a GGUF target). Load-level green had hidden it; only GENERATING found it. **GD5-GD7 = axis B**: `SharedHeadSource` `src/vllm/entrypoints/model_loader.cpp` re-expresses the shared bf16 `embed_tokens`+`lm_head` seam as a SOURCE and re-types `LoadDflashDraft`'s second parameter - THAT TYPE was the whole axis-B blocker - with the GGUF arm `LoadGgufSharedEmbedAndHeadBf16` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1035` reusing the trunk loader's tied-embedding rule and sidecar-aware dequant instead of restating them; the shared-head load moved into ONE common tail so all four (draft format x target container) combinations run identical code; the `dflash` half of the GGUF-branch rejection `model_loader.cpp` is deleted (the `mtp` half untouched) and the draft load is wired into the GGUF branch | `tests/vllm/models/test_qwen3_dflash_gguf.cpp:36,84` 2 cases / 47 assertions against the REAL published Qwen3.6-27B DFlash draft (env-gated `VLLM_DFLASH_GGUF_MODEL`, CI asset-free): the +1 target-layer offset undone against the KV read back from the same file, block_size/mask_token present, vocab_size left 0, layer_types cover every block, fc `[H, H*num_taps]` with `nk` SET, qkv/gate_up row-concat shapes, embed/lm_head left EMPTY for the target. **RED-first BEHAVIOURAL** (dropping the `-1` fails the offset checks). **`GD4` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp:343` (second case, draft source env-driven via `VLLM_DFLASH_DRAFT`/`_B`; asset-gated, CI-inert): on dgx GB10 sm_121a against the Qwen3.6-27B NVFP4 safetensors target, the Q4_K_M GGUF draft and the bf16 z-lab safetensors draft produce **token-for-token IDENTICAL** DFlash-ON continuations with **IDENTICAL** accepted/proposed (20/80 on a 24-token prompt, 42/96 on a 48-token prompt), spec-OFF self-reproducible 3/3 and 0 exact ties (min margin 0.197/0.400 nats). Regression: gguf_mtp 19, qwen35_gguf_spec_decode 10, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, ops_gdn 1825, llm_engine 196, capi 232, runner 257 all unchanged. **`GD5` unit gate** `tests/vllm/test_gguf_qwen36_loader.cpp` 3 new synthetic-GGUF cases (6 cases / 286 assertions total, CPU and the dgx CUDA build): the untied head really comes from `output.weight` and not the embedding (distinct fill values), the tied fallback aliases it onto `token_embd`, the `nk` flags separate the gather table from the MatmulBT weight, a file with no `token_embd` is refused. 3-mutant battery, 3 caught (`nk` flipped, head forced to the embedding, tied forced false). **`GD7` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp` third case (targets env-driven via `VLLM_DFLASH_TARGET_B`; asset-gated, CI-inert): on dgx GB10 sm_121a the Qwen3.6-27B NVFP4 **GGUF** target + `Q4_K_M` GGUF draft loads, takes the shared head from the GGUF, generates, and its DFlash-ON continuation is **token-for-token IDENTICAL to that same target's spec-OFF** (24/24, the STRICT form) with acceptance ALIVE at 14/160; 1 case / 15 assertions, exit 0. **The spike's highest risk is EMPTY on this asset, proven not assumed**: the 27B NVFP4 GGUF stores `token_embd`/`output` as ggml BF16, byte-identical to the safetensors sibling (2,542,796,800 bytes each, ZERO differing), so B1's shared-head read is verbatim, not a dequant. Acceptance IS lower than the safetensors-target arm and is NOT chargeable to the head: the two containers diverge at index 4 with NO speculation, because `QUANT-GGUF-NVFP4` is dequant-only so the GGUF target computes in bf16 while the safetensors target runs the true W4A4 kernels. **RE-MEASURED 2026-07-29 on a PRODUCTION-CONFIGURED build (`CLAIM-GGUF-SPEC-REVERIFY`), because every GD4/GD7 GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`.** Build proven correct three ways (see the `SPEC-MTP-GGUF` row; SACRED 27B **235/235**, `cuobjdump` 40 cubins all `sm_121a`). **AXIS B HOLDS EXACTLY**: `test_qwen27_dflash_spec_decode -tc="dflash axis-B*"` **15/15 assertions, exit 0**, GGUF-target DFlash-ON token-identical to that target's own spec-OFF 24/24, acceptance **14/160 unchanged**, cross-target spec-OFF divergence still at index 4, 81.01 GiB peak RSS, 6m53.08s. **AXIS A WAS RED ON THE 48-TOKEN PROMPT (reproducibly, 3 of 3 runs) AND IS NOW CLOSED.** The RED was real: cross-format TOKEN identity held on both prompts, but the exact accept-count half of bar (a) failed (`arm_a.proposed == arm_b.proposed` / `arm_a.accepted == arm_b.accepted`) because the Q4_K_M draft measured **46/112** against the bf16 z-lab draft's **47/96** (one extra 16-wide propose block, one fewer acceptance, zero token difference), 15/17, exit 1; the 24-token prompt stayed green at 17/17 with both drafts at 15/144. **`GD9` 2026-07-29 root-caused it IN WEIGHT SPACE as ordinary `Q4_K_M` cost, category (a), not a defect in our GGUF draft path - and the bar's own premise ("Same weights, two containers") was false for the asset it was pointed at.** The publishing repo also carries an UNQUANTIZED `BF16` GGUF (3,471,497,440 B) beside `Q8_0`/`Q6_K`/`Q5_K`/`Q4_K_M`, which the spec had recorded as nonexistent; that retired the `NOT APPLICABLE` on gate 2. CPU gate `tests/vllm/models/test_qwen3_dflash_gguf.cpp` third case (asset-gated `VLLM_DFLASH_GGUF_BF16_MODEL` + `VLLM_DFLASH_ST_DIR`): `LoadQwen3DFlashFromGguf(BF16)` is **BYTE-IDENTICAL to `LoadQwen3DFlash(z-lab shards)` on all 58 tensors, 302/302 assertions, exit 0**, and FUNCTIONALLY RED against the `Q4_K_M` file (21/302 red, exactly the 21 quantized matmul tensors), so not a vacuous pass. Supporting: our `DequantGgufRowToBf16` is bit-equal to `gguf-py`'s `gguf.quants.dequantize` on the real `fc.weight` (Q4_K), `blk.0.attn_q.weight` (Q4_K) and `blk.2.ffn_down.weight` (Q6_K), zero differing bf16 values; the ladder's mean relative weight error is monotone and uniform with NO outlier tensor (BF16 0, Q8_0 5.6e-3, Q6_K 1.85e-2, Q5_K 3.85e-2, Q4_K_M 7.6e-2); the only numeric config delta is `rms_norm_eps` at 2.5e-9 relative. Also landed: an off-by-default `VT_SPEC_TRACE=1` per-block propose/accept trace in `GPUModelRunner::sample_tokens_with_rejection` (`src/vllm/v1/worker/gpu/runner.cpp`). **`GD10` 2026-07-29 CONFIRMED IT END TO END ON GB10 and closed gates 3 and 5.** Build proven production-configured three ways (configure log 0 `CUTLASS not found`; `cuobjdump -lelf` 40 cubins ALL `sm_121a` zero `sm_75` on both binaries; SACRED `test_qwen27_paged_engine` **235/235, exit 0**, 31.34s, 23.67 GiB). The **`BF16` GGUF draft reads EXACTLY 47/96**, the safetensors draft's own number, at 48 tokens on the discriminating prompt - reproduced 2 of 2 - plus 27/64 = 27/64 at 24 tokens and 15/144 = 15/144 on the second prompt, tokens IDENTICAL throughout, 17/17 exit 0 each time; the `Q4_K_M` arm reads 46/112 on the SAME binary in the SAME `flock` series. Restoring only the draft's numeric precision restores the count, so quantization is the whole cause and nothing structural survives. Bar (a) is consequently SPLIT rather than relaxed (`tests/parity/test_qwen27_dflash_spec_decode.cpp`): tokens stay EXACT unconditionally; accept counts are EXACT on a cross-FORMAT arm and BANDED (`abs(d_accepted) <= 2`, `abs(d_proposed) <= k*2`) on a cross-QUANTIZATION one, with the arm chosen by `IsQuantizedGgufDraft` reading the draft file's ggml types (`GgmlTraits().block_elems > 1`) rather than by a flag. The band is derived, not picked: measured `d_accepted` is 0, 0, -1, so the bound is that maximum plus one quantum; and `d_proposed = -k * d_accepted` EXACTLY once the token streams match (confirmed at -1 / +16), so the proposed bound follows. **Mutation-proved non-vacuous**: rebuilt at band 0 the `Q4_K_M` arm is 15/17 exit 1 while the `BF16` arm stays 17/17 exit 0 on the exact branch. **AXIS B BROADENED from ONE prompt to THREE**, strict form green on all: "The capital of France is" IDENTICAL 14/160 (15/15), "Write a Python function that reverses a string:" IDENTICAL 24/64 (15/15), "Photosynthesis is the process by which" IDENTICAL 15/128 (9/9), all exit 0, ~6m30-6m52 and ~81 GiB peak RSS each. The second prompt REFINES the recorded acceptance claim: the safetensors-target arm is ALSO 24/64 there with the two containers' DFlash-ON streams IDENTICAL, so the GGUF target's lower acceptance is prompt-dependent (their spec-OFF streams diverge at index 4 on the first prompt, index 16 on the second) and not a standing penalty; the cause remains `QUANT-GGUF-NVFP4` being dequant-only, with the shared head excluded by a byte comparison. Gates 1-5 and 7 MET; gate 6 (speed) `PENDING` BY DESIGN and not owed - a DFlash-ON throughput A/B between the two target containers is not a fair comparison until a native NVFP4 GGUF GEMM exists. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md#L845) | [specs/gguf-dflash-draft.md](specs/gguf-dflash-draft.md) | `DONE` | `c62f2fa3` | | `SPEC-REJECTION` | Rejection sampler. **I3 verify half LANDED (2026-07-24)**: per-request logits EXPANSION to `1 + k_i` rows (`StepInputs::cu_num_logits` / `num_draft_tokens_per_req` / expanded `logits_indices`) plus the GREEDY rejection sampler — accept a draft iff it equals the target argmax at its own position, emit the target argmax on the FIRST mismatch and stop, emit the bonus argmax when all `k_i` accept, `num_sampled = accepted + 1`, `num_rejected = k_i - accepted` (feeds I2's `num_computed_tokens` rollback and `InputBatch::num_accepted_tokens`). One additive vt op (`kGreedyRejectionSample`) with a CPU reference and a CUDA two-phase mirror of upstream's row-argmax + one-thread-per-request accept walk. DEFAULT-OFF and INERT: with no `SpeculativeConfig` no drafts are ever scheduled, `cu_num_logits` is `arange(num_reqs+1)`, `logits_indices` is the pre-change array and the runner never enters the rejection branch. STOCHASTIC/Gumbel, block verification, `apply_sampling_params` over the expanded batch, and the spec grammar bitmask stay DEFERRED (M-mtp-3). **I5b DRAFTER PREFILL INPUT-PREP LANDED (2026-07-24, `CLAIM-SPEC-MTP-I5B`)**: the draft-token input splice this row's I3 note deferred to I5 — `vllm::v1::prepare_prefill_inputs` + its `SpecPrefillInputs` output struct shift each request's `input_ids` left one within its query span, splice the just-sampled next token (`num_sampled>0 ? last_sampled[idx_mapping[r]] : next_prefill_tokens[...]`) into the freed slot, `query_len -= num_rejected`, and emit last-token index / query_start_loc / seq_lens + CG padding (mirror `speculator.py:469-588`, k=1 early-exit :236-238). A HOST routine in a NEW spec_decode-tree TU (no new CUDA kernel; mirrors the DEVICE-NEUTRAL `prepare_inputs`/`combine_sampled_and_draft_tokens` family — the DGX runner leaf ports the loop to the Triton kernel at I5d), unit-gated `test_prepare_prefill_inputs` 7 cases / 27 assertions RED-first, DEFAULT-OFF INERT (nothing calls it until I5d), additive by construction. Row stays `ACTIVE` — the e2e greedy token gate (M-mtp-1) is owed before `DONE` | T1 | `vllm/v1/worker/gpu/spec_decode/rejection_sampler.py:43,101-160`; `rejection_sampler_utils.py:524,564-585,628,828-841,846-849,863-1125`; `vllm/v1/worker/gpu/model_runner.py:866-898,1065-1077`; `vllm/v1/worker/gpu/input_batch.py:303-397,408-453`; **I5b** `vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py:469-588,236-238` | `include/vllm/v1/spec_decode/rejection_sampler.h`; `src/vllm/v1/spec_decode/rejection_sampler.cpp`; `include/vt/ops.h` (`kGreedyRejectionSample`, `vt::GreedyRejectionSample`); `src/vt/cpu/cpu_sample.cpp` (CPU reference); `src/vt/cuda/cuda_sample.cu` (`RejectionRowArgmaxKernel` + `GreedyRejectAcceptKernel`); `src/vt/ops.cpp`; `include/vllm/v1/worker/gpu/prepare_inputs.h` + `src/vllm/v1/worker/gpu/prepare_inputs.cpp` (the expansion); `include/vllm/v1/worker/gpu/runner.h` + `src/vllm/v1/worker/gpu/runner.cpp` (`step_num_logits`, `sample_tokens_with_rejection`); **I5b** `include/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.h` + `src/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.cpp` — anchor `include/vllm/v1/spec_decode/rejection_sampler.h:96` | `tests/vllm/v1/spec_decode/test_rejection_sampler.cpp`; `tests/vllm/v1/worker/test_prepare_inputs.cpp` (expansion + no-draft byte-identity); `tests/vt/test_cuda_ops.cpp` (CUDA==CPU bit-exact at vocab 248320); **I5b** `tests/vllm/v1/spec_decode/test_prepare_prefill_inputs.cpp` (7 cases / 27 assertions, RED-first) — anchor `tests/vllm/v1/spec_decode/test_rejection_sampler.cpp:128` | [mtp-spec-decode.md §2.4,§5](specs/mtp-spec-decode.md) | `ACTIVE` | `CLAIM-SPEC-REJECTION-I3`, `CLAIM-SPEC-MTP-I5B` | | `SPEC-GDN-SEGMENTS` | GDN speculative metadata and slot-snapshot rollback. **I4 LANDED (2026-07-24):** the spec/non-spec metadata split with decode→prefill reclassification (the #34845 case), the `T>1`/`IS_SPEC` GDN recurrence with per-timestep state snapshots, the conv sliding window advancing by the ACCEPTED count, and the k+1 state-slot allocation. DEFAULT-OFF and INERT (`num_spec==0` ⇒ `num_spec_decodes==0`, no shipped kernel branched — both spec kernels are NEW op ids). ROLLBACK PROVEN bit-exact: for every rejection point j the surviving SSM state and conv window are memcmp-identical to running only the accepted prefix through the shipped `vt::GdnDecode`/`CausalConv1dUpdate`, at the real 27B (Hv=48) and 35B (Hv=32) GDN dims on CPU and CUDA. MEASURED state cost: one f32 SSM slot = Hv·Dv·Dk·4B ⇒ 144 MiB/req (27B, 48 layers) / 60 MiB/req (35B, 30 layers) per extra slot; k=1 doubles the GDN SSM state. **I5a GDN LAYER ROUTING WIRED (2026-07-24, `CLAIM-SPEC-MTP-I5A`):** `GdnBlockPaged`'s `num_spec_decodes>0` branch now routes a PURE-spec batch through `vt::CausalConv1dSpecUpdate` + `vt::GdnSpecDecode` (mirror `qwen_gdn_linear_attn.py:1344-1357,1455-1475`), and the runner per-step upload (`StepDevInputs`/`BuildStepDevInputs` + the two decode-graph `Refresh` copies) now carries I4's six spec device tensors, gated by the extended `ValidateGdnAttentionMetadata` spec contract. DEFAULT-OFF INERT (`num_spec_decodes==0` ⇒ stub uploads + the identical non-spec branch). BIT-EXACT vs the I4 ops applied as a token-sequential decode chain, at the real 27B/35B GDN dims, via `GdnBlockPagedForTest` (`tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp`, CPU bit-exact + CUDA on-device); RED-first by a reverted stub (spec recurrence zeroed ⇒ 4/8 fail, maxΔ 1.3-1.6). MIXED spec+non-spec batch refused loudly — lands with I5d's runner loop. Row advances to `ACTIVE`: the M-mtp-1 e2e greedy token gate (verify/propose runner wiring) is owed before `DONE`, and `SPEC-MTP` STAYS `GATING` | T1 | `vllm/v1/attention/backends/gdn_attn.py:189-326,413-462`; `fla/ops/fused_sigmoid_gating.py:66-72,103-116,156-166`; `mamba/ops/causal_conv1d.py:818-1067,1181-1184`; `qwen_gdn_linear_attn.py:1329-1576`; `mamba_utils.py:213-234`; `mamba/abstract.py:55-59` | `include/vllm/v1/attention/backends/gdn_attn.h`; `src/vllm/v1/attention/backends/gdn_attn.cpp`; `include/vt/ops.h` (`kGdnSpecDecode`, `kCausalConv1dSpecUpdate`); `src/vt/ops.cpp`; `src/vt/cpu/cpu_ops.cpp`; `src/vt/cuda/cuda_gdn.cu`; `src/vllm/model_executor/models/qwen3_5_common.{h,cpp}` (`MakeQwen3_5KVCacheSpec`); **I5a:** `src/vllm/model_executor/models/qwen3_5.cpp` (`GdnBlockPaged` spec branch, `StepDevInputs`/`BuildStepDevInputs`, `ValidateGdnAttentionMetadata`), `src/vllm/model_executor/models/qwen3_5_internal.h` (`GdnBlockPagedForTest`) | `tests/vllm/v1/attention/test_gdn_metadata_builder.cpp` (20 cases / 483 assertions incl. the full upstream `GDN_BUILD_TEST_CASES` + default-off byte-identity); `tests/vt/test_ops_gdn.cpp` (reject-at-every-j rollback, CPU + CUDA, real dims); `tests/vllm/models/test_model_registry.cpp` (k+1 slot / widened-conv sizing + `num_spec==0` identity); **I5a** `tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp` (spec-routing bit-exact, RED-first) — anchor `tests/vllm/v1/attention/test_gdn_metadata_builder.cpp:83` | [mtp-spec-decode.md §3,§5](specs/mtp-spec-decode.md) | `ACTIVE` | `CLAIM-SPEC-GDN-I4`, `CLAIM-SPEC-MTP-I5A` | | `SPEC-DFLASH` | Block-diffusion drafter. **READINESS RE-ASSESSED 2026-07-25 (`CLAIM-SPEC-DFLASH-READINESS`, design-only, DONE) against the LANDED MTP machinery (`SPEC-MTP` I1..I7).** Verdict **GREEN, dispatch-ready, NO hardware/oracle/download blocker** (spec [§0](specs/dflash-spec-decode.md)). Refreshed reuse-vs-new map: DFlash gets FREE from landed MTP — the frozen spec-metadata ABI, the greedy rejection sampler (k-general, I3 tested k∈{1,3}), the GDN spec slot path + rollback + mixed spec/non-spec batch (`GdnBlockPagedMixedSpec`/`IndexSelect`/`IndexCopy`, general `num_spec`), the widened-cache-aware conv ops (I5e), the draft-KV layer pattern (`fa_draft`), the I5d/I7 runner verify/propose loop, and **`num_lookahead_tokens=k+1` ALREADY coded** (`speculative.h:91-108` `use_dflash()`); EXTENDS the single I5d-pre `hidden_tap` seam to multi-tap `[T,H×taps]`; builds NEW the `qwen3_dflash` drafter, the project's FIRST non-causal in-block attention primitive, context-KV precompute, `prepare_dflash_inputs`, and the uniform-1+k FULL CG. **k>1 verdict:** the landed rejection + GDN machinery is MECHANICALLY k-general (no `k==1` hardwiring) — DFlash's k=15 blocks need NO mechanism extension, only exercise/validation at scale (D4) + the k+1-slot memory measurement (~2.3 GiB/req 27B GDN state at block-16, the #1 risk, §5). **Checkpoint-fit:** both z-lab drafts EXIST on HF (27B 1.73 GB / 35B 368 MB bf16, DFlashDraftModel) and FIT the 119 GiB pool trivially (drafts NOT yet on dgx — D0 downloads ≤1.73 GB); the active dgx oracle `vllm-oracle-v0.25.0-stage` CONSTRUCTS DFlash (registry `DFlashDraftModel→qwen3_dflash`, speculator dir present) — soft D0 risk = confirm it SERVES DFlash+NVFP4 on sm_121 (non-causal backend; community `AEON-7/vllm-dflash` container proves the combination runs on GB10). W-plan D0-D6 in the spec. **D0+D1 LANDED 2026-07-26 (`CLAIM-DFLASH-D0D1`) on the ADVANCED pin `555967922`/vLLM 0.26.0.dev0 — `SPEC-DFLASH` → `ACTIVE`.** D0 UNBLOCKED (vllm#40898 resolved under `VLLM_USE_V2_MODEL_RUNNER=1`): the mixed-attn z-lab 27B draft CONSTRUCTS + the drafter is ALIVE (acceptance 2.21/8.80/4.75/4.57 > 1, `num_spec=16`, flashinfer-native fp8-KV, goldens committed); gate FORM measured STRICT MODE-MATCHED (vLLM-ON run-deterministic K>=3 but != vLLM-OFF — the k=16 block verify diverges at bf16 near-ties, so NOT the MTP three-way identity). D1 `DF-AUX-TAPS` DONE: `Qwen3_5AuxTaps` + `ModelForwardInput::aux_tap` route to `Qwen3_5{,Dense}Model::ForwardDeviceMultiTap` capturing `(hidden+res)` at `target_layer_ids` into `[T,H×taps]` (eagle3 `_maybe_add_hidden_state`, aux key L+1); config-gated byte-identical off. Unit gate 598 assertions (independent truncated-model reference, RED-first reversed-concat 384 fail); CUDA 697/697 + compute-sanitizer 0; INERTNESS PROVEN — 27B MTP e2e 9/9 + 27B text SACRED 235/235 byte-identical on the new oracle. **D2 `DF-DRAFT-MODEL` CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D2`, kernel row `KERNEL-ATTN-DFLASH-BLOCK`):** the `qwen3_dflash` draft model (plain 5-layer Qwen3-dense reusing `dense_attn_block.h` ops), the project's FIRST non-causal / bidirectional attention primitive `vt::DFlashBlockAttention` (a SEPARATE op — causal `kAttention`/`kPagedAttention` byte-identical), the fc aux-combine, mask-embed, per-layer SWA/full resolution, and the z-lab loader. CPU gate GREEN (op 12/12 incl. RED non-causal; model forward 95/95 incl. RED full-layer-causal-flip + block isolation + fc RED); existing causal `test_ops_attention` 9/9 + `test_qwen3_forward` 1028 UNCHANGED. **D2 GPU PROMOTION GREEN on dgx (`CLAIM-DFLASH-D2`):** CUDA `-Werror` clean, CUDA==CPU 198412/198412 + compute-sanitizer 0, draft-forward parity vs the REAL vLLM draft (fc rel-L2 0.46%, hidden ≤1.3%, 11 STRICT + 5 near-tie ids), 27B SACRED 235/235 + MTP 9/9 byte-identical — **D2 DONE.** **D3 `DF-DRAFT-KV-PREP` DONE 2026-07-26 (`CLAIM-DFLASH-D3`):** `PrecomputeContextKV` + `PrepareDflashInputs` + `ForwardBlockLogitsWithContext` (reuse the UNCHANGED D2 kernel via [context;block]); GPU numeric-parity `test_qwen3_dflash_kvprep_parity` 61/61 (prepare INTEGER bit-exact vs vLLM's Triton kernel, context-KV K/V rel-L2 0.31%/0.26%, 13 STRICT + 3 near-tie = 16/16), CPU 114/114 RED-proven, inertness 235/235 + 9/9 + D2 37/37 byte-identical. **D4 `DF-ENGINE-INTEGRATION` propose brick + `dflash` config-select CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D4D5`):** `DflashProposeBlock`/`SampleDflashBlockDrafts` (the non-autoregressive whole-block propose composing D3 `ForwardBlockLogitsWithContext` + greedy per-mask argmax, anchor not sampled, `dflash/speculator.py:300-413`) + `ParseSpeculativeConfigJson`/`ResolveDflash` accept `method:"dflash"`. CPU gate `test_dflash_propose` 5/19 GREEN (RED-first anchor-read fails 4/5; brick composes forward+sampler; empty-ctx degenerates to D2; config lookahead k+1). Additive + config-gated ⇒ MTP + non-spec byte-identical BY CONSTRUCTION (`git diff --stat` = new speculator TU + config accept-list + CMake + test, NO runner/model/loader/scheduler edit). **D5 `DF-ENGINE-INTEGRATION` runner-loop LANDED + e2e RUNS on dgx 2026-07-26 (`CLAIM-DFLASH-D5`):** full verify/propose loop wired — loader loads the SEPARATE z-lab draft (`LoadDflashDraft`, host bf16 + target-SHARED bf16 embed/lm_head) via a `--speculative-config` `model` key + `ResolveSpecConfig` dflash branch + `runner.set_dflash_draft`; the verify forward captures the D1 multi-tap (`aux_tap`→`ForwardDeviceMultiTap`) instead of the MTP single tap; `propose_drafts_dflash` ACCUMULATES the per-request combined-feature context (`CombineAuxFeatures(aux_tap)`) across steps and honors the `num_rejected` rollback by appending only the `(T_req−num_rejected)` accepted-prefix features, then runs `DflashProposeBlock` (k=16 GDN-spec exercised first time). **e2e (`test_qwen27_dflash_spec_decode`, 4 prompts×32 tok, our-DFlash-ON vs the committed vLLM-DFlash-ON golden): 2/4 STRICT token-exact (fibonacci, three-laws) + acceptance ~ vLLM on ALL 4 (accepted 19/39/29/25 vs golden 17/39/30/25, deltas +2/0/−1/0 — the MANDATORY dead-drafter-trap condition MET).** The 2 divergences (France tok11 `2972`↔`11751`, 17*23 tok12 `567`↔`488`) are SINGLE bf16 near-tie flips (17*23 RE-CONVERGES after one token = proven near-tie; France cascades from one flip) — the ratified near-tie ROOT the D0 gate-form anticipated, rooted in the D3-documented inline bf16 context-KV recompute envelope (~0.3-1.3% rel-L2), NOT a wiring bug (proven by the 2 exact prompts + near-exact acceptance + a non-trivial shared prefix). Inertness GREEN on this build: SACRED `test_qwen27_paged_engine` 235/235 + MTP `test_qwen27_spec_decode` 9/9 byte-identical; CUDA `-Werror` clean; NO new CUDA kernel (host orchestration reusing D1/D2/D3-sanitized ops). **NOT a clean strict-4/4 pass; STRICT 4/4 token-identity + the speed A/B = D6 (the persistent paged draft-KV bit-matching vLLM's fused context-KV projections + the uniform-1+k FULL CG).** Row STAYS `ACTIVE` (correctness at the ratified near-tie envelope; D6 remains) **D6 2026-07-27 (`CLAIM-DFLASH-D6`) — c1 SPEED A/B DONE + STRICT-irreducibility RCA + CG feasibility (records-only, NO source code):** (1) **c1 speed A/B** (`examples/vllm-bench` at `361189a7`, 8 prose+code prompts×256 tok greedy c1, 2 reps): our DFlash-ON = **2.50x TPOT (40.4 vs 101.2 ms) / 2.48x output-tput (24.4 vs 9.86 tok/s)** over our OFF, acceptance 0.22 (3.56/16), rep-stable <1.5%; `benchmark_binding=true`. vs vLLM-DFlash-ON graphed (same workload): vLLM-DFlash-ON graphed = 28.5 tok/s / 35.1 ms TPOT / acceptance_len 4.30 (same 8 prompts, `VLLM_USE_V2_MODEL_RUNNER=1`, mm-off, gpu_util 0.30), so OURS IS ~14% BELOW vLLM-DFlash-ON on output throughput (24.4 vs 28.5 tok/s) - both ~on-par at spec-OFF (9.86 vs 9.83 tok/s), but vLLM extracts a larger DFlash speedup (2.90x vs our 2.47x) because its draft step is fully device-resident + CUDA-graphed (ours host-orchestrates 13 downloads/step) + slightly higher acceptance (~4.3 vs ~3.6 draft tokens/step). The DONE speed bar (ours >= vLLM) is NOT met; closing it = the device-resident draft rewrite + FULL CG (D6 part 2). (2) **STRICT-4/4 proven bf16-IRREDUCIBLE** — the draft KV cache is bf16 not fp8 (`torch_utils.py:398` `auto`→model dtype; the D0 "fp8-KV" was the backend name, not the KV storage dtype), the D3 golden already compares pre-storage bf16 (residual K 0.31%/V 0.26% = sub-ULP kernel noise), and a fused multi-layer KV GEMM is per-element invariant to our per-layer GEMMs ⇒ bit-exact needs vLLM's exact kernels ⇒ the ratified near-tie gate is the FINAL correctness form (no fused-KV code landed). (3) **FULL CG BLOCKED** on a device-resident draft-path rewrite (the D5 path does 13 device→host downloads/step + host `[context;block]` interleaving) — the remaining throughput-parity increment (the perf form of persistent-paged-KV + the graph). Inertness by construction (the gated binary is the D5 binary; SACRED 235/235 + MTP 9/9 stand). Evidence tool `scripts/spec/vllm_dflash_timing.py`. **D7 2026-07-27 (`CLAIM-DFLASH-D7`) — within-step draft forward made DEVICE-RESIDENT (source-owning): `PrecomputeContextKVDevice` keeps per-layer K/V on device; `ForwardBlockLogitsWithContext` builds [context;block] with `vt::IndexCopy`/`IndexSelect` (removes ~30 D→H `Download`s/step). BIT-IDENTICAL (identity bf16↔f32 round-trips replaced) — e2e `test_qwen27_dflash_spec_decode` 27/27 SAME tokens (2/4 STRICT + 2/4 near-tie, acceptance 19/39/29/25), SACRED 235/235 + MTP 9/9, CUDA `-Werror` clean, compute-sanitizer 0 (198412). But the direct old-vs-new A/B = +2.0% output-tput (IN-NOISE) ⇒ D6's "downloads = the ~14% gap" REFUTED by measurement; ours 19.68 tok/s STILL ~33% BELOW vLLM-DFlash-ON 29.2 tok/s (reconstructed 8-prompt set, more prose-heavy); OFF parity our 9.97 ≥ vLLM 9.66. Residual re-attributed: acceptance (ours 2.49 vs vLLM ~3.13 accepted draft-tok/step, bf16-irreducible) + per-step context-KV RECOMPUTE (O(context²), needs the cross-step persistent paged draft-KV store) + eager-vs-graphed. SPEED BAR NOT met; SPEC-DFLASH stays `ACTIVE`; next = persistent paged draft-KV store → then FULL CG. **D9 2026-07-27 (`CLAIM-DFLASH-D9`) — PERSISTENT PAGED DRAFT-KV LANDED (bit-identical, +22.7% throughput, 0.69×→0.917×); D8 acceptance-ceiling REFUTED; residual = FULL CG ONLY:** `qwen3_dflash.cpp` `AppendContextKVHost` (project ONLY newly-accepted rows → per-layer bf16 K/V, append to `PrecomputedContextKV`) + `ForwardBlockLogitsWithPrecomputedKV` (upload the persistent store, NO re-projection) share the core `ForwardWithCtxKVDev` with the old recompute; `runner.cpp::propose_drafts_dflash` swaps the O(context²) per-step recompute (`dflash_ctx_feats_`) for an append-only per-request `dflash_kv_store_` (rollback=don't-append). NO new CUDA kernel; config-gated. BIT-IDENTICAL: CPU `test_dflash_propose` two new D9 cases = exact float equality vs full recompute; GPU e2e `test_qwen27_dflash_spec_decode` **27/27 SAME tokens** (acceptance 19/39/29/25, same divergences France@11/17×23@12); SACRED 235/235 + MTP 9/9 byte-identical; CUDA `-Werror` clean. **A/B (c1, 8 prose+code×256 tok input-len 512, 2 reps <0.1%, `benchmark_binding=true`):** ours-ON **25.75 tok/s** (was D8 20.99, +22.7%) / 38.40 ms TPOT / acc **3.68/step** vs vLLM-ON graphed **28.09** / 35.60 / acc 3.31 = **0.917×** (~8% below, was 0.69×). **Part 1 same-trajectory:** on the 2 token-identical-trajectory prompts ours per-step acceptance == vLLM's EXACTLY (fibonacci 7.80/7.80, three-laws 3.571/3.571, ratio 1.00) AND on the A/B ours acceptance (3.68) is HIGHER than vLLM's (3.31) ⇒ D8's 0.80–0.85× "bf16 acceptance ceiling" is a trajectory-divergence CONFOUND, REFUTED. Residual (~8%) = eager-vs-graphed ONLY (ours ON/OFF 2.60× vs vLLM 2.91×, OFF at parity, recompute eliminated, acceptance higher) — NOT an irreducible ceiling; the FULL uniform-(1+k) CG (device paged-KV store + paged attn, new-CUDA multi-file) is the SOLE un-landed increment. SPEC-DFLASH stays `ACTIVE` (speed not yet ≥ vLLM; residual isolated to FULL CG). **D12 2026-07-27 (`CLAIM-DFLASH-D12`) — A-wire + Part B LANDED + GPU-gated; Part C (capture) remaining; 0.917×:** A-wire makes the D11 Part-A device store the PRODUCTION path (`runner.{h,cpp}` `dflash_kv_store_`→`shared_ptr`, `MakeDeviceKVStore`/`AppendContextKVDevice`/`ForwardBlockLogitsWithDeviceKV`; GPU-gated e2e `test_qwen27_dflash_spec_decode` 27/27 all-exact acceptance 19/39/29/25 + SACRED 235/235 + MTP 9/9 byte-identical, `-Werror` clean). Part B adds `vt::DFlashPagedBlockAttention` (`OpId::kDFlashPagedBlockAttention`), the capture-safe paged kernel with EVERY metadata input a persistent DEVICE tensor and NO function-local host `cu_seqlens` upload (fixes the `cuda_ops.cu:1277-1280` capture-UAF class), gated CPU==CUDA + cross-check vs materialized `DFlashBlockAttention` `test_ops_dflash_paged_block_attn` 795648/795648 + compute-sanitizer 0. Speed 0.917× (A-wire eager + Part B not yet wired into the forward); `benchmark_binding=false`. Part C (static-shape capture + device mask-scatter + `BeginCapture`/replay + the ≥vLLM c1 A/B) is the SOLE remaining piece; if ours-ON-graphed ≥ vLLM-ON → SPEC-DFLASH DONE. Stays `ACTIVE`. **D13 2026-07-27 (`CLAIM-DFLASH-D13`) — Part C LANDED + GPU-GATED; capture-correctness PROVEN; c1 throughput NEAR-PARITY (ours 0.978x, ~2% below vLLM); gap CLOSED 0.917x→0.978x; STAYS `ACTIVE` (≥vLLM bar not yet met):** single-file additive change (`qwen3_dflash.cpp` +368/-58). (C.1) `DflashDeviceKVStore` → fixed-capacity PAGED cache (per-layer pool `[max_pages,16,Hkv,Dh]` + identity `block_table` + `seq_lens`; append = `vt::IndexCopy` scatter at slot==abs-pos, bit-identical to the D9/D11 store). (C.2) `ForwardPagedBody` runs the (1+k) block through the D12 `vt::DFlashPagedBlockAttention` reading the paged store (no `[context;block]` materialization, no function-local host uploads); runner P==1 propose routes through it, P>1 bit-identical materialized fallback. (C.3) per-request CUDA GRAPH over the paged draft step (warm-in-step repopulates the shared pool free-list right before `BeginCapture` — the fix for a `cudaMalloc`-in-capture `Get` miss from the intervening 27B target forward — then `BeginCapture → ForwardPagedBody → EndCaptureGraph`, replay with growing context entering only via in-place `seq_lens`). **Capture-correctness (MANDATORY): `test_qwen27_dflash_spec_decode` 27/27 with the graph (VT_DFLASH_GRAPH=1) BIT-IDENTICAL to eager (=0)** — same divergence tokens (France@11 got[…2972…], 17×23@12 got[…567…]), same acceptance 19/39/29/25 as D5/D7/D9/D12; graph ENGAGED (5 captures C=2048/5/4/15/6, 32+ replays); the token-diff is the capture-safety proof ([[cudagraph-capture-bakes-stack-addresses]]). **c1 A/B (one flock series, cold rep discarded, 8 prompts×256 tok):** our OFF 10.24 / our ON eager-paged 28.65 (28.69,28.61) / **our ON GRAPHED 28.70 (28.70,28.70), TPOT 34.40** / vLLM-ON graphed steady-state 29.35 (tight 3-rep 29.33/29.37/29.33, TPOT 34.07, acc_len 4.44); D9's 28.09 was a colder cross-session outlier — **NEAR-PARITY: ours 0.978× (~2% below) on the rigorous same-session band** (across sessions ours 28.70 falls inside vLLM's observed 28.09–29.37 range). ON/OFF 2.80× (vLLM ~2.98×), our OFF ≥ vLLM OFF. Per the acceptance rule ("below on any axis = an open gap; near-parity is NOT met"), the ≥vLLM bar is NOT met; STAYS `ACTIVE`. Residual (data-grounded): NOT acceptance (ours realized ~3.68 accepted draft-tok/step > vLLM's 3.44) and NOT launch/graph (both graphed, CG neutral) — per-step COMPUTE (~2% slower target-step); next lever = nsys both draft steps (`--cuda-graph-trace=node`), no premature ceiling. **ATTRIBUTION (supersedes D9):** the CUDA graph is perf-NEUTRAL (+0.3%); the ACTUAL lever was the paged context read (C.1/C.2) removing the D9/D12 per-layer `[context;block]` `IndexCopy` materialization of the whole growing context (25.75 D9 → 28.65 eager-paged, +11%) — the roadmap's "the full CG closes the gap" premise is corrected by measurement. Inertness VERIFIED on the capture binary: SACRED 235/235 + MTP 9/9 byte-identical, CUDA `-Werror` clean, no new kernel (D12 paged kernel already memcheck-0 795648), `check-device-leakage` not increased (paged path REMOVES the materialized-buffer allocs + host uploads). `benchmark_binding=true`. Correctness-complete (ratified near-tie); throughput NEAR-PARITY (0.978×, ~2% residual) ⇒ STAYS `ACTIVE` (the capture-correctness gate is MET; the ≥vLLM speed bar is the sole remaining item, a ~2% per-step-compute residual for an nsys). Anchors: `src/vllm/model_executor/models/qwen3_dflash.cpp` (`DflashDeviceKVStore` paged store, `ForwardPagedBody`, the per-request graph in `ForwardBlockLogitsWithDeviceKV`). **D14 2026-07-27 (`CLAIM-DFLASH-D14`) — SPEED GATE MET → SPEC-DFLASH `DONE`:** an nsys (`--cuda-graph-trace=node`) of the graphed spec-on step attributed the D13 ~2% residual to the from-scratch `DFlashPagedBlockAttentionKernel` draft attention (242.9 ms = 1.8% of GPU time, median ~460 us/call over context C~500-640, vs vLLM's fused flash draft-attn ~0.15%; BOTH engines run identical `cutlass_80_wmma` for the draft bf16 GEMMs, so the GEMMs were NOT the gap). Ported it to a WARP-scoped online-softmax variant `DFlashPagedBlockAttentionWarpKernel` (mirrors the shipped `AttentionWarpKernel`: one warp per (block-query,head), `__shfl_xor` butterfly reduction, register accumulator, NO `__syncthreads` storm; SAME paged/block combined-index read + causal/SWA mask + GQA; default ON, `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel for A/B). Draft attn 242.9 → 77.9 ms (3.1x); our-ON c1 28.60 → 29.32 tok/s (+2.5%). **FINAL same-session 3-rep A/B (8 prompts×256 tok, cold leg discarded): our-ON graphed 29.42/29.27/29.32 (med 29.32) vs vLLM-ON graphed 29.240/29.247/29.233 (med 29.240) — our WORST rep (29.27) > vLLM's BEST (29.247), NON-OVERLAPPING bands, 1.003× ⇒ the ≥vLLM speed gate is MET.** Correctness UNCHANGED (output is exact by spec-decode construction — the target verify is untouched, only which draft proposals are accepted can shift): e2e `test_qwen27_dflash_spec_decode` 27/27 with graph==eager BIT-IDENTICAL, acceptance 19/39/29/25 unchanged (draft accepted 1629 identical warp-vs-block across the whole A/B set), 2/4 STRICT (France@11, 17×23@12 unchanged); CUDA==CPU `test_ops_dflash_paged_block_attn` 795648/795648 (warp within the f32 1e-4 / bf16 3e-2 envelope) + compute-sanitizer 0. Inertness SACRED 235/235 + MTP 9/9 byte-identical; CUDA `-Werror` clean; `check-device-leakage` not increased. `benchmark_binding=true`. Block-diffusion drafting is now correctness-complete (ratified near-tie) AND at/above vLLM throughput — this was the roadmap's FINAL open speed item. Anchors: `src/vt/cuda/cuda_ops.cu` (`DFlashPagedBlockAttentionWarpKernel` + `UseDflashAttnBlockKernel`; the D12 block kernel retained as the `VT_DFLASH_ATTN_BLOCK=1` reference). | T1 | `vllm/v1/worker/gpu/spec_decode/dflash/speculator.py`; `vllm/model_executor/models/qwen3_dflash.py`; `vllm/model_executor/models/interfaces.py:1382` (aux value); `eagle3_utils.py:41-56` (+1 shift) | `include/vllm/model_executor/models/qwen3_5.h` (`Qwen3_5AuxTaps`, `ForwardDeviceMultiTap`); `qwen3_5_dense.h`; `model_registry.h` (`aux_tap`); `src/vllm/model_executor/models/qwen3_5.cpp` (`MaybeCaptureAuxTap`/`ValidateAuxTapLayerIds`/`ForwardDeviceMultiTap`); `qwen3_5_moe.cpp`+`qwen3_5_dense.cpp` (routing); D2/D3 `include/vllm/model_executor/models/qwen3_dflash.h` + `src/vllm/model_executor/models/qwen3_dflash{,_weights}.cpp`; D4 `include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h` + `src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp` (`DflashProposeBlock`/`SampleDflashBlockDrafts`); D5 `src/vllm/entrypoints/model_loader.cpp` (`LoadDflashDraft`/`DflashDraft`) + `include/vllm/entrypoints/model_loader.h`; D5 `src/vllm/v1/worker/gpu/runner.cpp` (`set_dflash_draft`/`propose_drafts_dflash`/aux-tap capture) + `include/vllm/v1/worker/gpu/runner.h`; `src/vllm/config/speculative.cpp` + `include/vllm/config/speculative.h` (`ResolveDflash` + `dflash`/`model` parse); D14 warp kernel [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1433) | `tests/vllm/models/test_qwen27_paged_forward.cpp` (multi-tap 598); `tests/vt/test_ops_dflash_block_attn.cpp`; `tests/vllm/models/test_qwen3_dflash_forward.cpp`; `tests/vllm/v1/spec_decode/test_dflash_kvprep.cpp`; `tests/parity/test_qwen3_dflash_{draft,kvprep}_parity.cpp`; D4 `tests/vllm/v1/spec_decode/test_dflash_propose.cpp` (5/19, RED-first); D5 `tests/parity/test_qwen27_dflash_spec_decode.cpp` (e2e 27/27, 2/4 strict + acceptance~vLLM); `scripts/spec/d{0,2,3}_dflash_*.py`; `tests/parity/goldens/dflash_27b{,_draft,_kvprep}/`; D6 `scripts/spec/vllm_dflash_timing.py` (vLLM-DFlash c1 timing); D7 device-resident `src/vllm/model_executor/models/qwen3_dflash.cpp` (`PrecomputeContextKVDevice` + `ForwardBlockLogitsWithContext` via `vt::IndexCopy`/`IndexSelect`); D9 persistent paged draft-KV `qwen3_dflash.{h,cpp}` (`AppendContextKVHost`/`ForwardBlockLogitsWithPrecomputedKV`/`ForwardWithCtxKVDev`/`PrecomputedContextKV`) + `runner.{h,cpp}` (`dflash_kv_store_`/`propose_drafts_dflash`) + `tests/vllm/v1/spec_decode/test_dflash_propose.cpp` (2 D9 bit-identity cases); D12 A-wire `runner.{h,cpp}` (device store as production path) + D12 Part B `include/vt/ops.h`/`src/vt/ops.cpp`/`src/vt/cpu/cpu_ops.cpp`/`src/vt/cuda/cuda_ops.cu` (`kDFlashPagedBlockAttention`) + `tests/vt/test_ops_dflash_paged_block_attn.cpp` (CPU==CUDA + cross-check, 795648/795648 + sanitizer-0); D13 `src/vllm/model_executor/models/qwen3_dflash.cpp` (fixed-capacity paged `DflashDeviceKVStore` + `ForwardPagedBody` + the per-request draft-step CUDA graph in `ForwardBlockLogitsWithDeviceKV`); D14 [test_ops_dflash_paged_block_attn](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) + [ledger](parity-ledger.md#L738) | [dflash-spec-decode.md](specs/dflash-spec-decode.md) | `DONE` | `489a7544` | diff --git a/.agents/specs/tenstorrent-keepquant.md b/.agents/specs/tenstorrent-keepquant.md index 5a0f2c06f..fc37ce66d 100644 --- a/.agents/specs/tenstorrent-keepquant.md +++ b/.agents/specs/tenstorrent-keepquant.md @@ -284,7 +284,14 @@ to make a failure pass. - IQ-family / sub-IQ1_S encodings (unsloth fork formats). - The int8-dot perf lever (#3031); llama.cpp-comparable throughput numbers. - `docs/USAGE.md` vehicle pin when the arm first runs end to end (the W3 - capture leg hashes the local bytes); 27B arm entry at W4. + capture leg hashes the local bytes); the 27B arm entry LANDED with + wave-3b-2, provenance caveat included — the gate-completion half of that + pin stays owed to [#3042](https://github.com/mudler/vllm.cpp/issues/3042). +- The 27B e2e gate, left UNREACHED by wave-3b-2: + [#3042](https://github.com/mudler/vllm.cpp/issues/3042) owns it. The + committed goldens (`tests/parity/goldens/qwen38_gguf_q4km_27b/`) are + lane-ready, and the TT-side run against them closes the gate. Evidence and + the next lever: `## W4`, wave-3b-2. - Residency reconciliation (RESOLVED BY ARITHMETIC, 2026-09-06, #3030): the twin residency is a 0.8B-only shape. Measured on the pinned 27B artifact: non-expert keep-quant twins need 18.47 GiB, the expert tower @@ -382,6 +389,43 @@ accepted default. The twin-absence policy is unchanged: no whole-weight resident decoded shadow on the dense path; decode planes stay per-chain transients. +**WAVE-3B-2 LANDED AS A STAGED SLICE (2026-09-07, the coordinator resolved +the wave's NEEDS_DECISION as LAND AS A STAGED SLICE).** Landed: the +production MTP drafter skip — the loader's accounting deliberately passes a +declared head because its tensors ARE enumerated as expected, and the +trunk-only load then leaves them unread, so `LogQwen3_5GgufMtpHeadSkip` +prints the skip loud before any weight byte moves: all fifteen `blk.64.*` +tensors, 289,527,808 B, named in full, suppressed only when speculative +method `mtp` is configured (`model_loader.cpp`, +`qwen3_5_gguf_weights.cpp`). The skip message carries the denominator fact: +the pinned llama.cpp `b10451` oracle ignores the same tensors, so a gate +against it is matched work only with this skip loud. Landed with it: the +16-prompt oracle goldens `tests/parity/goldens/qwen38_gguf_q4km_27b/` +(`greedy_ids.npy`), derived from the byte-identical llama.cpp `b10451` +denominator harness — the TT-side run against them closes the gate; the +reachability test `tests/vllm/entrypoints/test_gguf_accounting_reach.cpp`, +which proves the skip through the PRODUCTION loader accounting rather than a +hand-built type; the checkpoint-gated TEST_CASE in +`tests/parity/test_qwen35_paged_engine.cpp`, inert until +`VLLM_CPP_QWEN38_27B_GGUF` names the artifact; and +`VT_TT_KEEPQUANT_CHUNK_BYTES` (env-doc allowlisted), the keep-quant chunk +plane budget — default 256 MiB surveyed on the 0.8B vehicle, empty/unset +keeps the default, a positive integer is a HARD CAP in bytes that trades +command-stream length for live memory (the ceil(N/8) trace term otherwise +forces a 606+ MB head plane whatever the budget says — exactly the alloc +that died at 27B). + +UNREACHED: the 27B e2e gate, owed to +[#3042](https://github.com/mudler/vllm.cpp/issues/3042). The OOM evidence, +eight runs: failing allocations 1,073,725,440 B and 134,184,960 B; +free-at-failure 244 MB → 46 MB → 12.7 MB/bank; ~34 GB allocated against the +32 GB device with a 3.7 MB largest free block; batch budget 512 fails +identically, so the demand is not activation-sized; three mitigations tried +and failed. The residency sits ~11 GB above the ~22-23 GB surveyed design +residency above. Suspects, named and unmeasured: the 2.5 GB bf16 embed +twin, f32 plane transients, possible words double-staging. The next lever +is the device-side allocation trace, not another mitigation. + **W4b — the int8-dot lever ([#3031](https://github.com/mudler/vllm.cpp/issues/3031)).** Quantized-domain integer vec_dot behind the same seam; profile-first attribution; recorded-only throughput floor. Sequenced after W4a, never @@ -424,4 +468,12 @@ dense; wave-3b = 27B wiring + gate + MTP skip + USAGE pin. AMENDED (444,424,192 B = ~113 chains × ~3.5 MB of decode command stream); the trace-region policy moves to the pinned tt-metal's dynamic / per-model practice, demand stays measured and reported, and wave-3b-1c repairs -under it. +under it. AMENDED 2026-09-07 (sixth): wave-3b-2 LANDS AS A STAGED SLICE +(the coordinator's NEEDS_DECISION resolution) — production MTP head skip +by name, byte-identical llama.cpp-b10451 denominator goldens committed, +reachability test through the production loader accounting, +`VT_TT_KEEPQUANT_CHUNK_BYTES`, and the `docs/USAGE.md` 27B arm entry. The +27B e2e gate is UNREACHED — eight OOM runs, ~34 GB allocated against 32 GB +— and [#3042](https://github.com/mudler/vllm.cpp/issues/3042) owns it (see +`## Owed` and `## W4`). The row stays `ACTIVE`: W4b and the 27B gate are +the open scope. diff --git a/docs/USAGE.md b/docs/USAGE.md index ca92c355f..44b31a46d 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -996,6 +996,7 @@ repository in this project's history. | LTX-2.5 Gemma-4 12B text encoder | `text_encoders/gemma4-12b-with-proj-nvfp4-torchao.safetensors` | 7,423,624,178 bytes | `vonkaiser/LTX-2.5-FP8-NVFP4` @ `5a40ba9ab209a90ddb7943d1e3d374c51cfd3256` | `12132b7157925332d2b21de9fc6f507c14f4f0cbc7081484d1968ebf8a19b4bf` | The `--encoder` argument of every render, NVFP4 torchao | This file carries NO `__metadata__` block, so `--encoder-config` is REQUIRED beside it and the loader refuses by name without it (`ltx2_text_encoder.cpp`) | | LTX-2.5 prompt-adherence scorer (an INSTRUMENT, not an oracle) | `pytorch_model.bin` plus `config.json`, `preprocessor_config.json`, `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `special_tokens_map.json` | 598,641,023 bytes for the weights, 602,356,502 bytes for all eight | `openai/clip-vit-base-patch16` @ `57c216476eefef5ab752ec549e440a49ae4ae5f3` | `ec89c7b09c749a60aae3c9cd910516f24b58214a7df060b48962d14c469cfbf0` for `pytorch_model.bin`; all eight measured digests are in `tests/parity/goldens/ltx25_adherence/scorer-pin.json`, which the tool checks before it reads a pixel | What `scripts/ltx25-render-compare.py --adherence-model` scores prompt adherence with (#1854 sub-question 1, #2295). It is upstream's own choice twice over: vLLM registers the family as `CLIPEmbeddingModel` (`registry.py:251` at `5559679229`) and vLLM-Omni's accuracy suite scores video prompt-faithfulness with this exact checkpoint (`tests/e2e/accuracy/helpers.py:497`). **THREE COSTS A READER MUST NOT DISCOVER LATER: the HuggingFace repository DECLARES NO LICENCE** (`cardData.license` is null; the `openai/CLIP` GitHub repository is MIT, but the weights repository asserts nothing), **it ships a PICKLE and no safetensors**, and **its text context is 77 positions**, so a longer prompt is REFUSED and never truncated. The #1864 reference request is 17 CLIP tokens and fits; #1854's own 70-word golden-retriever prompt needs at least 83 and does not, so this scorer cannot answer the example that issue uses to define the problem. Its tower is 224x224, so a 320x192 frame reaches it as a resized centre crop | **REFUSED: any prompt over 77 CLIP text positions**, and it is refused rather than truncated, so #1854's own 70-word golden-retriever example (at least 83 positions) cannot be scored by this instrument at all. **REFUSED: a checkpoint whose sha256 does not match the pin, or a pin whose digest is null.** **MEASURED, and our render FAILS: our own engine's frames at the reference's request** were scored on 2026-09-01 from the 25 that `rc` job `93a60151-7d4d-4718-842c-ef724208be0e` retained on the share. The true prompt still ranks first over all six decoys, by +0.3370 on 15 of 25 frames, so the render depicts the asked-for scene. Its mean CLIP score is 35.2719 against a bound of 36.0087 taken from the reference's own frames, a margin of -0.7368, so it depicts it less well than upstream's and #1854's first sub-question stays open on a measured shortfall. **That reading is ONE render, n = 1**: the lease retained only its first render's frames, so the run-to-run stability of our own adherence score is UNMEASURED, and a reading that moved by 0.74 between runs would make this verdict a coin toss rather than a finding. The numbers are in `.agents/specs/ltx25-prompt-adherence.md` under `## Outcome`. **WHY it falls short is also measured now, and the answer is not the one the record predicted**: `scripts/ltx25-adherence-detail-loss.py` loads this same checkpoint through this same identity check ([#2513](https://github.com/mudler/vllm.cpp/issues/2513)) and REFUTES the smoothness hypothesis -- measured border-free our render carries 1.4031x the reference's absolute high-band power at 1.0373x its mid-band power, so there is no rolloff to call smoothness. The intervention that settles the direction is on the REFERENCE's frames: blurring them until their sharpness is a fifth of ours costs only 1.9687 CLIP points against an observed gap of 2.7305, while still ranking the true prompt first on 23 of 25 frames, so no achievable smoothing reproduces our gap. **A "blurring OUR frames RAISES their score by 1.9131" figure was published here and is WITHDRAWN**: from sigma 0.50 upward a decoy outranks the true prompt on our blurred frames, so those arms are not adherence readings at all, and the only readable row of that sweep moves our score by -0.0029. That row also first published the band figure as "77.79% MORE" from a windowed whole-frame spectrum and WITHDREW it along with the separable-upsampler cause it had suggested; its `## CORRECTION` carries the four-convention table and `## CORRECTION 2` the arm-validity finding. **MISSING: `CLIPEmbeddingModel` itself** -- the ported upstream test runs the HuggingFace half only, because `VllmRunner` needs an installed vLLM and a GPU lease, so vLLM's own runner has never been loaded here. **NOT MEASURED: temporal adherence** -- frames are scored independently, exactly as vLLM-Omni's own middle-frame scorer does. Downloaded under authority recorded in `.agents/developer-preferences.md` (developer, 2026-08-31); loaded through `transformers` at f32, which is upstream's own dtype for this path (`test_clip.py:75`, `dtype="float"`) rather than a widening of ours; needs no GPU | | Qwen3.8-27B GGUF language model | `Qwen3.8-27B-Q4_K_M.gguf` | 17,106,775,008 bytes | `unsloth/Qwen3.8-27B-GGUF` @ `fe1e2a23d973adb629709749dc4f6756df66ef10` | `7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169` | Q4_K_M text model loads through `--model` and decodes on CPU | **The token gate against llama.cpp `b10451` FAILED** on 2026-08-23: tokenizer exact 6/6, generation divergent 5/6 ([evidence](bench-evidence/qwen38-27b-q4km-token-gate-20260823.md), #821). GGUF multimodal forward is missing | +| Qwen3.8-27B GGUF language model, Tenstorrent arm (STAGED SLICE — the e2e gate is UNMET) | `Qwen3.8-27B-Q4_K_M.gguf`, resolved from local path env `VLLM_CPP_QWEN38_27B_GGUF` | 17,106,775,008 bytes (16,314.3 MiB) | Presumed `unsloth/Qwen3.8-27B-GGUF`, but at revision `4ca720788d1e01f1bff70c033e0d0028fd02e502` no file matches this artifact by size or hash (that revision's `UD-Q4_K_M` is 16,464,440,224 bytes, LFS oid `322e194f…`); the exact upstream file and revision are unresolved, and the pin rests on the sha256 beside this. Reproduce the gate from the hash, not the repo path | `7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169` | Loader wiring and the production MTP-head skip are SHIPPED: arch `qwen35`, DENSE — 64 trunk blocks plus the `blk.64` MTP head, whose 15 tensors (289,527,808 B) the loader's skip message names in full on every non-`mtp` load. The gate TEST_CASE (`tests/parity/test_qwen35_paged_engine.cpp`) is checkpoint-gated and stays inert, printing a loud SKIP, until `VLLM_CPP_QWEN38_27B_GGUF` names the file | **The e2e gate is UNMET — device DRAM residency** ([#3042](https://github.com/mudler/vllm.cpp/issues/3042)): eight gate attempts OOM at first forward (~34 GB allocated against the 32 GB device; the row spec records the evidence). `VT_TT_KEEPQUANT_CHUNK_BYTES` is the kernel-internal keep-quant chunk-plane cap, default 256 MiB surveyed on the 0.8B vehicle; empty/unset keeps the default, and a positive integer is a hard cap in bytes | | Qwen3.8-27B GGUF Unsloth-Dynamic language model | `Qwen3.8-27B-UD-Q4_K_M.gguf` | 16,464,440,224 bytes | `unsloth/Qwen3.8-27B-GGUF` @ `4ca720788d1e01f1bff70c033e0d0028fd02e502` | `322e194ff79741c7baa497c240f677f54b201b0efab44ca8e50f122b39123482` | The reader accepts every one of its 866 tensors. This is the artifact `IQ3_S` (ggml id 21) was added for: 4 of those tensors carry it and `GgufFile::Open` refused the whole file over them until #2510, and 4 blocks of `blk.11.ffn_gate.weight` from it are the oracle goldens the decoder is gated on. **The SHA-256 was derived by hashing the local bytes**, not read off a tree API. | The `IQ3_S` keep-quant `vec_dot` is owed, so those 4 tensors EXPAND to bf16 on the GEMM arm of every device (146.13 MiB of blocks against 680.00 MiB of bf16, 3.4 % of the file); no token gate and no throughput number is claimed on this file — #2497 owns the `gfx1151` quant-matched decode number and #2510 owns the `vec_dot` | | Qwen3.8-27B GGUF projector | `mmproj-BF16.gguf` | 931,146,432 bytes | `unsloth/Qwen3.8-27B-GGUF` @ `fe1e2a23d973adb629709749dc4f6756df66ef10` | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` | BF16 `clip` projector loads and validates through `--mmproj` | No request path runs the loaded projector | | Qwen3.6-27B GGUF, the LIMB-3 VEHICLE (a DENOMINATOR, not a capability) | `Qwen3.6-27B-Q4_K_M.gguf` | 16,817,244,384 bytes; 851 tensors | `unsloth/Qwen3.6-27B-GGUF` @ `82d411acf4a06cfb8d9b073a5211bf410bfc29bf` | `5ed60d0af4650a854b1755bd392f9aef4872643dc25a254bc68043fa638392a0`, hashed THREE times and equal each time: by the fetch script as it landed, independently off the staged share afterwards, and a third time on the `strix:gpu0` worker's own copy after the transfer out of `/workspace`. It also equals the digest the forge advertises for that revision. Its vision tower `mmproj-BF16.gguf`, same revision, is 931,146,304 bytes, sha256 `05353347512982ee62317b9d8c89372bc815f4b4043580e7ef3ad411ec1a1cd3`; the tokenizer and config come from `Qwen/Qwen3.6-27B` @ `6a9e13bd6fc8f0983b9b99948120bc37f49c13e9` | ARM: Q4_K_M `qwen35`, `block_count` 64, histogram F32 449 / Q4_K 289 / Q5_K 48 / Q6_K 65 — **pure k-quant, with no non-k-quant weight tier at all**, so it exercises the same `DotQ4K`, `KQuantGemmK`, `QuantizeQ8KK` and `MatmulBTQuantKernelRocm` path the Qwen3.8-27B Q4_K_M ROCm arm runs. It LOADS AND GENERATES on `strix:gpu0`: 3 legs on the shipped default with no knobs, 2 clean and token-identical to each other, 1 `BOARD_FAULT` (GPU Hang, rc 139), zero `[vt reference-tier]` hits. Fetched under authority recorded in `.agents/developer-preferences.md` (developer, 2026-09-04), scoped to ONE vehicle meeting the six conditions [#2864](https://github.com/mudler/vllm.cpp/issues/2864) pre-registered, all six of which it meets ([`docs/bench-evidence/limb3-vehicle-pin-20260904.md`](bench-evidence/limb3-vehicle-pin-20260904.md)) | **THIS CHECKPOINT DID NOT DELIVER LIMB 3, AND THE REASON IS THE ORACLE RATHER THAN THIS FILE.** `STRICT_LIMB3 = NO` ([#2884](https://github.com/mudler/vllm.cpp/issues/2884), [`docs/bench-evidence/limb3-strict-gate-20260904.md`](bench-evidence/limb3-strict-gate-20260904.md)): the pinned vLLM is **not deterministic on this vehicle**, so there is no single denominator to be token-exact against. Its eager and compiled configurations each reproduce themselves exactly and **disagree with each other on 2 of 6 prompts**, and each also disagrees with its own one-pass prefill argmax (eager 3 steps, compiled 2). The strict free-running counts are 3/6 against eager and 3/6 against compiled — **recorded, and NOT usable as a gate result**, and not even the same three prompts. Whether any divergence is an exact tie is **NOT ESTABLISHED**: the run captured `prompt_logprobs=1`, so no runner-up margin exists. **REFUSED AS A CONCLUSION: picking whichever vLLM configuration agrees with us**, which `score_strict.py` forbids in code by short-circuiting on the oracle's self-consistency before it compares our tokens. No throughput, latency or memory figure was taken from this artifact and none may be quoted; `STRIX_ARM_SPEED_RATIFIED_BY` stays unset | diff --git a/include/vllm/model_executor/models/qwen3_5_gguf_weights.h b/include/vllm/model_executor/models/qwen3_5_gguf_weights.h index 6ede32d52..961b1b83a 100644 --- a/include/vllm/model_executor/models/qwen3_5_gguf_weights.h +++ b/include/vllm/model_executor/models/qwen3_5_gguf_weights.h @@ -255,4 +255,42 @@ void LoadGgufSharedEmbedAndHeadBf16(const GgufFile& gguf, OwnedTensor* embed, OwnedTensor* head, bool* head_was_quantized = nullptr); +// KEEPQUANT W4a wave-3b-2 (issue #3030): the MTP drafter head a TRUNK-ONLY +// load leaves unread. +// +// A Qwen3.5-family GGUF converted WITH the head folds it into the ordinary +// block list (`.block_count` counts it; `.nextn_predict_layers` +// announces it), and `LoadQwen3_5DenseFromGguf` / `LoadQwen3_5MoeFromGguf` +// read the trunk only — `config.num_hidden_layers` blocks. The head tensors +// (`blk.{L}.nextn.*` plus the head block's own attn/ffn set) are therefore +// loaded ONLY when speculative decoding is configured, and on every spec-off +// run — the production default, and the shape the pinned llama.cpp `b10451` +// oracle runs too (it loads 64 of this family's 65 blocks and ignores all 15 +// `blk.64` tensors; .agents/oracles/llama-cpp.md) — they stay in the file +// unread. Before wave-3b-2 that skip was SILENT, which is the worst way for a +// gate to be honest about its denominator. +struct Qwen3_5GgufMtpHeadSkip { + // True when the config declares a head AND the file carries at least one of + // its tensors. + bool present = false; + int64_t tensor_count = 0; + int64_t bytes = 0; + // The exact names, in file order. The loud skip line prints them all: a + // skipped set the reader cannot see is a silent one. + std::vector names; +}; + +// Enumerates the head tensors `LoadQwen3_5{Dense,Moe}FromGguf` will NOT read, +// with their file byte sizes. Kept in lockstep with the head block of +// `Qwen3_5GgufExpectedTensors`, whose accounting is what refuses a tensor the +// enumeration forgets. +Qwen3_5GgufMtpHeadSkip Qwen3_5GgufMtpHeadSkipTensors(const GgufFile& gguf, + const HfConfig& config); + +// The loud version: one stderr line naming every skipped tensor, the byte +// total, the trunk/head arithmetic, the speculative-config condition that +// would load them, and the llama.cpp `b10451` denominator-parity note. Inert +// (prints nothing) when `Qwen3_5GgufMtpHeadSkip` reports nothing skipped. +void LogQwen3_5GgufMtpHeadSkip(const GgufFile& gguf, const HfConfig& config); + } // namespace vllm diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index 6c65a292b..71efc7991 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -188,6 +188,7 @@ VT_SPEC_TEST_SELECT_SPIN_MS VT_SPEC_TRACE VT_SWIZZLE_IN_QUANT VT_TT_DUMP_KV +VT_TT_KEEPQUANT_CHUNK_BYTES VT_TT_NO_IDS_WARM VT_TT_NO_IDX_WARM VT_TT_NO_ROPE_REFRESH diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 83203ab49..3e747ff87 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -2980,6 +2980,19 @@ std::unique_ptr LoadedEngine::FromModelDir( // enumeration lives elsewhere is not accounted against qwen3_5's. if (vllm::IsQwen3_5Gguf(gguf)) { vllm::RefuseUnaccountedQwen3_5Gguf(gguf, config); + // KEEPQUANT W4a wave-3b-2 (#3030): the accounting above deliberately + // PASSES a declared head, because its tensors ARE enumerated as + // expected — and the trunk-only load below then leaves them unread. + // Say so, by name, before any weight byte moves. The one config that + // reads the head is speculative method "mtp" (the attach below); on + // every other run the skip is real and the line is its record. The + // pinned llama.cpp b10451 oracle ignores the same tensors, so a gate + // against it is matched work only with this skip loud. + const bool mtp_head_loads = params.speculative_config.has_value() && + params.speculative_config->method == "mtp"; + if (!mtp_head_loads) { + vllm::LogQwen3_5GgufMtpHeadSkip(gguf, config); + } } // LOAD-GGUF-MMPROJ (#821): the SECOND file. Opened, validated and READ // here — after the architecture resolve and the device-fit refusal, and diff --git a/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp b/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp index a79d93eed..f341873d0 100644 --- a/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -1767,4 +1768,65 @@ void RefuseUnaccountedQwen3_5Gguf(const GgufFile& gguf, ".agents/specs/qwen38-27b-quant-arms.md"); } +Qwen3_5GgufMtpHeadSkip Qwen3_5GgufMtpHeadSkipTensors(const GgufFile& gguf, + const HfConfig& config) { + Qwen3_5GgufMtpHeadSkip out; + const int64_t n_mtp = DeclaredMtpDepth(config); + if (n_mtp <= 0) return out; + + // The same names the expected-tensor enumeration lists for the head block: + // the four scalar `nextn.*` tensors on the FIRST head block, then one + // ordinary (always full-attention) block per head layer. Kept in lockstep + // with `Qwen3_5GgufExpectedTensors` — its accounting is what refuses a name + // this list forgets, so the two cannot drift silently. + std::vector want; + const int64_t L = config.num_hidden_layers; + for (const char* stem : + {"nextn.eh_proj.weight", "nextn.enorm.weight", "nextn.hnorm.weight", + "nextn.shared_head_norm.weight"}) { + want.push_back(Blk(L, stem)); + } + for (int64_t i = 0; i < n_mtp; ++i) { + AppendBlockTensors(L + i, /*linear_attention=*/false, + config.num_experts > 0, &want); + } + + // Intersect with the file, in file order, and total the bytes: the numbers + // the loud skip line reports. + std::set want_set(want.begin(), want.end()); + for (const GgufTensorInfo& t : gguf.Tensors()) { + if (want_set.count(t.name) == 0) continue; + out.names.push_back(t.name); + out.bytes += static_cast(t.nbytes); + } + out.tensor_count = static_cast(out.names.size()); + out.present = out.tensor_count > 0; + return out; +} + +void LogQwen3_5GgufMtpHeadSkip(const GgufFile& gguf, const HfConfig& config) { + const Qwen3_5GgufMtpHeadSkip skip = + Qwen3_5GgufMtpHeadSkipTensors(gguf, config); + if (!skip.present) return; + std::string names; + for (size_t i = 0; i < skip.names.size(); ++i) { + names += (i == 0 ? "" : ", ") + skip.names[i]; + } + std::cerr << "engine: qwen3.5 gguf: SKIPPING the MTP drafter head — " + << skip.tensor_count << " tensor(s), " << skip.bytes + << " B, none of them read by this trunk-only load: " << names + << ". They are the multi-token-prediction head at blk." + << config.num_hidden_layers << " (block_count " + << (config.num_hidden_layers + DeclaredMtpDepth(config)) + << " = " << config.num_hidden_layers << " trunk + " + << DeclaredMtpDepth(config) + << " head); they load only when speculative decoding is " + "configured (speculative-config method \"mtp\"). Denominator " + "parity: the pinned llama.cpp b10451 oracle IGNORES these same " + "tensors too (64 trunk layers, no MTP head; " + ".agents/oracles/llama-cpp.md), so a gate against it is " + "matched work only with this skip loud." + << std::endl; +} + } // namespace vllm diff --git a/src/vt/tenstorrent/tenstorrent_ops.cpp b/src/vt/tenstorrent/tenstorrent_ops.cpp index 76e0aa755..ed8be25c7 100644 --- a/src/vt/tenstorrent/tenstorrent_ops.cpp +++ b/src/vt/tenstorrent/tenstorrent_ops.cpp @@ -2625,12 +2625,35 @@ void MatmulBTQuantGroupedKernel(Queue&, Tensor& out, const Tensor& act, // a capture-time constant replayed verbatim. const int64_t chunk_override = KeepQuantChunkRowsOverride().load(std::memory_order_relaxed); - const int64_t chunk = + // W4a wave-3b-2 (#3030): the plane budget is env-tunable the way + // VT_TT_TRACE_REGION_MB is. The 256 MiB default was surveyed on the 0.8B + // vehicle; at 27B the first forward's ffn_down chunk died with 244 MB + // free and a 105 MB largest block, so the gate recipe can shrink the + // plane without a rebuild. Empty/unset keeps the surveyed default. + int64_t plane_bytes = kKeepQuantChunkPlaneBytes; + bool plane_env_set = false; + if (const char* plane_env = std::getenv("VT_TT_KEEPQUANT_CHUNK_BYTES"); + plane_env != nullptr && plane_env[0] != '\0') { + const long long parsed = std::atoll(plane_env); + if (parsed > 0) { + plane_bytes = static_cast(parsed); + plane_env_set = true; + } + } + int64_t chunk = chunk_override > 0 ? std::min(chunk_override, N) : std::min(N, std::max( - kKeepQuantChunkPlaneBytes / (K * 4), + plane_bytes / (K * 4), (N + 7) / 8)); + // W4a wave-3b-2 (#3030): the env knob is a HARD CAP — the ceil(N/8) + // trace term above forces N/8-row chunks for wide-N weights (the head + // [248320, 5120] would decode 31040-row planes, 606+ MB, whatever the + // plane budget says), which is exactly the alloc that died at 27B. A + // set knob trades command-stream length for live memory, the same + // trade VT_TT_TRACE_REGION_MB records on its axis. + if (plane_env_set) + chunk = std::min(chunk, std::max(plane_bytes / (K * 4), 1)); std::vector partials; partials.reserve(static_cast((N + chunk - 1) / chunk)); for (int64_t c0 = 0; c0 < N; c0 += chunk) { diff --git a/tests/parity/goldens/qwen38_gguf_q4km_27b/greedy_ids.npy b/tests/parity/goldens/qwen38_gguf_q4km_27b/greedy_ids.npy new file mode 100644 index 0000000000000000000000000000000000000000..6a0ee7a021824f6af6adab767e769ae986584332 GIT binary patch literal 1152 zcmbVMOK4M35It$q8f&#yjcusPQ(K=gsnRB75#6MMA0(wN#1z!>NFqwBF&~OWRIK2} zg_s+aIGiT17n`;C6QiDSt zjNz1>&6P4mJ7L+|j>hbWWgpHL%SGq-aK4z$$-l!HE#>Gh9dSC}N$q z{`1%Tz6V$gV20~^1CS2_2@i0f8z^uO?fLijzJ&R{g)x4v`$K@W6nMT6Xs4Z711w_B zA6~%TY+~9LnA~3&Q(;~=cJ339Bu2O!>rK5UZFRecygC|z@g@@&WgX=%WK2w9E6@`7 zlc%Q@xE%xTT?L}&fRp5*{C~%)?4nC3WIsmQt$m1HSWJ?Y7BM(X=0>Rm@&#r3RQc6b!A_9Xt2ty`<74rXq_QY zoWlD}O<#>inzeP_&U+bUKa-oNHRC>}fwSC&sW5@&C^>BI?2_(-cA1#3OMuTjcY00i zd1|g1bU6s0rBC=f6T}lUHDY+jSX loud SKIP. +TEST_CASE("qwen3.8-27B GGUF Q4_K_M paged-engine greedy near-tie gate (Tenstorrent, checkpoint-gated)") { + const char* gguf = std::getenv("VLLM_CPP_QWEN38_27B_GGUF"); + if (gguf == nullptr || gguf[0] == '\0') { + SkipGate("qwen38-gguf-q4km-27b", + "VLLM_CPP_QWEN38_27B_GGUF is absent — set it to the local " + "Qwen3.8-27B-Q4_K_M.gguf (unsloth/Qwen3.8-27B-GGUF @ fe1e2a23, " + "sha256 7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b" + "6c6fe169, 17106775008 bytes) to run the 27B keep-quant gate"); + } + RunGate("qwen38_gguf_q4km_27b", "qwen38-gguf-q4km-27b", std::string(gguf), + /*keep_quant=*/true); +} diff --git a/tests/vllm/entrypoints/test_gguf_accounting_reach.cpp b/tests/vllm/entrypoints/test_gguf_accounting_reach.cpp index e96394640..99dd5ec5e 100644 --- a/tests/vllm/entrypoints/test_gguf_accounting_reach.cpp +++ b/tests/vllm/entrypoints/test_gguf_accounting_reach.cpp @@ -24,6 +24,13 @@ // `RefuseUnaccountedClipMmproj` and the projector case. #include +#include +#include + +#include +#include +#include +#include #include #include "vllm/entrypoints/model_loader.h" @@ -142,6 +149,89 @@ std::string Load(const std::string& model_path, constexpr const char* kTokenizerStop = "tokenizer: GGUF missing kv"; +// ── stderr, redirected to a file for the duration of a load ───────────────── +// +// The loud MTP skip line is a PRINT, so the only way to assert it reached an +// operator is to read what the process wrote. Captured at REAL fd 2 by +// dup/dup2 — the shape `test_qwen38_27b_radixark_w4a4_notice.cpp` uses — and +// not a `std::cerr` rdbuf swap, because the surrounding load reports its phases +// with C-level writes an rdbuf swap cannot see, and the capture window must +// hold everything the load printed, in order. +class StderrCapture { + public: + StderrCapture() { + static int counter = 0; + path_ = (std::filesystem::temp_directory_path() / + ("vllm_gguf_accounting_stderr_" + std::to_string(counter++) + + ".txt")) + .string(); + std::fflush(stderr); + saved_ = ::dup(STDERR_FILENO); + sink_ = ::open(path_.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0600); + if (saved_ >= 0 && sink_ >= 0) ok_ = ::dup2(sink_, STDERR_FILENO) >= 0; + } + ~StderrCapture() { + Restore(); + ::unlink(path_.c_str()); + } + StderrCapture(const StderrCapture&) = delete; + StderrCapture& operator=(const StderrCapture&) = delete; + + bool ok() const { return ok_; } + + // Restores the real stderr and returns everything written while it was ours. + std::string Take() { + Restore(); + std::ifstream in(path_, std::ios::binary); + std::ostringstream ss; + ss << in.rdbuf(); + return ss.str(); + } + + private: + void Restore() { + if (saved_ < 0) return; + std::fflush(stderr); + ::dup2(saved_, STDERR_FILENO); + ::close(saved_); + if (sink_ >= 0) ::close(sink_); + saved_ = -1; + sink_ = -1; + } + std::string path_; + int saved_ = -1; + int sink_ = -1; + bool ok_ = false; +}; + +// `Load` with the spec axis exposed, returning {thrown message, stderr}. The +// MTP head loads only for method "mtp", so the skip line's polarity is a +// property of these params — the test asserts it, not the log function. +struct CapturedLoad { + std::string message; + std::string stderr_text; +}; + +CapturedLoad LoadCaptured(const std::string& model_path, + const char* spec_method = nullptr) { + CapturedLoad out; + vllm::entrypoints::EngineParams params; + if (spec_method != nullptr) { + vllm::SpeculativeConfig spec; + spec.method = spec_method; + params.speculative_config = spec; + } + StderrCapture capture; + REQUIRE(capture.ok()); + try { + (void)vllm::entrypoints::LoadedEngine::FromModelDir(model_path, params); + } catch (const std::exception& e) { + out.message = e.what(); + } + out.stderr_text = capture.Take(); + return out; +} + } // namespace TEST_CASE("accounting reach: a fully-accounted GGUF is NOT refused") { @@ -239,3 +329,69 @@ TEST_CASE("accounting reach: a DeepStack projector is accounted, not refused") { CHECK(message.find(kTokenizerStop) != std::string::npos); CHECK(message.find("NEVER reads") == std::string::npos); } + +// KEEPQUANT W4a wave-3b-2 (#3030): the accounting above PASSES a file whose +// declared MTP head stays unread, because the head tensors are enumerated as +// expected — and then the trunk-only load skips them SILENTLY, which is the +// worst way for a gate to be honest about its denominator. The pinned +// llama.cpp b10451 oracle ignores the same tensors (64 trunk layers, no MTP +// head), so a gate against it is matched work only when this skip is LOUD. +// +// The loud line must reach the operator through the production entry point: +// a test that calls the log function directly stays green when the loader's +// call site is deleted, which is the mutation this case is built to fail. +// The fixture stops at the tokenizer — the step AFTER the skip line's place — +// so the tokenizer stop message in the thrown text proves the load walked +// past it, and the captured fd 2 proves what it printed. +TEST_CASE("a trunk-only load NAMES the MTP head tensors it leaves unread") { + // The artifact's shape at depth 5: the head declared AND shipped, the + // production-default spec-off load. + TempFile drafter( + BuildLanguageGguf({/*nextn_tensors=*/1, /*declare_nextn=*/true, ""})); + const CapturedLoad run = LoadCaptured(drafter.path()); + + // The load reached past the skip line's place (it died at the tokenizer, + // several steps later), so whatever it printed is evidence about the skip. + CAPTURE(run.message); + REQUIRE(run.message.find(kTokenizerStop) != std::string::npos); + + const std::string& err = run.stderr_text; + CAPTURE(err); + CHECK(err.find("SKIPPING the MTP drafter head") != std::string::npos); + // The exact names, from the same enumeration the accounting refuses a + // forgotten name with: the four `nextn.*` scalars, then the head block's own + // full-attention set. + for (const char* name : + {"blk.4.nextn.eh_proj.weight", "blk.4.nextn.enorm.weight", + "blk.4.nextn.hnorm.weight", "blk.4.nextn.shared_head_norm.weight", + "blk.4.attn_q.weight", "blk.4.ffn_down.weight"}) { + CHECK(err.find(name) != std::string::npos); + } + // 4 nextn scalars + the 11 tensors of one full-attention block = 15, the + // same shape the 27B artifact skips at blk.64. + CHECK(err.find("15 tensor(s)") != std::string::npos); +} + +TEST_CASE("the MTP head that WILL load prints no skip line") { + // The one config that reads the head: speculative method "mtp". The line + // printing here would tell an operator the head was skipped while the loader + // was about to attach it. + TempFile drafter( + BuildLanguageGguf({/*nextn_tensors=*/1, /*declare_nextn=*/true, ""})); + const CapturedLoad run = LoadCaptured(drafter.path(), /*spec_method=*/"mtp"); + CAPTURE(run.message); + REQUIRE(run.message.find(kTokenizerStop) != std::string::npos); + CHECK(run.stderr_text.find("SKIPPING the MTP drafter head") == + std::string::npos); +} + +TEST_CASE("a head-less file prints no skip line") { + // No `nextn_predict_layers`, no head tensors: nothing is skipped, so the + // line must be inert rather than narrate an empty set. + TempFile plain(BuildLanguageGguf()); + const CapturedLoad run = LoadCaptured(plain.path()); + CAPTURE(run.message); + REQUIRE(run.message.find(kTokenizerStop) != std::string::npos); + CHECK(run.stderr_text.find("SKIPPING the MTP drafter head") == + std::string::npos); +}