feat(BACKEND-TENSTORRENT-KEEPQUANT): the 27B arm — on-core decode residency, TT grouped keep-quant GEMM, e2e gate

[lu-zero]

Row: BACKEND-TENSTORRENT-KEEPQUANT

Qwen3.8-27B Q4_K_M on the P150: restore the on-core decode residency the
27B memory budget forces, run the arm end to end, and gate it against the
pinned llama.cpp oracle.

The 27B artifact decides the residency reconciliation W3 left open
(.agents/specs/tenstorrent-keepquant.md ## Owed, NEEDS_DECISION).
Measured on the pinned artifact (unsloth/Qwen3.8-27B-GGUF @ fe1e2a23,
sha256 7e78da5d...fe169, 17,106,775,008 bytes; 866 tensors; zero IQ, zero
vision):

set	packed	bf16 twin
token_embd (Q4_K, [5120,248320])	0.68 GB	2.37 GB
output (Q6_K, untied head)	0.97 GB	2.37 GB
experts (Q4_K x294)	9.81 GB	32.37 GB
non-expert keep-quant (attn_qkv Q6_K x48, attn_v, ssm_out, ...)	6.01 GB	18.47 GB

The shipped W3 residency — a memoized bf16 twin per keep-quant weight
(DecodedWeightShadow) as MatmulBTQuantKernel's weight operand — is a
0.8B-only shape. On 27B the non-expert twins alone are 18.47 GiB and the
experts are 32.37 GiB: the twin path cannot fit, full stop. W4 therefore
promotes the word-shadow on-core decode (landed, bit-exact vs
vt::cpu::BlockToFloat, capture-guarded, and currently read by no
production path) to the production residency for matmul consumers, and
keeps the twin only where semantics require a materialized table: the
embedding gather (token_embd, 2.37 GiB — acceptable).

Budget on the 32 GB P150: ~16 GiB resident i32 word shadows (all four
encodings; 210/34-byte blocks pad to 53/9 words) + 2.37 GiB embedding twin

• activations/KV. Fits with headroom; record the axis.

Scope:

1. Production residency flip: MatmulBTQuantKernel consumes the per-call
   on-core decode from the resident word shadow for {Q4_K,Q5_K,Q6_K,Q8_0};
   the twin survives for token_embd only. Red-first: the smallest failing
   test is the 0.8B vehicle gate under the flipped residency (the W3 gate
   must stay green — the OOM it memorized was twin CONSTRUCTION, not shadow
   use). Capture-safety: the eager warm step stages every word shadow once
   pre-capture; the captured replay reads stable tensors and stages nothing
   (the property the spec already records for this machinery).
2. kMatmulBTQuantGrouped on kTENSTORRENT (pulled in from ## Owed — the
   27B arm is not e2e without it): the seam already exists (ops.cpp:220,
   weight[E*N,K]) with a production ROCm kernel
   (MatmulBTQuantGroupedKernelRocm, native Q8_0/Q4_K/Q6_K) fed by the
   stacked keep-quant expert tower loader (qwen3_5_gguf_weights.cpp:1287).
   TT work = port the grouped kernel beside the W3 dense decode chains
   (which already cover Q5_K too), consuming the same stacked tower; verify
   the 27B rides the qwen3_5 MoE registration at implementation. Experts
   stay packed and fully resident (9.81 GB stacked). Note: runtime
   per-token on-demand expert loading does NOT exist in the tree — what
   exists is load-phase streaming + host-release (qwen3_5_weights.cpp:1720,
   cuda.cpp:118); the fit comes from packed residency, not from demand
   loading.
3. MTP blk.64.* tensors: skip or refuse by name with a message that names
   the missing part (decide at implementation; record the one-liner).
4. Gate: 27B e2e greedy near-tie gate on the TT arm, checkpoint-gated
   opt-in loud-skip (Qwen3-4B on Tenstorrent: both decode arms put 5/16 gate prompts beyond the 0.5-nat near-tie band #2811 precedent), golden pair minted against the
   pinned llama.cpp b10451 oracle (already gateable) on identical prompts;
   500-mnat band + 0 forward-divergent bar as W3.
5. docs/USAGE.md pin in the same change: repo, revision, file, size,
   sha256, refused arms named.

Out of scope: IQ-family (zero tensors in this artifact), the int8-dot lever
(its own PR), vision towers (none in the artifact).

Risks:

• The 0.8B gate under the flipped residency is the red-first test AND the
  regression risk (capture must stage shadows pre-capture exactly once —
  the staging counter CHECK from W3 guards it).
• 27B oracle goldens need the llama.cpp oracle run on the host (17 GB
  artifact, CPU-side RAM fine); the near-tie band may need more prompts at
  27B scale — keep the minting script parameterized.
• Expert grouped decode is expert-count-dependent; if profile shows decode
  dominates, the int8-dot lever is the booked repayment — sequence, do not
  bundle.
• If the 27B does NOT ride the qwen3_5 MoE registration, the loader needs
  the A3 stacked-KQ path mirrored for the 3.8 registration — discover at
  implementation, one-line record either way.

FOLLOWING_AGENTS_PROTOCOL

[lu-zero]

Wave-1 amendment (spec commit 9026467f1 on row/BACKEND-TENSTORRENT-KEEPQUANT-W4): the blanket residency flip in scope item 1 is falsified on the 0.8B vehicle. The head is tied — [248320,1024] Q6_K, 254,274,560 elements — and its per-call on-core decode re-materializes the 4,068,474,880 B ttnn::where transient, the exact OOM the W3 gate memorized.

Scope item 1 is now threshold-class: weights above 128M elements keep the memoized bf16 twin (embedding gather, logits-head-class weights — a decode transient costs ~16 B/elem peak against the twin's 2 B/elem permanent, and a head decodes every step); weights at or below the threshold consume the per-call word-shadow decode.

The 27B memory axis is updated accordingly: ~16 GiB word shadows + 2.37 GiB embedding twin + 2.37 GiB output-head twin (threshold-class) + activations on 32 GB, expert tower packed. It fits with headroom.

Everything else in this issue stands.

[lu-zero]

Second falsification — the flip waves are cancelled (spec commit a9d6321ea).

Wave-1b's threshold flip passed focused and suite legs (59/59 cases, 6704/6704 assertions), then the first decode-step capture died: TT_FATAL: Creating trace buffers of size 425754624B ... only 52428800B is allocated for trace region (mesh_trace.cpp:81). A weight decoded per call persists as a bf16 tile inside the captured graph, and trace replay re-runs the decode every step — a throughput regression against the twin with no compensating win. The twin-for-all diagnostic re-ran the same gate 16/16 PASS, 147/147 (11 strict / 5 near-tie, max 0.188 nats, 0 forward-divergent), isolating the cause to the flipped weights. No element-count threshold repairs the class, and each candidate repair fails: trace-region enlargement needs ~18 GB on 27B; capture-mode twins leave the word shadow feeding only the eager arm (dead in production); persistent decode scratch defeats the arithmetic; a no-capture fallback breaks the capture model.

Re-anchor (scope item 1 replaced): production dense keep-quant matmuls keep the landed W3 twin. The packed-word residency lands where a kernel consumes words natively — the TT provider for vt::MatmulBTQuantGrouped (src/vt/ops.cpp:220, weight[E*N,K]), porting the ROCm reference's native packed-word dequant (rocm_grouped_gemm.hip:1456; Q8_0/Q4_K/Q6_K, Q5_K owed — the 27B pin carries 48 Q5_K tensors). E=1 covers dense; E=N covers the stacked expert tower (qwen3_5_gguf_weights.cpp:1282). The seam's production consumers (qwen3_5.cpp:6147) reach a registered provider without new wiring. Capture compatibility is wave-3's committed obligation.

Revised budget: steady state 9.81 GiB experts packed + 6.01 GB non-expert packed + 2.37 GiB embedding twin + 2.37 GiB output twin ≈ 20.6 GB + activations on 32 GB. Interim after experts alone is ~33 GB, so the 27B gate is reachable only behind the dense E=1 arm — wave-2 scopes both in the same change.

[lu-zero]

Classification correction (tensor-name scan of the pinned artifact): the 27B is dense. Arch qwen35; zero ffn_*_exps / ffn_gate_inp / exp_probs tensors; blk.0 carries singular ffn_down/ffn_gate/ffn_up. The "experts Q4_K ×294" line in the earlier budget comments was a dtype-only misread — those 294 Q4_K tensors are dense ffn/attn weights.

Corrected 27B path: the whole keep-quant set beyond the gather class (~14.27 GB packed) cannot exist as twins (~42 GB). It is served packed through the grouped provider's E=1 arm with a chunked slice-decode + accumulate matmul, so the captured graph holds chunk buffers and never a whole-weight tile: ≈ 15.92 GB packed + 2.37 GiB embedding twin + 2.37 GiB output twin + chunk tiles + activations ≈ 22-23 GB on 32 GB. Replay re-decodes each step — correct, slower; throughput is #3031's lever.

Expert arm (E=N): stays staged-owed. It serves the family's MoE models (30B-A3B class); no MoE artifact is on disk and a ~17-20 GB download needs authority. Both waves of the provider (5e43d77, 14e8fe4) are landed on the row branch and operator-passed; wave-3a (E=1 chunked capture-compat) is next, then 27B wiring + gate + MTP skip + USAGE pin.

[lu-zero]

Wave-3b-1 landed on the row branch (f46d10283, mine @ row/BACKEND-TENSTORRENT-KEEPQUANT-W4; operator-passed on the committed HEAD: preflight green, backend suite 66/66 · 524,428 assertions, vehicle 147/147 · 16/16 prompts · 10 strict + 6 near-tie · 0 forward-divergent). This resolves the capture falsification chain opened earlier on this issue:

Capture: resolved by policy, not by arithmetic. The chunked E=1 arm's vehicle capture demanded 444,424,192 B against the 52,428,800 B region (mesh_trace.cpp:81) — ~113 keep-quant weights × ~3.5 MB/chain of serialized decode commands (the wave-3a per-chain constant), so chunk sizing cannot pay it down. The fifth spec amendment moves the trace region onto the pinned tt-metal's own practice (models/demos/utils/trace_region_sizes.py): dynamic (trace_region_size=0, the upstream default for unconfigured models and deepseek-v3's explicit choice), with VT_TT_TRACE_REGION_MB as the fixed-region opt-in. The fatal does not reproduce under either variant, and the gate now reports the measured demand (444,424,192 B) as the memory axis instead of assuming a carve-out.

The anchor flip was justified, not silenced. The packed path flips prompt[2] tok=5 (11→13). Mechanism: the per-chunk grouped matmul selects per_core_N=19 vs the twin path's per_core_N=71 — a different destination accumulation order over bit-identical bf16 operands. Gap evidence: the flipped cell is a 0.0-mnat exact tie, the re-derived battery holds 0 forward-divergent (max 375 mnats vs the 500 band), and the oracle artifact was regenerated byte-identical to rule out teacher drift. Goldens re-derived through the documented path (VT_DUMP_IDS=1 + qwen3-neartie-gap.py).

Accepted cost, recorded: packed replay re-decodes every step — 18.02 vs 2.95 s/cycle (~6.1×) on the vehicle. That is the spec's "correct, slower" state; #3031's int8-dot lever owes the recovery.

Next (wave-3b-2): 27B dense wiring on this arm, the blk.64.* MTP skip, the e2e gate vs the pinned llama.cpp b10451 oracle, and the docs/USAGE.md pin.

Two red-first device defects were fixed on the way: a chained ROW_MAJOR activation reaching ttnn::matmul unconverted (per_core_M = 0 at the mcast-1D config), and the E=1 arm committing ROW_MAJOR into the output slot so replay's tile-padded view overran the buffer (mesh_tensor_impl.hpp:33).