Escha-W2 port: 2-bit trellis decoded in the GEMV — Qwen3.6-35B-A3B and Qwen3.8-27B - #694
Open
nwoolmer wants to merge 101 commits into
Open
Escha-W2 port: 2-bit trellis decoded in the GEMV — Qwen3.6-35B-A3B and Qwen3.8-27B#694nwoolmer wants to merge 101 commits into
nwoolmer wants to merge 101 commits into
Conversation
Design for consuming EschaLabs' 2-bit trellis format natively in hipfire, targeting Qwen3.6-35B-A3B-Escha-W2 then Qwen3.8-27B-Escha-W2. Both models land on existing arch ids 5/6 — no architecture port. New: quant types ESCHA2T16=42 / ESCHA3T16=43, RotationPlan::EschaH128, a converter, and kernels in two phases (decode-at-load to Q8_0 first, fused decode+GEMV second). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Second review pass, checking the design's factual claims rather than re-reading them. Verified: ref.py's reconstruct reproduces both committed goldens bit-exactly at the documented tile grid, for K=2 and K=3; the goldens are byte-identical to the shipped layer-0/expert-0 tensors, so they gate against real model data. Base identity confirmed both ways (35B differs from Qwen3.6-35B-A3B only in transformers_version). DType::Q8_0 and weight_backend::dequant_f32 confirmed to exist. Corrected: gate_up's escha_rout is not a plain sign x scale vector. It carries a per-expert channel prune mask — 55% of layer-0 expert 0's output channels are exactly zero, gate and up halves masking the same 280 of 512 intermediate channels. Rate varies 14-58% per expert and the masks are not shared. Adds a Phase 2 optimisation (down_proj can skip those input rows) and a correctness trap (pruned outputs must stay exactly zero). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Sampled the parts of the models the earlier passes never looked at, and checked whether the gates can actually be executed here. - 27B coverage table omitted the full-attention layers entirely. Layer 0 is linear_attention, so sampling it alone hides that self_attn q/k/v/o are escha-coded at K=2 across 16 layers. Real total is 10 projections over 64 layers = 400 escha tensors, and dispatch needs the 3-way FusedQkvQ8_0 path as well as the 4-way QKVZA one. - G5 could not be run. escha-mlx is Metal, the escha wheel is CUDA, ZML needs an NVIDIA driver — no Escha runtime executes on gfx1151. Re-anchored to escha-ref on CPU, which ref.py declares to be the semantic contract for their kernels, plus a second KLD against the bf16 parent. - Fourth metadata trap: `ignore` means "not escha-coded", not "not quantized". Both models list embed_tokens and lm_head there and ship them as weight_int8 regardless. - int8 -> Q8_0 is lossless only by replicating the per-row scale into every 32-element block; recomputing block scales adds avoidable error. - Sizing redone: 36.7 GB not ~35, expert bandwidth 3.6x not 3x. Both reconstructions land within a few percent of the published on-disk sizes, which independently confirms the format model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Read Escha's test suite rather than only their reference implementation. Their tests state a leaf contract the design had partly wrong. - escha_config is OPTIONAL, not guaranteed. The design keyed K off escha_config[1]; an export without the end-to-end stage ships no s_in/s_out/config/bias at all and must still load. K now comes from the code tensor's own shape (last dim = 16K), which cannot disagree with itself, with escha_config and layer_meta as cross-checks when present. - Required leaves are exactly escha_code/escha_rin/escha_rout. Missing any is a hard error — their test docstring says "fail loudly at load, not decode into noise". - Unknown escha_* leaves must be rejected by name. Their example, escha_rotation_theta, shows the format anticipates a Givens-style rotation variant today's checkpoints do not use. Out of scope, but it must stop conversion rather than decode under the wrong rotation. G1 now asserts the contract in both directions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Executed the moeblk golden against real shipped layer-0 weights (55 routed experts fetched by range request) instead of describing it. - The MoE-block golden is NOT bit-exact. ref.moe_block against it gives max|diff| 1.22e-4, mean 2.1e-6, 4752/16384 values differing at ULP level on outputs of mean magnitude 0.0185 — it was produced by the Metal path, not ref.py. G4 now carries a concrete tolerance (max 2e-4 / mean 1e-5) instead of "within fp16 rounding", and the bit-exact codec goldens are explicitly excluded from it. - The fixture injects moeblk_ids/moeblk_scores, so it bypasses the router entirely. G4 never gated selection. Added G4b, which does: reproducing selection from mlp.gate.weight gives the identical top-8 set on all 8 tokens with scores agreeing to 3e-8. - Router logits are rounded to f16 BEFORE top-k, which manufactures exact ties f32 would not produce — one occurs in 8 tokens. Ties inside k are harmless (the combine is a sum), ties at the k boundary change the selected set and are a legitimate source of hipfire vs escha-ref divergence at G5. The design's "router untouched" claim is now flagged for verification rather than assumed. - ref.moe_block re-decodes each expert per (token, slot); memoize reconstruct or the reference is unusably slow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
The id rationale was derived from the QuantType enum in the loop/gfx1151 checkout, not from origin/master which this branch targets. On master the authoritative registry is hipfire-quantize/src/hfq.rs (enum + from_u8, kept in sync by contract), and it is much further along: 38/39 are MQ2G256GL/MQ3G256GL and 40/41 are TQ2G128/BQ1G128 (Bonsai ternary/binary), all merged, plus 44/45/47-51. 42/43 survives as the lowest free pair, so the decision stands — but the stated reasoning was wrong and would have misled the next person. Also flags that Neutrino's reservation of 40/41 for FV5 is now stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
11 TDD tasks from the CPU oracle through to a served model, each ending in an independently testable deliverable gated at G0-G5. Scoped to Phase 1 / 35B only. Phase 2 (fused decode+GEMV, plus the prune-mask optimisation) and the Qwen3.8-27B dense model each get their own plan — neither produces working software on its own here. Self-review caught three issues, fixed inline: the int8 -> Q8_0 row-scale replication from design §4.2.1 had no task at all; the H128 host wrapper said "mirror Task 7" instead of showing code; and Task 10 referenced Gpu::escha_decode_tiles when only the _host variant is defined. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Pre-flight scan of the plan. The Task 6 verifier searched for each code tensor as a substring of the whole .hfq — quadratic over a 12 GB file, 80 times — and its missing-vs-mismatch classification tested for the tensor NAME in the blob, so it could not distinguish the two cases. It also carried a half-written index parser it never called. Replaced with a real HFQ index parser that memcmps each tensor at its indexed offset and checks the quant_type is 42/43. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
- Codec logic is implemented twice (Rust oracle + HIP kernel) on purpose: that duplication IS the G2 gate. Generating either from the other makes the gate circular. Rationale now documented in both files so reviewers stop re-raising it. - Task 9 was an example asserting on a hand-built array while asking the implementer to READ the router — a test that does not exercise the system under test. Rewritten to call the production arch-6 router on the fixture and assert the selected top-8 SET matches escha's shipped ids, which answers the f16-rounding question by observation. - Task 10 stays prose-specified; it is integration against code not in context and will be dispatched on a capable model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Ports EschaLabs escha-mlx's cbA trellis codec (K=2/K=3, 16x16 tiles) to a pure-Rust CPU reference in hipfire-quantize::escha_ref. This is Task 1 of the Escha-W2 port: the numerical oracle every later GPU kernel is gated against, so it must be bit-exact against escha-mlx's own golden vectors. Fixes a real defect surfaced by TDD: cba_decode's final fp16 add must be round-to-nearest-even per the published codebook, but crate::float16::f32_to_f16 truncates by design (documented, to keep existing HFQ encoder output byte-stable). Using it produced a 1-ULP miss on the very first published constant (state 3: 0x3ab7 vs the correct 0x3ab8) and failed both golden-tensor SHA-256 checks. Routes the single RNE-sensitive rounding through half::f16::from_f32 instead — half is already a workspace dependency used inside float16.rs, so this adds no new dependency, just bypasses the crate's non-RNE convenience wrapper at the one call site where RNE is the spec. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 1 found by TDD that crate::float16::f32_to_f16 truncates — the module doc says so explicitly, to keep existing HFQ bytes stable — while every f16(...) in the escha contract is round-to-nearest-even. Truncating misses the published cbA constants at states 3, 6 and 7. Tasks 2, 3 and 5 all repeated that call and would have reintroduced the bug. Swept 9 call sites onto a new escha_ref::f16_rne helper, corrected Task 1's now-stale brief text, and added the rule to Global Constraints so later tasks inherit it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
…xed cases input_transform/output_transform had no direct unit test despite being the exact contract the GPU kernels get gated against. Add tests that pin the scale-before-vs-after-H128 ordering (with expected/wrong values built independently to prove the chosen inputs discriminate the two orderings) and the per-channel rin/rout broadcast across multiple rows via .iter().cycle(). Also round out fold_scales_handles_absent_scales with the two mixed (s_in only, s_out only) cases alongside the existing all-None/all-Some corners. Test-only change; verified by temporarily swapping the scale order inside input_transform, confirming the new tests fail, then reverting.
The title promised a MoE block that neither the interfaces nor the tests build. G4 gates hipfire against Escha's shipped moeblk_out golden directly, so a Rust moe_block would be a second thing to keep in sync while gating nothing. Design §4.1's oracle list never included it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
The accumulation loop treated a zero activation as a no-op shortcut, but this module is the numerical oracle GPU kernels are gated against and 0.0 * NaN = NaN under IEEE-754: skipping the multiply silently turns a corrupted (non-finite) weight into a clean zero instead of surfacing it. Removed the skip so expert_linear is an unconditional matmul, and added a test that poisons one weight with NaN behind an all-zero activation and asserts the output is NaN, not zero. Verified the test fails with the skip restored before removing it again. Also corrected the doc comment on swiglu_uses_gate_first_half: its first sub-case (gate=[0,0]) only pins silu(0) == 0 and does not by itself discriminate a gate/up swap; the second sub-case is what actually catches a swap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Review found the skip diverges from a plain matmul when a weight row is non-finite (0.0 * NaN = NaN, not 0). escha_ref is the oracle GPU kernels are gated against, so swallowing that would mask a corrupted decode rather than surface it. Plan text now matches what shipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
…plan Registers the two Escha-W2 quant type ids (hfq qt=42/43, the lowest free pair) in the authoritative QuantType registry, adds DType::Escha2T16/ Escha3T16 to the runtime dtype enum, and adds RotationPlan::EschaH128 so dtype_rotation_plan routes both types to it instead of falling through to RotationPlan::None. Escha weights live in a rotated domain (128-point Hadamard on both sides of the matmul); reaching an unrotated Plain GEMV would not crash but would silently produce fluent, wrong output, so every exhaustive match the new variants broke got an explicit named arm rather than a widened catch-all, and prepare_rotation_scratch now hard-errors on RotationPlan::EschaH128 (no rotate kernel exists yet). Guarded by escha_types_never_resolve_to_plain / escha_types_use_the_escha_rotation_plan in hipfire-dispatch-tests and escha_quant_types_round_trip in hfq.rs.
…haH128 gemv_steps arm Escha2T16/Escha3T16 previously claimed ArchPredicate::Always for their GEMV arch gate. That's inert today (no gemv_table.rs registration, no reachable call site) but is an affirmatively wrong claim with no compiler signal to revisit: a future kernel registration would silently advertise availability on every architecture. Add ArchPredicate::Unimplemented, which eval_arch maps to false on every arch, and gate the escha types on it instead, so a future registration fails closed with an explicit dispatch error. Also give gemv_steps an explicit RotationPlan::EschaH128 arm instead of letting it fall into the RotateFwht catch-all, which would mislabel the 128-point Hadamard as an FWHT rotate. The arm panics with an explanation rather than guessing a step list, since no kernel exists yet and the path is unreachable (for_gemv already rejects Plain for Escha dtypes).
gemv_steps had an explicit panic!() arm for RotationPlan::EschaH128. It's
unreachable today (registration dtype lists in gemv_table.rs exclude Escha),
but the call path (GemvFamily::new -> populate -> register_*) runs on a real
per-decode-step path, so a future accidental call would crash a serving
process instead of failing gracefully. for_gemv, evaluated one line above
each gemv_steps call in the same registration loops, already models this as
Result<_, DispatchError> with the `let Ok(..) else { continue }` idiom.
Convert gemv_steps to return Result<&'static [PipelineOp], DispatchError>:
every existing arm now returns Ok(<same slice>) unchanged, including the `_`
catch-all; the EschaH128 arm returns an explicit UnsupportedVariant error.
Updated the four call sites in gemv_table.rs to mirror the existing
for_gemv idiom, and the one test caller in tests.rs. Added a test asserting
Err for both Escha dtypes and Ok (unchanged step lists) for a rotated and
an unrotated dtype; verified it fails if the Escha arm is reverted to Ok.
…rced Adds pipeline_escha.rs: classify_leaf/k_from_code_shape/quant_type_for_k/ check_linear_complete plus convert_escha, which turns an Escha-W2 checkpoint directory into a single .hfq. K is derived from escha_code's own shape (never escha_config or layer_meta.bits, both unreliable per the port design doc); classification keys off the tensor suffix so ignore-listed but still-quantized tensors (embed_tokens, lm_head as weight_int8) are not dropped; missing escha_rin/rout/code hard-errors instead of silently decoding into noise; and int8_rows_to_q8_0 replicates each row's scale into every Q8_0 block so the int8 payload round-trips bit-identical to escha's w8a16 rather than being re-quantized. Wires a CLI arm (--format escha/escha-w2/eschamoe) into pipeline.rs's handle_early_special_formats, following the existing per-format dispatch pattern, so convert_escha is reachable rather than dead code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Task 5 review caught that the brief's metadata snippet omitted the top-level "config" key. hipfire-arch-qwen35's config_from_hfq errors "qwen35: missing config" before reading any tensor, so every file the converter produced would have been unloadable — and Task 6 would have burned a 12 GB conversion discovering it. Fourth plan-authored defect the review loop has caught. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
… convert_escha test
Finding 1 (critical): convert_escha's emitted metadata was missing the
top-level "config" key, so every .hfq it produced failed to load —
hipfire-arch-qwen35's config_from_metadata_json requires meta["config"]
before reading a single tensor. Embed the already-parsed config.json
Value verbatim, matching pipeline_gguf.rs's metadata shape.
Finding 2 (important): replace .unwrap() with .ok_or_else(...)? on the
escha_rin/escha_rout lookups, the Int8 passthrough lookup, the generic
passthrough lookup, and the name.rsplit_once('.') calls, matching this
file's Result convention. None of these are reachable today, but a
panic mid-conversion is a worse failure mode than a clean error.
Finding 3 (minor): add an in-module integration test that builds a
minimal synthetic checkpoint (config.json + one safetensors shard with
one complete escha linear and one int8 pair), runs it through
convert_escha, and asserts the round-tripped metadata carries "config"
and that escha_code survives byte-identical. Verified empirically that
the test fails against the pre-fix metadata shape and passes once
config is embedded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Both wrappers called a fictional Gpu::launch_kernel(name, grid, block, &[&buf,...]). The real convention, used throughout gemv.rs (gemv_q8_0 at 13563), is: bind_thread -> upload_raw/alloc_tensor -> ensure_kernel -> build a Vec<*mut c_void> of &mut locals -> look the function up in self.functions -> unsafe hip.launch_kernel. Rewritten to match, and verified memcpy_dtoh / bind_thread / alloc_tensor / upload_raw all exist with the signatures used. Same class as the quant-id error: anchors taken from the other checkout rather than this branch. Line references re-verified here (kernels.rs 4810 and gemv.rs 13563 correct; dispatch.rs spec push is 4263, not 4251). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Converted the real 12.30 GB Qwen3.6-35B-A3B-Escha-W2 checkpoint in 8.6s. G1 PASS: all 80 escha_code tensors (40 layers x 2 projections) are byte-identical in the .hfq and carry quant_type 42/43. Tensor accounting is exact — 1362 source tensors map to 870 outputs with nothing dropped. The verifier's HFQ index parser was pre-validated against two unrelated existing models; on both, the last tensor's end offset lands exactly on the file size. escha_config_smoke proves Task 5's critical metadata fix works: the file loads as arch_id 6 with dim=2048 layers=40 experts=256 top_k=8 moe_inter=512 vocab=248320 is_vl_text=true, all matching design §1.1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Task 7 (G2): one-shot decode of Escha-W2 packed trellis tiles into bare fp16 weights on gfx1151. Inline codebook arithmetic only (no LUT, would be 128 KB fp16 vs 64 KB LDS); __hadd kept as the fp16 RNE add so the GPU path is bit-exact against escha_ref::reconstruct. Gate passes 0/2097152 mismatched (K=2) and 0/1048576 mismatched (K=3) against the real shipped layer-0 expert-0 golden tensors.
…ascade indexing, add device-resident path Review fixes for Task 7's escha_decode_tiles GPU kernel: - Header comment claimed Q8_0 output with a transpose; the kernel actually writes bare row-major fp16 with no transpose. Rewrote to match reality. - escha_decode_tiles_host had no input validation, so a short code slice or a non-multiple-of-16 shape caused an out-of-bounds device read. Added shared validation (shape, K, code length) returning Err before any launch. - words[24] was indexed at runtime from lane-derived offsets. Disassembly of the compiled .hsaco showed no scratch-memory spill, but the compiler lowered the dynamic index into ~90 v_cndmask/94 v_cmp select-cascade instructions per invocation instead. Replacing the staged array with two direct loads of the exact words each lane needs removed the cascade entirely (467->232 static instructions, 34->25 VGPRs) and measured a consistent ~1.48x kernel throughput improvement (24.2us -> 16.25us per launch, K=2 2048x1024, 3 runs each side). - Added a device-resident escha_decode_tiles(&GpuTensor, &GpuTensor, ...) for the load path (no host round trip); escha_decode_tiles_host now calls it instead of duplicating the launch. Gated with a device-resident-vs-host equivalence check. - Removed the dead `lane >= 32` guard (block size is always 32). - Widened the G2 gate with two production-scale shapes from the dense 27B checkpoint (5120x17408 K=2, 17408x5120 K=3, pseudo-random code) to prove the tile/lane indexing generalises past the two small golden fixtures. G2 still passes 0/0 mismatches for both K after every change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
…eference Naive one-thread-per-block butterfly (128 threads stage into LDS, thread 0 does all 7 stages serially), plus a device-resident escha_h128 dispatch and host-roundtrip wrappers for the G3 parity gate and a throughput benchmark. Bit-exact against escha_ref except for one HIP toolchain quirk: __float2half(-0.0f) on this ROCm build returns +0.0 (0x0000) instead of the sign-preserving -0.0 (0x8000). escha_ref's f16_rne (half::f16::from_f32) does preserve the sign of an exact zero, and output_transform's final multiply by a pruned rout==0 channel can land on exactly -0.0 when the pre-scale value is negative. Worked around with f2h_rne(), which special- cases exact-zero inputs (bits >> 16 gives the correct signed fp16 zero) and otherwise defers to __float2half, which is bit-exact for every other value tested including fp16-underflowing values of both signs. G3: h128_in 0/2048 mismatched, h128_out 0/2048 mismatched, pruned channels (idx 7, 1000) exactly zero. Task 7's G2 gate re-run clean.
The naive kernel had thread 0 execute all 7 butterfly stages alone while 127 threads idled. Replace with a ping-ponged (bufA/bufB) parallel butterfly: every thread owns one element and computes exactly one add or subtract per stage, syncing between stages. Ping-ponging (rather than updating in place) avoids a read-after-write race between the "low" and "high" thread of a pair, and each pair's result depends only on its own two pre-stage operands, so the schedule change cannot alter any add/sub's inputs or rounding. Verified bit-exact against escha_ref after parallelising (G3: 0/2048 mismatched for both transforms, re-run 3x) and Task 7's G2 gate still passes. Throughput (bench_escha_h128, n=2048, host-side upload done once, kernel-only loop): naive ~3.72 us/launch -> parallel ~2.35 us/launch, ~1.6x, measured stable across repeated runs.
…H128 bench Review fixes for Task 8, no kernel arithmetic changed: - The comment in escha_h128.hip overstated the toolchain defect as "__float2half(-0.0f) returns +0.0" without qualification. A reviewer disproved the general claim (a bare -0.0f literal converts correctly, both as an immediate and loaded from memory) and reproduced the actual, narrower trigger: the sign is lost only for a runtime multiply-produced exact zero. Reworded the comment with the reviewer's repro table so a future auditor checking another __float2half call site knows the bare -0.0f test will NOT discriminate the bug. - Added a note that the f2h_rne workaround is required only for bit-exactness against the escha_ref CPU oracle, not for correctness: -0.0 == 0.0 under IEEE-754 and nothing downstream divides by an activation, so the pruned-channel contract holds with either sign. - Rewrote bench_escha_h128.rs to add a no-op-kernel overhead floor, sweep production channel widths (2048/6144/17408), and re-measure the naive-vs-parallel speedup at those widths via a benchmark-only copy of the pre-parallelisation kernel. The original 1.58x figure holds (it's the smallest speedup in the swept range, not the largest — it grows to ~3x raw / ~8x overhead-subtracted at n=17408). Both gates re-verified bit-exact after these changes: test_escha_h128_gpu_vs_cpu (G3 PASS) and test_escha_decode_gpu_vs_cpu (G2 PASS, wide-shape PASS, equivalence PASS). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
`HIPFIRE_ESCHA_FOLD=mq6|mq4v2` bakes the H128 rotations and both diagonals
into each escha linear and emits an ordinary `{proj}.weight`, so the
runtime needs no escha awareness in the forward pass. Default is off and
the unfolded output is unchanged — verified the 35B still converts
byte-identically to what is published.
FIRST WORKING 27B. Converted, loaded and ran:
22.63 GB, 400 MQ6G256V2 tensors (24.3 G elems), 2 Q8_0 (embed/lm_head)
loads in 8.6 s, 64 layers, non-finite 0
H128 launches = 0 per token — the fold really did remove the runtime
transform, which is the whole point
prefill 25.9 tok/s, decode 9.6 tok/s at n=64
PPL 13.9142 (wikitext slice, 384x4, kv f32)
Conversion is 6m26s on 25 threads, dominated by two Hadamard passes over
24 G weights.
THE PPL IS NOT SHIPPABLE AND THE REASON IS KNOWN. A bias is additive; no
weight matrix absorbs it. `WeightTensor` has no bias slot, so a folded
model with biases in the file has them silently ignored at load — the same
failure class as the MTP head this converter was dropping two commits ago.
Measured on the 27B the bias is ~1.3% of a projection's output magnitude:
small per layer, compounding over 64 of them.
So fold mode now REFUSES to emit when biases are present, unless
`HIPFIRE_ESCHA_FOLD_DROP_BIAS=1` says the caller wants that artifact
anyway — which is legitimate for measuring the fold in isolation, and is
how the 13.91 above was produced. The artifact already built stays valid
as a measurement; it is not a candidate for publication.
Next: a bias slot in the runtime, then re-measure. 13.91 should not be
read as the format's quality — it is the format minus its bias correction.
Tests: hipfire-quantize 66 passed.
Escha's dense export carries an additive output bias on every coded
projection — base Qwen3.8-27B has `attention_bias: false` and no MLP bias,
so these are purely Escha's end-to-end correction. They cannot be folded
into a weight (additive), and the folded 27B was silently ignoring them:
PPL 13.91 instead of the format's real quality.
Ten distinct biases across the two layer families, enumerated from the
checkpoint index rather than assumed:
48 DeltaNet layers in_proj_qkv, in_proj_z, out_proj, mlp.{gate,up,down}
16 full-attn layers self_attn.{q,k,v,o}, mlp.{gate,up,down}
`in_proj_a`/`in_proj_b` have none — escha's `ignore` list keeps them plain.
`WeightBackend` gains `bias_opt`. The existing `bias` PANICS on a missing
tensor, which is right for qwen2 where it is mandatory; escha makes it
optional per the leaf contract (§1.4), so that path needs absence to be a
value. ParoBackend returns None rather than an error — those checkpoints
simply have no biases.
Biases live on the layer struct, NOT on `WeightTensor`: that type has 126
construction sites, and the layer structs have two each.
One probe decides per layer, then the rest are MANDATORY via `need_bias`.
Escha ships a layer's biases together or not at all, so a half-biased
layer is a corrupt checkpoint — substituting zeros there would degrade
output without failing, which is the same trap as the dropped MTP head.
Loading only; the adds are next. Nothing consumes `biases` yet, so
behaviour is unchanged for every model including the 27B.
Tests: hipfire-arch-qwen35 199, hipfire-runtime 597+12, all pass.
Decode (`forward.rs`) and the batched DeltaNet prefill (`prefill.rs`) now
add the escha dense export's per-projection output biases.
Placement is the whole point. Every op has several branches filling the
same buffers (prerotated / scalar-prep / execute-steps / fp4 / fused-Q8),
so the adds go at the ONE point each chain converges, never inside a
branch — a bias applied in some branches and not others is a silent wrong
answer, not a crash. In decode that is the single `res` exit of `run_proj`
and `run_residual_gemv`, keyed on opcode.
Ordering matters twice over:
- gate/up biases land BEFORE SwiGLU. After the activation would be a
different function, not a rounding difference.
- out_proj/down_proj biases land on the residual stream. After the
residual add is the same value as before it — both additive — so the
post-hoc add is exact, not an approximation.
Uses `bias_add_f32(x, bias, batch, n)`, which already existed, for BOTH
decode (batch=1) and prefill (batch=n), so the two cannot drift.
Measured on the 27B, wikitext 384x4, kv f32:
no biases 13.9142
biases applied 13.6957
KNOWN INCOMPLETE, stated because it is not visible from the diff: the
full-attention QKV/o_proj/FFN biases added to `batch_chunk_full_attn_attn`
and `batch_chunk_full_attn_ffn` are NOT REACHED. A probe in that function
never fired, and adding them moved PPL by exactly zero. The batched arm
there is gated on `fa_batched_ok`, yet a rocprof trace shows
`gemm_qkv_mq6g256v2_wmma` running 16 times per pass — once per
full-attention layer — so those layers do execute a batched QKV through
some other route I have not located. The code is correct where it sits and
harmless where it does not run; the 16 FA layers of 64 are still missing
their biases on that path.
Tests: qwen35 199, quantize 66, runtime 597+222, all pass.
Adds the q/k/v, o_proj and gate/up/down biases to both full-attention
prefill routes — the batched `batch_chunk_full_attn_attn`/`_ffn` and the
per-token `run_fa_layer_body` behind `batch_chunk_full_attn_fallback`.
Placement follows the DeltaNet side: at the point each branch chain
converges, q/k/v BEFORE the Q/gate deinterleave and q_norm, gate/up BEFORE
SwiGLU, o_proj/down onto the residual stream where the add is exact rather
than approximate.
VERIFIED EXECUTING. A probe at the batched FA bias site fires under
`escha_prefill_bench` with `biases=true`, and prefill argmax moved from
220 (logit 6.14) to 248046 (logit 8.39) on the same artifact.
Two earlier probe failures made this look unreached, and both were
artefacts of what was being run rather than missing code — recorded so the
next person does not re-derive them:
- `build_kld_ref_native` scores TOKEN-BY-TOKEN through the DECODE path
(~4-5 tok/s, matching decode not prefill), so no prefill probe can
fire under it. The measured 13.9142 -> 13.6957 therefore came from the
`forward.rs` op-dispatcher biases, which already covered all 64 layers
including full-attention via PROJ_QKV.
- `run_fa_layer_body` legitimately does not run: `fa_batched_ok` admits
these layers, so the batched arm is taken and the fallback is dead
code for this model. Its biases are there for the models that do take
it.
There is no third route; an earlier commit message speculating about one
was wrong.
Coverage now: decode all 64 layers (measured via PPL), DeltaNet prefill
(observed via argmax), full-attention prefill (probe-confirmed).
Tests: qwen35 199, quantize 66, runtime 597+222, all pass.
`escha_h128_batched` and `escha_swiglu_batched` launched with a raw `launch_kernel(.., None, ..)` — the NULL stream — while every GEMV around them goes through capture-aware `launch_maybe_blob`. Both run INSIDE the forward pass, so under graph capture the null-stream work is not ordered against the captured stream. A genuine data race. It surfaced as the G5 gate failing its own negative control: the reference arm scored against its own reference must print 0.000000, and instead two runs of the SAME binary against the SAME reference gave 0.000361 and 0.000152. The gate correctly refused to attribute any KLD to the codec, so escha quality has been ungated since. Bisected rather than guessed: HIP_LAUNCH_BLOCKING=1 -> 0.000152, 0.000152 deterministic HIPFIRE_GRAPH=0 -> 0.000152, 0.000152 deterministic Both point at capture, and only at capture; an earlier bisect had already cleared the `k >= 128` wide-GEMV threshold change. After the fix, capture ON and no blocking: 0.000152 three runs running. This also explains an unrelated oddity measured earlier in this work — graph capture benchmarked as worth only 0.4% on escha. Two of the per-token kernels were never being captured. Not fixed here: the control is deterministic but still 1.5e-4 rather than 0.000000, so something else keeps the f16 arm from reproducing its own reference bit-for-bit. That is now a tractable single question instead of noise on top of noise. Regression: escha-35b MQ6 728.9 tok/s prefill / 57.2 decode (was ~702 / ~55.6). rdna-compute 243, qwen35 199, dispatch 212, all pass.
The 27B's value is 2-bit RESIDENCY (11.16 GB), and the fold throws that away for a 22.63 GB dense model. This is the path that keeps it: the trellis code stays verbatim and is decoded INSIDE the GEMV. No new kernel. Every escha GEMV is expert-INDEXED, and rather than fork the trellis inner loop, a dense linear is served as the degenerate one-expert case — `expert_ptrs = [&code]`, `ids = [0]`, `slots = 1`. That reuses the kernel G2 already gates bit-exact against the oracle instead of creating a second thing to keep bit-exact. `EschaDenseLinear` gains a one-element device pointer table, built only for `EschaWeightStore::Native`. Verified against `escha_ref` on the real 27B, every dense projection shape and both K: proj K rel_rms (Native) linear_attn.in_proj_qkv 2 1.765e-5 linear_attn.in_proj_z 2 2.730e-5 linear_attn.out_proj 2 4.511e-5 mlp.gate_proj 2 2.858e-5 mlp.up_proj 3 1.752e-5 mlp.down_proj 3 3.272e-5 Native is MORE accurate than the F16 store (2.858e-5 vs 1.033e-4 on gate_proj): both deliver exactly-decoded weights, but native goes straight into fp32 accumulation with no f16 store round-trip. WHY THIS IS WORTH FINISHING, measured on the same slice and harness: qwen3.8-27b.mq6 21.75 GB PPL 11.0092 escha-W2 codec 11.16 GB PPL 11.8654 qwen3.8-27b.mq3 12.62 GB PPL 12.3203 escha folded -> MQ6 22.63 GB PPL 13.6957 Against the size-comparable baseline escha is 1.46 GB SMALLER and 0.46 PPL BETTER, and it reaches within 0.86 PPL of a 6-bit quant at half the bytes. The fold's re-quantisation costs 1.83 PPL — more than the codec itself — which is why the fold has no product case and this path does. Still to wire: the layer forward must call `escha_dense_linear_forward` for native layers instead of the fused MQ paths, since a trellis weight is not something FusedQkv/gate_up can consume. The primitive and the GEMV are both gate-verified; what remains is routing. Tests: hipfire-arch-qwen35 199 passed.
The gate now reports us/call and achieved bandwidth alongside accuracy, so the question "is the native trellis path fast enough to justify wiring it into the layer forward" has a measured answer before that work starts. mlp.gate_proj (ic=5120 oc=17408 K=2), 200 iterations after 10 warmup: store us/call weight bytes achieved BW rel_rms Native 258.1 22.3 MB 86.3 GB/s 2.858e-5 Q8_0 424.1 94.7 MB 223.3 GB/s 5.157e-3 F16 2427.8 178.3 MB 73.4 GB/s 1.033e-4 mlp.down_proj (K=3) is the same shape of result: Native 250.7 us vs Q8_0 431.8 us. Native wins on all three axes against Q8_0 — 1.7x faster, 4x fewer bytes, 180x more accurate — because the bandwidth saving beats the decode cost. BUT it is NOT bandwidth-saturated: 86-133 GB/s against Q8_0's ~220 GB/s on the same part. The trellis decode is VALU-bound, exactly as the 35B's expert GEMV measured at 17.5 instructions per weight. Extrapolated, a native 11.16 GB model lands near ~10 tok/s decode — about the same as the 22.63 GB folded MQ6, at half the memory and better quality, but slower than a bandwidth-bound plain quant of similar size (qwen3.8-27b.mq3, 12.62 GB, would be nearer ~17 tok/s while scoring PPL 12.32 against escha's 11.87). So the native path's trade is better quality and half the footprint for ~1.7x slower decode. Recording that here because it is a product choice, and it is cheaper to know now than after the layer-forward routing. Tests: hipfire-arch-qwen35 199 passed.
`load_weight_tensor_raw` handled quant types 6-31 and stopped there, so
the generic loader could not produce a trellis `WeightTensor` at all —
`b.proj()` on an escha dense projection had nothing to return. That was
the foundation gap under the native path.
Code is kept VERBATIM, decoded inside the GEMV. Opaque raw buffer like the
MQ arms, but the resemblance ends there and the comment says so: an escha
weight is NOT self-contained. It needs its `escha_rin_eff`/`escha_rout_eff`
vectors and an H128 on BOTH sides of the GEMV, which is why the fused MQ
paths cannot consume one — each projection needs its own rin-rotated
activation, so FusedQkv/FusedQkvza/gate_up have nothing to share.
This unblocks, but does not complete, the native route. Remaining, scoped
precisely:
1. per-layer escha metadata (rin/rout/ptr0) alongside `biases`
2. loader populates it when a projection's dtype is Escha2T16/3T16
3. forward.rs + prefill.rs branch to `escha_dense_linear_forward` at the
~14 projection sites the bias work already mapped
Only (3) carries real risk: the fused paths must be bypassed wholesale for
escha layers, and a partial bypass is a silently wrong model rather than a
crash. There IS a gate for it — a native 27B should score PPL ~11.87,
matching the f16 fold, since the weights are identical.
Tests: hipfire-arch-qwen35 199 passed.
`EschaProj` holds what a trellis weight needs beyond its `WeightTensor`:
the two rotation vectors and the one-element pointer table. The weight
itself stays a `WeightTensor` (dtype Escha2T16/3T16, buffer = verbatim
code) so every existing `layer.wqkv.gpu_dtype` check keeps working.
`EschaProj::forward` runs H128-in -> trellis GEMV -> H128-out for `slots`
tokens at once: 1 for decode, n for batched prefill. The indexed GEMV
serves a dense linear as `slots` copies of expert 0, with `ids` a
slots-long run of zeros and x_group PerSlot above 1.
Bias is deliberately NOT applied here — the existing per-op bias path owns
that, so exactly one place knows bias ordering.
Verified on the real 27B before any wiring, because a wrong slot stride in
batched prefill is per-token garbage that only surfaces as a bad PPL much
later:
store=Native (slots=1) rel_rms 2.858e-5 259 us/call 86 GB/s
BATCHED slots=1 worst_rel 1.526e-3
BATCHED slots=4 worst_rel 1.526e-3 (identical across slots,
as they must be — same x)
`load_escha_proj` returns None for any non-trellis dtype, which is the
signal a layer takes its ordinary path.
Remaining for the native route: per-layer fields holding these, loader
population, and the forward/prefill branch at the ~14 projection sites the
bias work mapped.
Tests: hipfire-arch-qwen35 199 passed.
`DeltaNetEscha`/`FullAttnEscha` hold an `EschaProj` per coded projection plus the shared `ids` table, populated by the loader when a projection's weight is `Escha2T16`/`Escha3T16`. The weight itself stays a plain `WeightTensor` holding the verbatim code, so every existing `layer.wqkv.gpu_dtype` check keeps working and the `escha: Option<..>` field is the single signal that a layer must bypass the fused MQ paths. `WeightBackend` gains `escha_sidecars` (returns the `(rin, rout, ptr0)` triple — a tuple rather than a typed struct because hipfire-runtime must not depend on the arch crate that owns `EschaProj`) and `zeros_i32` for the ids table. ParoBackend returns None/zeros; those checkpoints are not escha. Both constructor arms now bind coded weights to locals before building the struct, because the sidecars are keyed off each weight's dtype and device pointer and so must be built from the loaded tensor. One probe decides per layer, then the rest are MANDATORY via `need_eproj` — same rule as the biases. A half-escha layer is a corrupt checkpoint, and falling back per projection would silently run some of them through the fused MQ paths on trellis bytes. Decode scratch gains `escha_xh`, sized to the largest `ic` any projection uses. It cannot share `x`/`tmp`: each escha projection rotates the SAME input with its OWN rin, which is precisely why the fused paths cannot serve a trellis layer. `mid` needs no buffer — `escha_h128_out_batched` stages into LDS and syncs before writing, so it is safe in place and the projection's output tensor serves as both. Loading only. Nothing reads `layer.escha` yet, so behaviour is unchanged for every model. The forward/prefill branch is next. Tests: qwen35 199, runtime 597+12, all pass.
The 2-bit model now executes with the trellis code resident and decoded
inside the GEMV. No fold, no decode-at-load.
loads in 3.74 s at 11.16 GB (vs 8.60 s at 22.63 GB folded)
decode 11.1 tok/s (vs 9.7 folded — faster at HALF the memory)
PPL 11.8377 non-finite 0
THE GATE PASSED. Predicted ~11.8654 from the f16 fold, which carries
identical weights; measured 11.8377, a 0.23% difference in the direction
accumulation order predicts — the native GEMV goes straight to fp32 with
no f16 store round-trip, and measured more accurate per-projection
(2.9e-5 vs 1.0e-4). That agreement is what says the routing is right
rather than merely finite.
`escha_run_proj` / `escha_run_resid` intercept the op dispatcher and
bypass the fused MQ paths WHOLESALE for an escha layer. They have to: each
projection rotates the same normed input with its own `rin`, so
FusedQkv/FusedQkvza/gate_up have nothing to share and cannot read a
trellis code anyway. They also do their own plain RMSNorm rather than the
fused rmsnorm+rotate — a pre-rotated input would be rotated twice.
`in_proj_a`/`in_proj_b` stay on the ordinary GEMV; escha's `ignore` list
keeps them uncoded.
Two supporting changes:
- `qwen35_tensor_name_candidates` offers `{stem}.escha_code` as an alias
for `{stem}.weight`, LAST in the list so a checkpoint with both
resolves to the real weight rather than silently preferring a code.
- `escha_h128_batched`'s `out` check is `>=` not `==`. The kernel writes
exactly `slots*n` and never reads `out`, and the dense path's `xh`
scratch is sized to the largest `ic` so one buffer serves every
projection. Undersized is still fatal.
PREFILL IS NOT YET ROUTED: 12.0 tok/s at n=512 with 800 H128 launches per
token, i.e. falling back to per-token. The batched prefill branch is the
remaining work; `EschaProj::forward` already takes `slots` and is
gate-verified at slots=4.
Tests: qwen35 199, runtime 597+12, dispatch 212, all pass.
Escha arms in both LA and FA chunk functions, plus both FFN functions, so
a trellis layer bypasses every fused MQ matcher for the whole batch rather
than dropping to the per-token path. `is_batchable_la` admits
Escha2T16/3T16 — not because a batched GEMM can read a trellis code (none
can) but because the escha arms intercept before those matchers.
H128 launches for a 512-token prefill: 409,600 -> 800. That is the whole
signal — prefill was doing decode's work once per token.
Correctness holds: PPL 11.8377, byte-identical to the pre-routing decode
number, and prefill argmax matches the decode-path result (248046).
Tests: qwen35 199, runtime 597+12, dispatch 212.
Two things this exposed, both fixed here:
- `batched_gemm_single_weight` had no F16 arm, so the uncoded
`in_proj_a`/`in_proj_b` siblings failed the moment escha layers were
admitted. `plain_gemm_key_for` already sanctioned
`F16 => GemmF16WmmaMb8`; this match simply lacked it.
- The FA `o_proj` escha branch initially ran ALONGSIDE the fused
epilogue, which writes the residual itself — that double-counted the
projection. It now replaces the epilogue rather than supplementing it.
PREFILL IS CORRECT BUT SLOW: 14.0 tok/s against a decode of 11.0. The
cause is diagnosed, not mysterious — `escha_gemv_native_moe_k8_indexed_
batched` re-reads the weight ONCE PER SLOT. That is right for MoE, where
every slot is a different expert, and badly wrong for a dense linear where
512 tokens share one weight. The fix is the grouped-GEMM shape the 35B
already uses (`escha_gemm_grouped_wmma_k2/k3`), with all slots in a single
group; wiring it is the next piece.
`escha_gemv_native_moe_k8_indexed_batched` re-reads the weight ONCE PER
SLOT. That is right for MoE, where every slot is a different expert, and
512x the weight traffic for a dense linear where every token shares one
weight. Batched prefill now uses `escha_gemm_native_moe_grouped_wmma`
with all slots in a single group — one expert, `expert_offsets = [0, n]`,
identity permutation — so each weight is read once per (layer, batch).
Same fix, same reason, as the 35B expert path (4.525 -> 2.657 ms/token).
prefill 14.0 -> 71.8 tok/s (5.1x)
decode 10.7 tok/s unchanged; decode is slots=1 and keeps the
GEMV, where there is nothing to group
PPL 11.8377 identical to three digits before and after
`EschaProj::forward` takes the grouped tables as an Option and falls back
to the per-slot GEMV when they are absent or slots == 1, so decode and
prefill share one code path and cannot drift.
Offsets are built once per LAYER, not per projection — six projections
share the same `[0, n]`. The identity permutation is precomputed at load
(`iota_i32`) since it never varies.
The gate now exercises both shapes. Grouped is looser than the GEMV
(7.6e-3 vs 1.5e-3 worst_rel at slots=4) exactly as
`escha_moe_gemm_grouped_wmma`'s own docs say it should be — WMMA
accumulates over a different partition of the contraction — and well
inside tolerance. PPL unchanged confirms it end to end.
Tests: qwen35 199, runtime 597+12.
`bench_escha_grouped_gemm` runs one A3B prefill chunk (2048 slots, ic=2048, oc=1024) and at that shape the slot-parallel kernel already moves 2.147 GB at ~175 GB/s against this box's ~209-220 ceiling — it is 80%+ bandwidth-bound, so no instruction-level change can register there. Three kernel ablations read "null" on it for that reason alone. This bench is the opposite regime: one slot, real 27B projection shapes, weights streamed once. It reproduces the 259 us gate_proj figure and makes decode cost visible. It also shows K=2 and K=3 taking the same time despite K=3 reading 50% more code bytes, i.e. the kernel is pinned at ~343 G weights/s independent of bytes moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
`infer.rs` seeded its anti-repeat state with `prompt_tokens.clone()` and
passed that to `apply_ngram_block`, which hard-bans (-INF) whatever token
followed any repeated 3/4/5/6-gram. Quoting the prompt necessarily emits
prompt 3-grams, so the next token of the quote was banned every time:
verbatim quotation of the prompt was impossible by construction.
`run.rs` had the same defect, passing `&conversation_tokens` — which
includes the user's own messages.
`test_long_ctx.rs:317` already documents this hazard and slices both
corrections to the current turn. These two files did not.
It presented as a model bug. A long-context probe showed
`VIOLET-ANVIL-62` coming back as `VIOLETANVIL62`, one planted name
spelled three ways in a single output, `14 March 2019` degrading to
`4 March`, and then an unbounded "Wait, I'll copy exactly:" retry loop
that read as long-context degeneracy. Ablation at 1k, greedy:
default (pen 1.15 + ngram + prompt-history) VIOLETANVIL62 wrong
--repeat-penalty 1.0 VIOLETANVIL62 wrong
--no-ngram-block VIOLET-ANVIL-62 ok
prompt-free history, both corrections ON VIOLET-ANVIL-62 ok
The last row is the fix: keep both corrections, scope the history to
generated tokens. The machinery was fine; feeding it the prompt was not.
Verified greedy, before -> after: needle at 1k `VIOLETANVIL62` ->
`VIOLET-ANVIL-62`; needle at 8k likewise. A context sweep (needle pinned
~607 tokens in, distance to the question 524 -> 7487) is EXACT at 1k, 2k,
4k, 4.5k and 8k — there was never a retrieval problem. The 8.5k coherence
probe at temp 1.0 with no repetition penalty, which previously collapsed,
now returns 903 coherent words and reproduces a planted sentence word for
word.
Severity is lab-only: `apply_ngram_block` has no caller in
hipfire-engine.
Also adds the flags the diagnosis needed — `--kv-seq` (with a pre-flight
check; the hardwired 4096 turned an over-long prompt into
`hipMemcpy H2D: illegal memory access` from inside prefill),
`--repeat-penalty`, `--temp` (the 0.3 default is not greedy, so
single-sample A/Bs across it are noise), `--dn-state`, `--no-ngram-block`,
`HIPFIRE_LOGIT_PROBE` — plus `tokenizer_roundtrip`, which ruled the
tokenizer out early (14/14 strings exact).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
hw-gate sol prelimsummary: Preliminary review could not be performed because the supplied prompt failed UTF-8 decoding before PR metadata, diff, changed files, mandatory buckets, and runner fixtures were available. run_hardware: false routes:
unavailable_routes: (none) questions_for_author:
|
… tok/s
Three independent causes, all in the NATIVE escha path and all invisible
until now because the FOLD path never touches these kernels (it is plain
MQ6 at runtime) and `profile_prefill_qwen35` drops escha kernels from its
category map — its totals came out ~40x too fast to be believed.
Measured on the native escha 27B, 2k prompt, gfx1151:
as shipped 52 tok/s
+ h128 length check 74
+ gfx1151 chunk default ~96
+ NT 1 -> 8 108 (2.1x overall)
1. `escha_h128_batched` validated its INPUT with `!=`. Prefill scratch is
sized for the maximum chunk, so any shorter chunk was rejected for being
too BIG — a 2009-token prompt yields a 217-slot chunk against 256-slot
scratch and failed with "a has 1310720 elements, need 1111040". Batched
prefill therefore never ran for a dense escha model; it fell back to the
decode kernel at ~10 tok/s. The `out` check three lines below already
documented why equality is wrong there.
2. gfx1151 had no measured prefill chunk default and fell through to the
generic 256. Swept at 8k: 256 -> 52 tok/s, 512 -> 73, 1024 -> 69.
3. `ESCHA_WMMA_GEMM(..., 1)` put `escha_decode_tile_lds` inside the loop
over batch slots with NT=1, so one weight decode was amortised across
only 16 slots and a 512-slot chunk decoded the ENTIRE weight matrix 32
times — the exact work batching exists to avoid. A rocprof trace shows
these two kernels at 90.4% of prefill (k3 51.5%, k2 38.9%) running the
FFN's ~69 TFLOP at ~2.5 TFLOP/s, i.e. the WMMA units mostly idle.
NT: 1 -> 74 tok/s, 2 -> 96, 4 -> 100, 8 -> 108. 181/182 VGPR, 0 spills.
NT changes only how many activation tiles share one weight decode; each
`acc[t]` accumulates independently, so output must be bit-identical. Gated
on that: the KLD reference was built at NT=1, and both NT=4 and NT=8 score
mean KLD = 0.000000 / PPL 9.6486 against it — unchanged to the digit.
199 qwen35 lib tests pass; the arch-default assertion that listed gfx1151
among the arches which must stay at 256 was updated deliberately, not
silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
`infer.rs` ran the DECODE kernel once per prompt token, so it prefilled at decode speed — ~13 minutes for an 8k prompt on the dense 27B, against seconds through `forward_prefill_batch`. Nothing required the slow path for text; it was just what this example did, and it made every long-context experiment pay minutes of avoidable wall-clock. The per-token loop is retained for the two cases that need it: VL mode (interleaved image embeddings) and `HIPFIRE_LOGIT_PROBE`, which wants a next-token distribution at each position rather than one at the end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
Every escha kernel holds one output tile column `nt` fixed and walks `kt`.
The checkpoint stores the grid kt-major (`[ic/16][oc/16]`), so consecutive
`kt` are a full tile-row apart — 139 KB on the 27B's gate_proj — and each
step is a fresh 64-bit address against a cold line. Transposed to nt-major
they are adjacent. Paired ablation on the wide decode GEMV, n=4:
243.9 -> 186.3 us, **24%**. End to end on the 27B that is decode
10.8 -> 11.4 tok/s (+5.6%, three samples) — the GEMV is only part of a step.
The transpose happens at LOAD, in the dense arm of `load_weight_tensor_raw`,
so the `.escha` payload stays verbatim from upstream: no re-convert, no
re-upload, no format version. Whole tiles move and their contents are
untouched, so every decoded weight is identical.
MoE experts load through `escha::load_escha_moe_experts`, which this does
NOT touch, so they stay kt-major. The kernels therefore take an `nt_major`
flag rather than assuming a layout: both orders reduce to two
loop-invariant strides computed before the hot loop, so there is no branch
cost and no duplicated kernel. Dense call sites pass `true`; MoE and the lab
harnesses pass `false` and are bit-identical to before by construction.
An earlier attempt hardcoded nt-major in the kernels and broke the shipped
35B, which is what motivated the flag.
Two traps worth recording, both found by instrumenting rather than reading:
* `launch_maybe_blob` carries TWO independent kernarg descriptions — the
`params` vec and a `blob_builder` closure used while a graph is being
recorded. Updating only `params` sent the old 6-arg list on the capture
path and the kernel read `nt_major` as garbage. Isolated by hardcoding
the flag inside the kernel: the 27B came right, so the logic was fine and
the argument was not arriving.
* The first version of the load hook went into `decode_raw_codec` in
hipfire-runtime, reasoning from the `escha_code` -> `.weight` alias. That
path never fires for escha; a debug counter printed zero permutations.
Check the hook FIRES before debugging the arithmetic.
Gated on `mean KLD = 0.000000` / PPL 9.6486 against a reference built before
any of this. 199 qwen35 lib tests pass, workspace builds all targets, and
the 35B plus all three 27B SKUs generate correctly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
The escha builds shipped with suffixes naming the DENSE tensor format
(`-q8`/`-mq6`/`-mq4`) while the bulk of each model is 2-bit trellis. That
collided with the rest of the ladder, where the name states the bulk format:
`qwen3.8-27b.mq4` is a genuinely 4-bit 15.66 GB model, but
`qwen3.8-27b-escha-mq4.escha` was a 2-bit 10.45 GB one. Same token, not
comparable.
Now `-xt` / base / `-pro`, ordered by size like every other SKU, with **MQ6
dense as the default** — it costs +0.28% PPL over Q8_0 dense on the 27B
(KLD 0.000534) and +0.10% on the 35B, for +12% and +17% decode respectively.
Filenames also corrected to `<model>.<format>[-variant]`:
`qwen3.8-27b-escha.escha` said escha twice and buried the format in the stem.
docs/MODELS.md already documented the intended `qwen3.6-35b-a3b.escha`; the
shipped artifacts had drifted from it.
qwen3.6-35b-a3b.escha-xt 11.39 GB PPL 8.0643
qwen3.6-35b-a3b.escha 11.84 GB PPL 7.6940 default
qwen3.6-35b-a3b.escha-pro 12.34 GB PPL 7.6864
qwen3.8-27b.escha-xt 10.45 GB PPL 9.7242
qwen3.8-27b.escha 10.77 GB PPL 9.6753 default
qwen3.8-27b.escha-pro 11.16 GB PPL 9.6486
The 27B trio is new to the registry — it was converted, verified and
published but never registered. Its MODELS.md row also carried stale Phase-1
numbers (min VRAM 40 GB from the decode-at-load era, 40 tok/s decode); the
native path resident is far smaller and the numbers are remeasured.
Renames on HF were server-side copies, so no bytes moved and the payloads
stay verbatim from upstream. All 80 non-escha registry entries verified
byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
nwoolmer
changed the title
The three 27B entries went in with min_vram estimated from file size (12/13/13)
while the 35B's came from measured GTT deltas. Measured them the same way —
peak GTT minus an idle baseline, 8192-token q8 KV:
.escha-xt 10.45 GB file -> 11.89 GB resident
.escha 10.77 GB file -> 12.22 GB resident
.escha-pro 11.16 GB file -> 12.66 GB resident
So `-xt` alone needs 11.89 GB before headroom and the old 12 would have let a
12 GB card try and fail. Now 14/15/15, the same ~2 GB margin the 35B rows use.
Measuring these back-to-back does not work: GTT is not released between
processes, so the second and third builds each appear to need 0.5 GB. Wait for
it to fall back to idle first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
The 35B's min_vram came from a resident measurement whose KV configuration was
never recorded; the 27B's I had just set from a measurement at an 8192-token
q8 KV. Two different bases, indistinguishable in the file.
Re-measured the 35B the same way (peak GTT minus idle, 8192 q8 KV):
.escha-xt 11.39 GB file -> 12.78 GB resident (was published as 12.04)
.escha 11.84 GB file -> 13.19 GB resident (12.45)
.escha-pro 12.34 GB file -> 13.70 GB resident (12.94)
The ~0.7 GB gap is the larger KV, not a regression. min_vram 15/15/16, keeping
the same margin the 27B rows now use. `min_vram_gb` is advisory — validated for
finiteness and shown in the TUI, not enforced — so erring high costs a reader
nothing while erring low sends them into an OOM.
Both model cards now state the KV the resident figures were taken with, so the
numbers are reproducible and comparable across the two models.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM
The escha converter passes upstream's MTP head through as ordinary
`mtp.*` tensors inside the trunk container. Neither existing resolver
looks there — the bundled path wants an HFBNDMTP trailer, the sidecar
path wants a sibling `.mtp` — so the 849 MB head in qwen3.8-27b.escha
(7.9% of the file) was shipped, downloaded, and never used.
Adds a third resolver after those two:
* `trunk_namer` maps the head's internal tensor names to their HF
spellings. All 15 `mtp.*` tensors in the shipped 27B resolve; the
MoE entries are unused there but keep the 35B shape loadable.
* `Qwen35MtpHeadConfig::from_trunk_text_config` derives the head
config from the trunk's own `text_config`. Verified against the
27B: hidden_size/heads/kv-heads/head_dim/intermediate_size match
the tensor shapes exactly, and partial_rotary_factor (0.25) and
rope_theta (1e7) resolve rather than falling back to defaults.
* the arch_id==21 assertion is scoped to the sidecar case, since the
trunk legitimately reports its own arch.
Measured on the 27B: accept rate 0.568 (151/266 drafts), 2.685 tokens
committed per verify window.
The 35B declines cleanly instead of panicking — its trunk ships a
router and a shared expert but no routed experts, so the MoE FFN
cannot be built and speculative decode stays off.
Also adds HIPFIRE_MTP_ACCEPT_STATS=1, which prints a cumulative accept
rate from `lower_mtp_window`. Speed alone cannot distinguish "drafts
rejected" from "drafts accepted but verify is expensive", and reasoning
from wall-clock to acceptance gets it backwards.
Note: enabling MTP changes greedy output. That is pre-existing and not
specific to this path — the ornith1.5 sidecar diverges the same way.
mtp_spec.rs only reconciles the batched-WMMA/decode-GEMV ULP mismatch
for advance==1, and at ~2.7 tokens/window advance>=2 dominates.
Splits the AR loop's wall time into forward / sample / detokenise and prints ms-per-token every 40 tokens. Added while chasing a reported "the daemon decodes 4.4x slower than examples/infer" defect. It does not: this profiler shows the loop at 11.86-11.96 tok/s on qwen3.8-27b.escha, matching infer's 11.9 and the model card's 12.1. The apparent gap was two measurement artifacts — a stdout consumer throttling the token stream, and the 30s CLIENT_TERMINAL_COMMIT_TIMEOUT charged to every request whose client never acks `commit_ready` (which also shows as finish_reason "aborted"). Acking correctly took a 160-token run from 49.3s to 19.4s. The profiler stays because wall-clock around the daemon includes load, prefill and that handshake, so it cannot answer "is decode slow?" — this can. It also prices something that looks like a bug and is not: the loop re-decodes the entire streamed sequence at each of 2-3 sites per token (14,520 tokens re-decoded over a 160-token run). Measured cost is 0.0 ms/token, so it is left alone deliberately.
6 tasks
Collaborator
|
This one I'm holding as Also: please re-run G1–G6 on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports Escha-W2 — EschaLabs' 2-bit trellis quantization — into hipfire, and
ships six artifacts across two models.
Published: qwen3.6-35b-a3b-escha · qwen3.8-27b-escha
What it is
A quantization format, not a model family. 16×16 tiles, K bits per weight,
packed
int16[in/16, out/16, 16K]. The codebook is a 3-op hash rather than atable:
r = ((s*0xCBAC1FED) & 0x8FFF8FFF) ^ 0x3B603B60, value= f16_lo(r) + f16_hi(r). Rotation is an unnormalised H128 (Sylvester) on bothsides. Consecutive weights are overlapping 16-bit windows sliding by K bits, so
each weight is a full 16-bit state with a 65536-value alphabet.
Codes are stored verbatim and decoded inside the GEMV — no decode-at-load,
no re-quantisation. Lossless repack into MQ2 is impossible rather than merely
lossy: one expert projection uses 10,746 distinct fp16 values against MQ2's 4
per group. New quant types
ESCHA2T16 = 42/ESCHA3T16 = 43, newRotationPlan::EschaH128.Qwen3.6-35B-A3B (MoE, arch 6)
-proqwen3.6-35b-a3b.escha-xtqwen3.6-35b-a3b.escha(default)qwen3.6-35b-a3b.escha-proQwen3.8-27B (dense, arch 5)
-proqwen3.8-27b.escha-xtqwen3.8-27b.escha(default)qwen3.8-27b.escha-proAgainst the non-escha ladder, same slice and harness:
qwen3.8-27b.mq6qwen3.8-27b.eschaqwen3.8-27b.mq3Every escha build beats
mq3on quality while being smaller, and beatsmq6on speed and size. Against
mq3it trades tokens/sec for quality-per-byte.Why decode trails the plain MQ quants.
mq3/mq6reach ~198–207 GB/s, thememory ceiling, so for them smaller means faster. Escha reaches ~127 GB/s
because it also decodes the trellis. Ablated on the shipped kernel: removing
the decode arithmetic is 1.83×, which would put it at 203 GB/s (~18 tok/s).
That is the codec's 7 ops per weight, and it buys the 2-bit residency.
Naming
<model>.<format>[-variant], matchingqwen3.8-27b.mq4-xt.-xt/base/-proorder by size; the suffix describes the dense tensors only — coded weights
are byte-identical across all three builds of a model. MQ6 dense is the default
(+0.28% PPL on the 27B, +0.10% on the 35B, for +12%/+17% decode).
-prois abit-exact repack of Escha's per-row int8 into per-32-block
Q8_0.Performance work
escha_h128_batchedrejected any chunk smaller than its scratch (so batched prefill never ran
and everything fell back to the decode kernel); gfx1151 had no measured chunk
default and fell to the generic 256 where 512 is the peak; and the grouped
WMMA GEMM decoded the whole weight matrix 32× per 512-slot chunk with
NT=1.rocprof puts those two kernels at 90.4% of prefill.
ktwere a fulltile-row apart (139 KB on gate_proj); transposed at load so the payload stays
verbatim. MoE stays kt-major, so kernels take an
nt_majorflag — both ordersreduce to two loop-invariant strides. Gated on
mean KLD = 0.000000.own control: two per-token kernels launched on the null stream inside a
captured region.
Bugs fixed along the way
reachable at N<96.
in_proj_a/in_proj_bare F16 and fellthrough passthrough while their siblings were quantised, rotating the shared
normed-x out from under them. Presented as a GDN "quality cliff" (PPL 2.4M).
read_dirmissed the
mtp/subdirectory.apply_ngram_blockover prompt-seeded history made quoting the promptimpossible by construction, which read as long-context degeneration.
Lab-only; no caller in
hipfire-engine.Status
hw-gatecannot go green here — it needs a gfx1201 runner, so the reviewerseats are skipped and the gate fails on a missing decision artifact regardless
of the code. Registry tags stay inert until this merges and the daily workflow
republishes
v1.json.🤖 Generated with Claude Code
https://claude.ai/code/session_01FfwZUVGT7QpjMdHikoERfM