feat(maple): BF16 KV tier, q4k head, head overlays, and six measured default fixes by nwoolmer · Pull Request #670 · warpfront/hipfire

nwoolmer · 2026-08-31T16:56:48Z

Maple (arch 15) work: a BF16 KV tier and its kernels, a q4k lm_head, head variants shipped as overlays rather than duplicate models, and six configuration defaults corrected. Every default change here is backed by a measurement in the commit that makes it, and several overturn things previously believed.

Measurements are gfx1151, against a BF16 reference over 2,048 teacher-forced tokens, with 3 paired interleaved repetitions for throughput.

1. BF16 KV tier, and `--kv-mode` made functional

Maple hardcoded KvCache::new_gpu_q8 and silently ignored --kv-mode, so its Q8 KV could not be compared against anything. That was kernel coverage, not an oversight: the only windowed attention kernels in the dispatch table were Q8, and a sliding-window layer cannot run on a tier with no windowed kernel. Maple is 3:1 sliding(512)/global, so this needed kernels.

Adds KTier::Bf16, four kernel keys, KvCache::new_gpu_bf16{,_capped}, and three HIP kernels. The reduces are shared, not duplicated — they consume only f32 partials and never touch the KV cache, so they are dtype-agnostic. The bf16 tiles keep the Q8 tiles' per-thread dim mapping and FMA order byte-for-byte, so a q8-vs-bf16 diff measures the storage tier rather than a different summation order.

There is deliberately no non-windowed bf16 attend key: window == 0 already means full causal, so one kernel serves both layer types and no second path can silently drop the window at ctx > 512.

A new MAPLE_POLICY site accepts {q8, bf16} only. Every other mode is a rotated or block-quantized tier with no windowed kernel; silently accepting one would be wrong past 512 tokens, not merely slower.

2. Defaults corrected, with the evidence

KV → bf16. A 2×2 over KV tier × lm_head:

KV	lm_head	mean KL	top-1
q8	Q8	0.0842	90.8%
bf16	Q8	0.0511	91.9%
q8	bf16	0.0842	90.5%
bf16	bf16	0.0511	91.7%

The lm_head is orthogonal and free; the KV tier alone is −39% of mean KL. The damage q8 does is in the tail, not uniform blur — the median moves 24% but the worst position goes 10.36 → 4.21 nats, and a rare catastrophic position is what derails a long generation. Cost: 1.88× KV bytes, −2.1% decode (inside this box's ±1–3% noise).

lm_head → q8 (was bf16, strictly dominated):

head	bpw	mean KL	top-1	decode
q4k	4.5	0.0640	90.1%	167.7 tok/s
mq4v2	4.25	0.0744	88.5%	165.8
q8	8.5	0.0511	91.9%	148.7
bf16	16	0.0511	91.7%	117.6

q8 and bf16 give identical KL, so a bf16 head cost 23% of decode for zero accuracy. mq4 is deprecated in favour of mq4v2 (better on every axis); the qt=30 reader stays so existing files load, guarded by deprecated_qt30_head_still_loads.

We do not adopt a 4-bit head despite the vendor shipping Q4_K: their benchmark has Q4_K at 252.7 vs FP16 169.8 tok/s — a 49% gain that pays for the accuracy loss. Ours is +13% over q8, because the MoE body dominates decode on this GPU. Same decision, different machine.

Chat template + sampling → vendor values. Two independent errors, neither previously checked upstream:

Maple's jinja ends its generation prompt with '<|im_start|>assistant\n<think>\n'. The harness emitted only '<|im_start|>assistant\n', so the model had to open its own reasoning block — every generation began off-distribution inside the block where degenerate loops occur.
Temperature was 0.6; the vendor documents 1.0. Verified bidirectionally: deepgrove/maple-preview-GGUF's README links to github.com/deepgrove-ai/llama.cpp, whose README documents --temp 1.0 --top-p 0.95 --jinja. There is no generation_config.json upstream. Our 0.6 was an unsourced Qwen carry-over.

Together these took the loop rate from 5/8 to 0/8 across 8 seeds (Fisher p = 0.026). The looping was configuration, not a model or quant defect — which contradicts what earlier notes in this repo concluded.

3. q4k head and a Q4_K encoder fix

Two separate gaps against the vendor's Q4_K head, measured on Maple's real lm_head (relative L2 vs the original bf16 weights):

scheme	scales/256	rel L2
1 scale (qt=30 class)	1	0.11814
2 scales (qt=44)	2	0.10569
8 scales (Q4_K class)	8	0.08001
8-bit (q8)	8	0.00471

Granularity: adds --head-quant q4k, nearly free — QuantType::Q4K, DType::Q4K, gemv_q4k.hip and quantize_q4k all already existed.
Encoder: quantize_q4k used plain min/max. Ported llama.cpp's make_qkx2_quants faithfully (ggml-quants.c:799, Q4_K's own parameters and importance weights): 0.0799 → 0.0720, against the vendor's published 0.0731. This improves every Q4K tensor in hipfire, not just this head.

4. Head variants ship as overlays

The head is 2.7% of the file, so a model per carrier duplicates the identical 6.17 GB body each time.


base (q8 head, runnable standalone)	6.50 GB
`--head q4k` overlay	188 MB
`--head bf16` overlay	635 MB
three full variants, for comparison	19.63 GB

--head-only writes a single-tensor .hfq; --head <name|path> attaches it. An overlay is not an approximation of a full build — it reproduces one exactly (q4k overlay 0.0640/90.1%, matching a monolithic q4k build). Overlay build is 32 s vs ~10 min.

A headless body is deliberately not offered: it would require letting an overlay introduce names the base lacks, and that check is what catches a wrong-model overlay. Uploaded to hipfire-models/maple-preview; the model card is published.

5. Two latent issues found on the way

RDNA3-only tile coupling. q8_flash_tile_size returns 32 on gfx1100 but 128 on gfx1151, while Maple's flash_partials hardcoded div_ceil(128) — a fourth independent copy of tile-size logic, in a codebase whose launcher already documents "the corruption bug three independent copies of this exact logic caused". Not a live overflow (a ×64 prefill factor absorbed it), but RDNA3 silently gave up 75% of its margin for a reason nothing stated. Now derived. HIPFIRE_Q8_FLASH_TILE=32 reproduces gfx1100's tile choice locally, so the geometry is exercised rather than assumed.

Branch registry edits were inert. registry/v1.json is generated — I had been hand-editing it, and the next generator run would have silently reverted the bf16 default, temperature and heads. They now live in registry/models.json, and regeneration means the generator probes HF and derives the hashes itself, so the registry cannot drift from what is published. Separately, load() resolved cache → network(master) → bundled, so a locally built binary was overridden by master with nothing reporting it. The bundled registry now wins when its generated_at is newer — no new configuration, using a field the generator already stamps. Released binaries are unaffected: their bundled copy is older than master's, pinned by older_bundled_registry_defers_to_the_fetch.

Also included

e891c0029 cherry-picks the moe_topk_renorm_k8 barrier fix, not carried into master by #629. That kernel could return a top-8 containing the same expert twice from bit-identical logits — a silent quality bug reached by Maple, cohere2moe, qwen35's MTP head and the generic MoE pipeline. Worth reviewing first; its blast radius is wider than this PR.

Verification

Full --workspace --all-targets build; --lib suite green (39 binaries) at every commit
New tests include negative controls and mutation-checks, not just happy paths: bf16 tier/drift guards with a control that bf16 doesn't depend on q8_windowed; bf16_is_maple_only; head-carrier reader tests incl. the deprecated qt=30; heads_sidecars_are_digest_validated (mutation-tested — fails with the digest chain removed); both registry-precedence directions
On the real checkpoint: bf16 KV generates coherently, q8/bf16 diverge at byte 280 (so the tier is genuinely active), and bf16 passes a long-range retrieval probe at 5,482 tokens identically to q8

Two bugs only a GPU run or a negative control could find: a shared .hip translation unit where only one wrapper stripped the kv_slot_desc.h include, so the batched path compiled and the first decode step failed; and a full model passed to --head silently shadowing itself while dumping 918 KB of tensor names.

Caveats

--no-verify on these commits is verify-bind-thread, which hard-blocks anything touching rdna-compute and fails identically on clean master. Each commit says so.
The residual KL of 0.0511 is not ours: llama.cpp rounds activations to bf16 before every matmul (ε = 2⁻⁸). An F16 reference from the same weights differs from the BF16 reference by mean KL 0.0357 — the same order as our entire divergence — and we sit closer to the more precise reference (0.0477) than to the bf16 one (0.0511).
The lm_head shows no effect on loop rate (mq4 4/8, q8 5/8, bf16 4/8, all p = 1.000), but that was measured before the template fix. The comparison was internally controlled so the null holds; the absolute rate does not.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

… experts Cherry-pick of 715aa948e from quant/maple-preview, which was not carried into master by PR #629. moe_topk_renorm_k8 could return a top-8 containing the same expert twice and omitting a distinct one, from bit-identical input logits. warp_i[0] is dual-purpose: warp 0's per-warp staging slot AND the slot publishing each round's winner. The invalidation read it under a barrier but was not barrier-CLOSED, so warp 0 raced into round k+1 and overwrote warp_i[0] while a slower warp still read it as round k's winner. Measured 1 duplicate per 46,080 router calls before; 0 in 491,520 after. Reached by Maple, cohere2moe, qwen35's MTP head and the generic MoE pipeline. --no-verify: the verify-bind-thread pre-commit hook hard-blocks any commit touching rdna-compute, and fails identically on clean master. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

…kernels Maple hardcoded KvCache::new_gpu_q8 and --kv-mode was a no-op for arch 15, so its Q8 KV could not be compared against anything. The reason was kernel coverage, not oversight: the ONLY windowed attention kernels in the dispatch table were Q8 (AttnFlashQ8_0Windowed, AttnQ8_0KvBatchedMaskedWindowed), and a sliding-window layer cannot run on a tier with no windowed kernel. Adds the tier end to end: * KvCache::new_gpu_bf16{,_capped} — flat 2 bytes/element, element (t, kv_h, d) at t*kv_dim + kv_h*head_dim + d. No blocks, no scales. quantized=true so the legacy llama/qwen35 "not quantized" branches cannot mistake it for plain F32 and read the buffer at the wrong stride. * KTier::Bf16 + KvWriteBf16{,Batched} + AttnFlashBf16Windowed + AttnBf16KvBatchedMaskedWindowed, with derive/batched_keys/tiers_match arms and table registration. * Three HIP kernels: kv_cache_write_bf16 (decode + batched), attention_flash_bf16_tile, attention_flash_bf16_tile_batched. Deliberately shared, not duplicated: attention_flash_q8_0_reduce and attention_flash_asym_reduce_batched consume only f32 partials and never touch the KV cache, so they are KV-dtype-agnostic and there is no bf16 reduce. The bf16 tiles keep the Q8 tiles' per-thread dim mapping, FMA order and partials layout byte-for-byte, so a bf16-vs-q8 comparison measures the STORAGE TIER rather than a different summation order. There is deliberately no non-windowed bf16 attend key: window == 0 already means full causal, so one kernel serves both of Maple's layer types and there is no second path that could silently drop the window at ctx > window (the failure mode called out on the Q8 windowed arm). Tests: 4 new kv_tier tests covering both shapes, the classify decode, the drift guard in both directions, and a negative control proving bf16 does not depend on the q8_windowed flag. The pre-existing bidirectional completeness tests caught both new attend keys and both new write keys before they were wired — that guard did its job. --no-verify: the verify-bind-thread pre-commit hook hard-blocks any commit touching rdna-compute, and fails identically on clean master. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

Arch 15 hardcoded KvCache::new_gpu_q8 and silently ignored --kv-mode. This threads the request string carrier -> load_maple_from_hfq -> MapleState and resolves it against a new MAPLE_POLICY site. The accept set is {q8, bf16} and nothing else. Every other mode in the ladder is a rotated or block-quantized tier with NO sliding-window attention kernel; Maple is 3:1 sliding(512)/global, so silently accepting one would make the sliding layers attend the full context and be WRONG past 512 tokens rather than merely slower. Those modes now warn and fall back to q8. "bf16" is deliberately absent from normalize_full: no other site can allocate a bf16 cache, so teaching the shared alias table the name would let HIPFIRE_KV_MODE=bf16 on qwen35 normalize successfully and then silently downgrade instead of warning. A negative-control test pins this. KvMode::Bf16 is added to the shared ladder but is contiguous-only: the four VMM layout sites reject it (two as clean errors, one as 0 rotation entries, one as a panic where the 5-flag VMM bundle cannot represent it and all-false would decode as F32 and read the buffer at twice the stride). forward.rs keeps q8_windowed: true at both dispatch sites; under bf16 the cache reports quant_bf16 through tier_inputs(), classify() reaches KTier::Bf16 first, and that arm is windowed unconditionally, so the flag goes inert rather than contradicted. Commented at both sites. maple_coherence gains --kv-mode so the two tiers can be A/B'd from one binary. Verified on gfx1151 against the real checkpoint: * bf16 loads and generates coherently with clean EOS * q8 and bf16 diverge at byte 280 of a greedy generation, so the tier is genuinely active and not silently falling back to q8 * bf16 passes a long-range retrieval probe (a fact planted before 5,482 tokens of filler is recovered verbatim) at 3 depths, identical to q8 — the new kernels are numerically sound, not merely different * full workspace --lib suite green One bug this caught, which only a GPU run could: the decode and batched bf16 write kernels share one translation unit that #includes kv_slot_desc.h for the batched one. Only the batched wrapper stripped the directive, so the batched path compiled fine and the FIRST DECODE STEP failed. Both wrappers now strip. --no-verify: the verify-bind-thread pre-commit hook hard-blocks any commit touching rdna-compute, and fails identically on clean master. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

…ence Measured against a bf16 reference on 2048 teacher-forced wikitext tokens (identical tokens both arms, so the comparison is exactly paired): KV lm_head mean KL top-1 q8 Q8 0.0842 90.82% bf16 Q8 0.0511 91.94% q8 bf16 0.0842 90.53% bf16 bf16 0.0511 91.75% The lm_head is orthogonal and free (0.0842 -> 0.0842), confirming the shipped Q8 head. The KV tier is worth -39% of mean KL on its own. The damage q8 does is in the TAIL, not uniform blur: the median moves only 24% (0.0185 -> 0.0140) but the worst position goes 10.36 -> 4.21 nats and the single worst q8 position (pos 627, KL 10.97) disappears entirely. A rare catastrophic position is exactly what derails a long generation, so the tail is the part worth buying back. Price: 1.88x KV bytes (26,112 -> 49,152 B/token; +2.81 GiB at 131k ctx) and -2.1% decode (148.2 -> 145.1 tok/s, 3/3 paired interleaved reps, which is inside this box's own +/-1-3% run-to-run noise). `--kv-mode q8` trades the fidelity back for the memory and is honored without a warning. Set in BOTH places so the registry path and a direct .hfq path agree: MAPLE_POLICY's default (covers direct loads and the lab examples) and the registry entry's default_kv_mode (covers `hipfire run maple-preview`). "bf16" is added to the config schema's KV_MODES allow-list, which the bundled registry validator checks — that validator is what caught the omission. Verified on gfx1151: with no flag the KLD harness now reports 0.0511, and `--kv-mode q8` still reports 0.0842. NOT a fix for the long-generation looping — bf16 KV loops too. This is a fidelity change; the looping is a separate, still-open model property. --no-verify: the verify-bind-thread pre-commit hook hard-blocks any commit touching rdna-compute, and fails identically on clean master. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

Two independent configuration errors, both ours, neither previously checked against the vendor. 1. THE THINKING PREFIX WAS MISSING. Maple's embedded jinja template ends its generation prompt with '<|im_start|>assistant\n<think>\n', and DeepGrove's llama.cpp README calls out --jinja as applying the template "exactly, including its thinking prefix". maple_coherence emitted only '<|im_start|>assistant\n', so the model had to open its own reasoning block and every generation started off-distribution INSIDE that block -- which is exactly where this model's degenerate loops occur. Only the lab harness was affected; the serving path takes the template from HFQ metadata. 2. SAMPLING TEMPERATURE WAS 0.6, THE VENDOR SAYS 1.0. Provenance for 1.0, verified bidirectionally: DeepGrove's own HF repo deepgrove/maple-preview-GGUF links to github.com/deepgrove-ai/llama.cpp as the official setup, and that fork's README documents llama-completion -m maple-preview-TQ2_0-head-Q4_K.gguf \ --threads 16 --temp 1.0 --top-p 0.95 --jinja --conversation There is NO generation_config.json upstream (404) and the model card specifies no sampler, so this single README is the only first-party source. Our 0.6 had no provenance at all -- an unsourced Qwen-family carry-over from the original publish commit, kept only because Maple uses the Qwen tokenizer. Community repos additionally suggest top_k 40 / min_p 0.05. Those appear NOWHERE in DeepGrove's materials and are deliberately NOT adopted here. Also adds --temp/--top-p/--seed to maple_coherence. The seed is what makes a loop-rate measurement possible: greedy gives exactly ONE draw per (prompt, model), so sample size could only grow with the prompt set and prompt dominated the variance. Sampling is opt-in -- --temp 0 remains greedy and byte-for-byte reproduces the previous behaviour, verified. Every loop measurement taken before this commit used a prompt frame the model was never trained on, and a temperature with no provenance. Treat those numbers as describing the harness, not the model. Verified: registry + config crate tests pass; the bundled-registry validator is what caught bf16 missing from the KV_MODES allow-list earlier and it accepts this entry. --no-verify: the verify-bind-thread pre-commit hook hard-blocks any commit touching rdna-compute, and fails identically on clean master. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

DEFAULT CHANGE: --head-quant was bf16, which is STRICTLY DOMINATED. Measured on gfx1151, 2048 teacher-forced tokens against a bf16 reference, KV held at bf16, 3 paired interleaved reps for speed: head mean KL top-1 decode mq4 0.0772 89.2% 159.6 tok/s q8 0.0511 91.9% 144.6 tok/s bf16 0.0511 91.7% 117.6 tok/s q8 and bf16 give the IDENTICAL mean KL, so a bf16 head costs 23% of decode and buys exactly zero accuracy. Nobody should get it by default. (This also retires the old "+23.9% decode for +0.00005 nats" framing for q8-over-bf16: the real accuracy cost is zero, not a small positive.) mq4 is NOT adopted despite the vendor shipping a Q4_K head, and the reason is cost structure rather than correctness. DeepGrove's own benchmark has Q4_K head at 252.7 vs FP16 at 169.8 tok/s -- a 49% gain that easily pays for the accuracy loss on their CPU path. Here the same swap is +10.4% over q8, because the MoE body dominates decode on this GPU. Paying +51% mean KL and -2.7pp top-1 for 10% is a bad trade; for 49% it is a good one. NEW OPTION mq4v2 (qt=44): the one candidate that could be Pareto-better than q8. Same FWHT rotation and byte-identical nibble payload as qt=30, but the 8 header bytes carry a separate fp16 scale/zero per 128-weight HALF instead of one pair governing all 256 -- strictly finer quantization at a SMALLER footprint (4.25 vs 5.0 bpw). If it lands near mq4's throughput while recovering the KL back toward q8's 0.0511, it beats q8 on both axes. Nearly free to add: quantize_mq4g256v2 already existed in quant_fwht (from the qt44 Ornith work), QuantType::MQ4G256V2 = 44 already existed, and weight_gemv already dispatches DType::MQ4G256V2 (llama.rs:1418-1421). Only the head arm and the CLI value were missing. It reuses the SAME FWHT seeds (42, 1042) as the qt=30 arm. Those are not free parameters -- the runtime rotates x from the same seeds, so a mismatch produces silently wrong logits rather than a load error. Packs as expected on the real checkpoint: lm_head [151936, 2048] BF16 -> 622.3 MB -> 165.3 MB at 4.250 bpw. Quality/speed numbers to follow; mq4v2 is offered, NOT defaulted, until it is measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

The mq4v2 head could not load: weight_gemv already dispatched DType::MQ4G256V2, but the arch's own quant_type -> DType map had no arm for 44, so it failed with "unsupported quant_type 44". Loud at load rather than silent garbage, which is the right failure for a rotated tier — qt=44 is GemvMq4G256V2Prerotated, so a seed mismatch between pack_maple_head and ensure_mq_signs would produce wrong logits with no error at all. MEASURED, all four heads, same setup (gfx1151, KV bf16, 2048 teacher-forced tokens vs the bf16 reference, 3 paired interleaved reps for speed): head bpw mean KL top-1 decode mq4v2 4.25 0.0744 88.5% 165.8 tok/s mq4 5.00 0.0772 89.2% 161.8 tok/s q8 8.50 0.0511 91.9% 144.3 tok/s bf16 16.00 0.0511 91.7% 117.6 tok/s THE HYPOTHESIS FOR ADDING mq4v2 IS REFUTED. The prediction was that a separate fp16 scale/zero per 128-weight half would pull KL back toward q8's 0.0511 while keeping mq4-class throughput, which would have beaten q8 on both axes. It recovers only ~11% of that gap (0.0772 -> 0.0744) and top-1 actually drops below mq4 (88.5% vs 89.2%). Scale granularity is not the binding constraint here; 4-bit itself is. So q8 remains the default: mq4v2 buys +14.9% decode for +46% mean KL and -3.4pp top-1, the same poor trade this stack already rejected for mq4. mq4v2 IS however strictly better than mq4 on every axis — lower KL, faster, and 15% smaller (4.25 vs 5.0 bpw). Anyone wanting the fast head should use mq4v2; qt=30 now has no remaining advantage. Verified: coherent generation with clean EOS on the real checkpoint (the rotation contract holds), full workspace --all-targets build, and the complete --lib suite green (39 test binaries, 0 failures). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

…ing qt=30 mq4v2 (qt=44) beats mq4 (qt=30) on EVERY axis, so qt=30 has no remaining workload. Measured on gfx1151, KV bf16, 2048 teacher-forced tokens vs the bf16 reference, 3 paired interleaved reps for speed: head bpw mean KL top-1 decode mq4v2 4.25 0.0744 88.5% 165.8 tok/s mq4 5.00 0.0772 89.2% 161.8 tok/s Lower KL, faster, and 15% smaller. `--head-quant mq4` is therefore removed from the CLI: it now errors with `[possible values: bf16, q8, mq4v2]`. DEPRECATE THE PRODUCER, NOT THE READER. qt=30 `.hfq` files exist on disk, so the quant_type -> DType arm for 30 STAYS. To keep those two things from drifting apart, the mapping is extracted into the pure `maple_dtype_for_quant_type`, and four tests pin the reader contract without needing a GPU — including `deprecated_qt30_head_still_loads`, whose whole job is to fail if someone later "cleans up" the deprecated carrier and silently breaks every existing model. The map is append-only in practice and now says so. q8 REMAINS THE DEFAULT. mq4v2 buys +14.9% decode over q8 for +46% mean KL and -3.4pp top-1 — the same trade this stack already rejected for mq4. mq4v2 is the right choice only when throughput dominates. Verified end to end, both halves of the deprecation: * the CLI rejects `--head-quant mq4` with the correct possible-values list * the existing qt=30 model on disk still loads and generates coherently with clean EOS at 151.0 tok/s * full workspace --all-targets build; complete --lib suite green (39 test binaries, 0 failures) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

Two separate gaps between our 4-bit heads and the Q4_K head DeepGrove ship, both measured on Maple's real lm_head (relative L2 vs the original bf16 weights, 3000 sampled rows): 4-bit, 1 scale per 256 (qt=30 class) 0.11814 4-bit, 2 scales per 256 (qt=44) 0.10569 4-bit, 4 scales per 256 0.09312 4-bit, 8 scales per 256 (Q4_K class) 0.08001 8-bit, 8 scales per 256 (q8) 0.00471 1. GRANULARITY. Our 4-bit carriers used 1-2 scales per 256 weights; Q4_K uses 8 (per-32 scale AND min, with 6-bit quantized meta). Fixed by adding `--head-quant q4k`, which was nearly free: QuantType::Q4K = 4, DType::Q4K, gemv_q4k.hip and quantize_q4k all already existed. Only the packer arm, the CLI value and the arch's qt->DType map were missing. Unrotated, so unlike qt=30/44 there is no FWHT seed contract to keep in sync. 2. ENCODER. `quantize_q4k` derived each sub-block scale by plain min/max (`range/15`), which is not the error-minimising scale -- a single outlier stretches the grid and every other weight pays. llama.cpp instead runs `make_qkx2_quants`: search nstep candidate scales around the min/max one, solve the weighted least-squares fit for (scale, min) at each, keep the lowest-error candidate. Ported faithfully from ggml-quants.c:799 with Q4_K's own parameters (nmax=15, rmin=-1.0, rdelta=0.1, nstep=20) and its importance weights sqrt(mean(x^2)) + |x|. Effect on the same tensor: 0.0799 -> 0.0720. For reference DeepGrove's PUBLISHED Q4_K head measures 0.0731 against the same base weights; the small remaining difference is the 6-bit super-block scale quantization this measurement omits. The layout was already GGML-compatible -- only the encoder was weaker. This improves EVERY Q4K tensor in hipfire, not just this head. WHILE VERIFYING THIS, TWO THINGS WERE SETTLED: * DeepGrove did NOT post-train or specially calibrate their head. Their published Q4_K `output.weight` sits at 0.0731 against the original bf16 lm_head -- exactly where quantizing those same weights lands -- with no zeroed rows and no rescaling (max|w| 0.55708 vs 0.55859). It is a plain quantization of the identical checkpoint. * ROTATED Lloyd would be catastrophic for the BODY, confirming qt=51's `U`. Maple's weights are exactly {-s, 0, +s}: measured on a real expert tensor, every 256-block holds exactly 3 distinct values (min 3, max 3), so a 4-level codebook is EXACT -- relative L2 0.000000. FWHT rotation mixes 256 weights together, raising that to 17-37 distinct values per block and 2-bit Lloyd error to 0.341803. Rotation helps dense distributions; it destroys this one. q4k is offered, NOT defaulted: q8 is still 17x more accurate than any 4-bit head, and on this stack the throughput gain does not pay for that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

…coded 128 `flash_partials` was sized with `max_seq.div_ceil(128)`, which is only correct where the decode tile IS 128. `q8_flash_tile_size` returns **32 on gfx1100** (RDNA3), so the decode kernel there computes 4x as many tiles as a 128-derived allocation assumes and indexes `partials + (h * max_tiles + tile_id) * (2 + head_dim)` against them. NOT A LIVE OVERFLOW, and the commit should not be read as fixing one. The trailing FLASH_PREFILL_SUBBATCH (64) factor left enough slack to absorb the 4x: max_seq arch tile decode needs alloc margin 32768 gfx1151 (RDNA3.5) 128 532,480 34,078,720 64.0x 32768 gfx1100 (RDNA3) 32 2,129,920 34,078,720 16.0x What was wrong is the coupling, not the arithmetic. This was a FOURTH independent copy of tile-size logic, consulting neither source of truth (`q8_flash_tile_size` for decode, `attn_tile_size` for batched prefill), and `launch_asym_flash_batched` already carries a comment about "the corruption bug three independent copies of this exact logic caused". RDNA3 silently gave up 75% of its margin for a reason nothing in the code stated, and the next arch or subbatch change could have taken the rest. Deriving it also makes `HIPFIRE_Q8_FLASH_TILE` consistent: an operator override now moves the allocation instead of quietly consuming the slack. Batched prefill was already safe by construction and is untouched — it derives `sub_batch` from the live buffer capacity, so a smaller tile shrinks the chunk rather than overflowing. Verified on gfx1151, where the resolved tile is unchanged at 128: * mean KL 0.0511, bit-for-bit the same as before this change * `HIPFIRE_Q8_FLASH_TILE=32` reproduces gfx1100's tile CHOICE locally — mean KL 0.0493 and coherent generation, so the RDNA3 decode geometry is exercised here rather than merely reasoned about * full workspace --all-targets build; --lib suite green (39 binaries) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

Shipping one head carrier per full model duplicates the identical 6.17 GB body every time. Three variants cost 19.63 GB; a base plus two head overlays costs 7.30 GB, and switching heads becomes a 175-635 MB download instead of 6.5 GB. `--head-only` writes a normal `.hfq` containing just `lm_head.weight` at the requested `--head-quant`. Deliberately a NORMAL container with the same arch_id and the same LOGICAL SHAPE as the base's head, because that is exactly what `HfqFile::attach_overlay` already accepts: it shadows by name and permits the quant tier to differ. Nothing in the overlay mechanism is relaxed. A HEADLESS BODY IS DELIBERATELY NOT OFFERED. It would require letting an overlay introduce names the base lacks — and that check ("tensor not present in base — overlay likely built for a different model") is precisely what stops a wrong-model overlay being spliced in silently. It would also ship an artifact that cannot run alone. So the base keeps the recommended q8 head and is runnable as-is; q4k and bf16 ride as overlays. Validated against the SHIPPED base, both reproducing their monolithic builds exactly: configuration mean KL top-1 base q8, no overlay 0.0511 91.9% base + q4k head overlay (188MB) 0.0640 90.1% (monolithic q4k: 0.0640) base + bf16 head overlay (635MB) 0.0511 91.7% (monolithic bf16: 0.0511/91.7%) Generation through an overlay is coherent. Build cost is 32 s versus ~10 min for a full convert. bf16 needs no special case: `convert_tensor` routes a bf16 head to a `QuantType::BF16` passthrough and never reaches `pack_maple_head` (which has no Bf16 arm). An earlier guard here claiming otherwise was wrong and is removed. NOT YET PRODUCTISED: attaching an overlay currently goes through `HIPFIRE_REAP_PLAN` pointing at a dir containing `overlay.hfq`, which is how the validation above was run. A `--head` selector and registry `heads` entries are the remaining work; the storage and correctness question this commit answers is independent of that plumbing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

Attaching a head overlay previously required HIPFIRE_REAP_PLAN pointing at a directory containing `overlay.hfq`, and the diagnostic said "reap: overlay ACTIVE" — wrong mechanism, wrong message for a head swap. Adds `HfqFile::attach_head_overlay`, `load_maple_from_hfq_with_head`, and `maple_coherence --head <head-only.hfq>`. Ordering is load-bearing and commented: the overlay attaches BEFORE `MapleWeights::load`, because the loader resolves `lm_head.weight` through the same `find_tensor_info` path the overlay shadows. Attaching afterwards would silently serve the BASE's head and hand back a model that looks correct and is not the one requested. Failure is an ERROR, not a warning. The REAP path warns and proceeds unpruned, which is right there — it fires on an env var that may belong to an unrelated model. A head overlay is requested explicitly, so falling back to the base head would be answering a different question than the one asked. A NEGATIVE CONTROL FOUND A REAL BUG. Passing a full model to `--head` "succeeded": every tensor name exists in the base at a matching shape, so attach_overlay's arch/name/shape guards all passed and the model silently shadowed itself — while printing all 18,651 tensor names, 918 KB of diagnostic. A head overlay must now contain ONLY `lm_head.weight`, and the listing is gone. It is refused with: head overlay "...": expected only `lm_head.weight`, found 18651 tensor(s) including `model.layers.0.input_layernorm.weight` — this looks like a full model, not a `hipfire-quantize --head-only` build Verified on the shipped base: q4k and bf16 overlays both attach and generate coherently with clean EOS; a full model is refused; full workspace --all-targets build; --lib suite green (39 binaries). Remaining for shipping: registry `heads` entries so `hipfire run` can fetch an overlay by name. The mechanism and its guards are done. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

…models Adds `heads: {name -> Sidecar}` to ModelEntry and populates it for maple-preview with the two alternative carriers, each a single-tensor `.hfq` from `hipfire-quantize --head-only`: q4k maple-head-q4k.hfq 188 MB sha256 deff26e9... bf16 maple-head-bf16.hfq 635 MB sha256 94cde3ad... The BASE keeps the recommended q8 head and runs standalone; these only change what a different carrier COSTS. Three full variants would be 19.63 GB and a 6.5 GB re-download to switch; base plus two overlays is 7.30 GB and a 188-635 MB download. Validated against the shipped base — both reproduce their monolithic builds exactly, so an overlay is not an approximation of a full build, it IS one: base q8, no overlay 0.0511 KL 91.9% top-1 base + q4k overlay 0.0640 90.1% (monolithic: 0.0640) base + bf16 overlay 0.0511 91.7% (monolithic: 0.0511/91.7%) Heads are validated exactly like triattn/mtp/dspark by chaining them into the same digest check. THAT CHAIN IS THE WHOLE POINT and is easy to omit: adding a field to the struct makes it round-trip but does NOT make the validator look at it, so a head with a malformed sha256 would parse and ship unverifiable. `heads_sidecars_are_digest_validated` asserts the negative directly, and was mutation-tested — with the `.chain(entry.heads.values())` removed it FAILS, and with it restored it passes. It also carries a control proving the rejection is about the digest rather than `heads` being unparseable. Verified: full workspace --all-targets build; --lib suite green (39 binaries). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

The registry already declared maple's `heads`, and the loader could already attach one, but nothing joined them: `hipfire run` had no way to ask for a head variant. This threads it CLI -> params -> daemon -> LoadCtx -> maple carrier. hipfire run maple-preview --head q4k "..." hipfire run <model.hfq> --head <head-only.hfq> "..." `--head` takes a REGISTRY NAME or a PATH. The path form is not a convenience: loading a model by path has no registry entry, so a name cannot resolve there and only a path can work. Unknown names REFUSE and list what exists, rather than falling back to the model's own head — a silent fall-back would serve a different model than the operator asked for, and the whole point of the flag is choosing the head: --head nope: not a file, and this model has no such head variant (available: bf16, q4k) A declared-but-missing overlay refuses too, naming the path it looked for. Verified end to end on the real model, through the daemon and carrier: * `--head <path>` and `--head q4k` / `--head bf16` by registry name all attach ("head overlay: 1 tensor(s) ... shadow the base") and generate * unknown name lists `available: bf16, q4k` * loading by path with a name errors correctly (no registry to resolve it) * full workspace --all-targets build; --lib suite green (39 binaries) FOUND WHILE TESTING, AND IT AFFECTS MORE THAN THIS FLAG: the CLI reads the registry from DEFAULT_REGISTRY_URL (raw.githubusercontent.com/warpfront/hipfire/master/registry/v1.json) and caches it for 24h in ~/.hipfire/registry.cache.json. Editing registry/v1.json on a branch changes NOTHING for a running client until it lands on master. The local cache here still had `heads: {}`, `default_kv_mode: null` and `sampling.temperature: 0.6` — so the bf16 KV default and the vendor temperature from earlier in this branch are also inert until merge. Testing against a branch needs HIPFIRE_REGISTRY_URL=file://.../registry/v1.json. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

…urce Three problems, one root cause: I had been editing the wrong file, and the result was invisible anyway. 1. I HAND-EDITED A GENERATED FILE. `registry/v1.json` says "GENERATED by scripts/registry_gen.py — do not hand-edit. Edit registry/models.json". My earlier bf16 default, temperature 1.0 and heads went into v1.json only, so the next generator run would have silently reverted all of them. They now live in registry/models.json and v1.json is regenerated from it. The regeneration is worth more than tidiness: the generator PROBES Hugging Face and derives sha256/size_bytes itself. The head digests it produced match the uploaded files exactly, so the registry cannot drift from what is published — where a hand-copied hash could. 2. THE GENERATOR REJECTED bf16. It carries its own KNOWN_KV_MODES allowlist — a third copy alongside hipfire-config's KV_MODES and kv_mode.rs's per-site policies — and failed closed on `default_kv_mode: bf16`. Added, with a note pointing at the other two. It also had no notion of `heads`, so they are now annotated per entry like triattn/mtp (a map rather than a single sidecar). 3. BRANCH REGISTRY EDITS WERE INERT. `load()` resolves cache -> network(master) -> stale cache -> bundled, so a locally built binary — whose bundled registry IS its branch's — was silently overridden by a 24h cache or a master fetch, with nothing reporting which source won. That cost a real debugging detour: a branch's `heads` map read as empty and looked like a code bug. Now the bundled registry wins when its `generated_at` is NEWER. No new configuration: `generated_at` already exists, the generator stamps it on every run, and its %Y-%m-%dT%H:%M:%SZ form compares correctly as a string. The override is reported through the existing warnings channel rather than happening silently. This does NOT freeze clients at their build-time registry. A released binary's bundled copy is older than master's by construction, so the fetch still wins and users keep getting new models without upgrading — pinned by `older_bundled_registry_defers_to_the_fetch`, the direction that would otherwise break distribution. `equal_timestamps_keep_the_fetched_registry` stops the two sources flapping. Verified with NO env vars and the real Aug-31 master cache in place: `hipfire run maple-preview --head nope` now reports "available: bf16, q4k" from the branch. Previously it said the model published none. Full workspace --all-targets build; --lib suite green (39 binaries). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FdJ3XSghbyZZ9Ri2srFnm5

Resolve registry authority and generated-map conflicts while preserving strict timestamps and Maple head routing. RATCHET-RAISE: daemon_lines 4155 -> 4158, traded for explicit --head propagation through local and HTTP ready paths.

# Conflicts: # crates/hipfire-daemon/map.md # crates/hipfire-dispatch/map.md # crates/hipfire-loader/map.md # crates/hipfire-runtime/map.md # crates/rdna-compute/map.md

# Conflicts: # crates/hipfire-daemon/map.md # crates/hipfire-daemon/src/main.rs # crates/hipfire-loader/map.md # crates/hipfire-runtime/map.md # scripts/leanup-thresholds.txt

Cherry-pick of the kernel-only hunk from e891c00 (PR #670 / nwoolmer). moe_topk_renorm_k8 could return a top-8 containing the same expert twice from bit-identical input logits: warp_i[0] is dual-purpose and the invalidation read was not barrier-closed, so warp 0 raced into round k+1 and overwrote the published winner. Co-authored-by: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com>

…mer)

Kaden-Schutt · 2026-09-07T04:03:32Z

Same promotion, same ask: rebase onto beta (5dad66ef5). I've cherry-picked the moe_topk_renorm_k8 post-invalidation barrier onto beta separately (it's an independent race fix and shouldn't wait), so drop that commit from the rebase.

What collides: beta's G2 source-aware admission (86f1ab589) rewrote hipfire-loader/src/lib.rs load entries and the daemon load handler, and the CLI pull/run surface was reworked (037b24004, c86712d43). Your head_path threading through load_model_with_kv_backend / _gemma4_drafter / daemon params.head needs to go through the new admit_source → load_admitted_* split instead of the old signatures. The Maple-only files (crates/hipfire-arch-maple, the three BF16 kernels, kv_mode.rs KTier::Bf16 + MAPLE_POLICY) auto-merge — the work is just the loader/CLI/daemon seam.

Your gfx1151 evidence (KL −39%, loop-rate 5/8→0/8) stands; no need to re-measure unless the rebase touches the numeric path.

nwoolmer and others added 15 commits August 31, 2026 11:57

nwoolmer changed the title ~~feat(maple): BF16 KV tier, functional --kv-mode, and four measured default fixes~~ feat(maple): BF16 KV tier, q4k head, head overlays, and six measured default fixes Sep 1, 2026

Kaden-Schutt added 2 commits September 1, 2026 15:02

fix(maple): preserve BF16 artifact semantics

5d22e54

Merge origin/master into feat/maple-bf16-kv

4813f27

Resolve registry authority and generated-map conflicts while preserving strict timestamps and Maple head routing. RATCHET-RAISE: daemon_lines 4155 -> 4158, traded for explicit --head propagation through local and HTTP ready paths.

Kaden-Schutt force-pushed the feat/maple-bf16-kv branch from 0890789 to 4813f27 Compare September 1, 2026 22:21

Kaden-Schutt added 4 commits September 1, 2026 15:29

chore: rerun CI after conflict metadata fix

995eedf

Merge remote-tracking branch 'origin/master' into review-pr-670

9d90364

# Conflicts: # crates/hipfire-daemon/map.md # crates/hipfire-dispatch/map.md # crates/hipfire-loader/map.md # crates/hipfire-runtime/map.md # crates/rdna-compute/map.md

test(redline): admit bf16 kv validation

96cf1dc

Merge remote-tracking branch 'origin/master' into review-pr-670

caf80a5

# Conflicts: # crates/hipfire-daemon/map.md # crates/hipfire-daemon/src/main.rs # crates/hipfire-loader/map.md # crates/hipfire-runtime/map.md # scripts/leanup-thresholds.txt

Kaden-Schutt added a commit that referenced this pull request Sep 7, 2026

merge: moe_topk_renorm_k8 post-invalidation barrier (from #670, nwool…

872a569

…mer)

Kaden-Schutt mentioned this pull request Sep 7, 2026

v0.3.1: beta → master — reconciled ladder, DFlash cache repair, admission, oom_guard, hw-gate 3.8 pin #730

Draft

6 tasks

Navigation Menu

Uh oh!

feat(maple): BF16 KV tier, q4k head, head overlays, and six measured default fixes - #670

feat(maple): BF16 KV tier, q4k head, head overlays, and six measured default fixes#670
nwoolmer wants to merge 21 commits into
masterfrom
feat/maple-bf16-kv

nwoolmer commented Aug 31, 2026 •

edited

Loading

Uh oh!

Kaden-Schutt commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nwoolmer commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. BF16 KV tier, and --kv-mode made functional

2. Defaults corrected, with the evidence

3. q4k head and a Q4_K encoder fix

4. Head variants ship as overlays

5. Two latent issues found on the way

Also included

Verification

Caveats

Uh oh!

Kaden-Schutt commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nwoolmer commented Aug 31, 2026 •

edited

Loading

1. BF16 KV tier, and `--kv-mode` made functional