fix(gemma4): enable lowered 26B-A4B generation with row-safe HFQ4-G128 and rollover-safe bounded Q8 KV

[HUSRCF]

Summary

This enables the lowered Gemma4-26B-A4B generation path with the mixed expert formats produced by the current quantizer, while fixing two correctness boundaries required by that model:

• pack HFQ4-G128 tensors row-by-row so K=704 expert down-projection rows do not share a partial quantization group;
• predicate HFQ4-G128 GEMM/GEMV tail lanes instead of reading beyond the logical row;
• route lowered Gemma4 MoE through the native AR/Jinja/sampling path and the hd512 asym3 attention kernels;
• allocate the lowered sliding/full KV caches by their actual layer counts;
• keep the 1024-token sliding Q8 cache bounded with modulo writes and compact-ring attention after rollover.

The rollover fix is correctness-critical. Before it, the lowered opcode path selected AttnFlashQ8_0 at pos=1927 for a cache with physical_cap=1024, so the ordinary Q8 writer used the absolute position and raised HIP illegal-memory-access. The fixed path remains AttnFlashQ8_0Windowed and completed a 29,788-token prompt.

Artifact provenance

The validation artifact was generated from google/gemma-4-26B-A4B-it, downloaded from ModelScope on DiamondHill, then quantized with Hipfire's current quantizer after the HFQ4-G128 row-tail correction. It is a locally produced validation artifact, not yet an official Hipfire model-repository artifact.

• local artifact: gemma-4-26b-a4b-it.hfq4g128-tailfix.hf4
• size: 15,632,064,572 bytes
• SHA-256: 45da530b43e0e8ea2cffd77fe17c69bfb8dce3ec4dde2a26a9b68ad428d01f1c

If this route is accepted, the corresponding converted artifact may need to be published in the Hipfire model repository before the model can be pulled through the normal registry flow.

Validation

Local GPU: W7900 gfx1100, GPU0, HIP 7.14. Tests used the artifact above.

• cargo test -p hipfire-quantize --bin hipfire-quantize hfq4g128_row_tests: 2 passed
• cargo test -p hipfire-dispatch kv_tier: 42 passed
• cargo build --release --workspace --features deltanet: passed
• target/release/examples/test_kernels: 16 passed, 0 failed, 0 skipped
• Gemma4 long-decode smoke: 1/1 valid, 1,536 generated tokens; prefill 18.00 tok/s, decode 17.77 tok/s
• LongBench rollover smoke: 29,788 prompt tokens, 1/1 valid, 0 execution errors; prefill 17.33 tok/s; prompt MD5 51156095f76ddd012929d547eb3ea912
• Qwen3.5-4B serve battery, thinking off: 5/5 non-empty, 0 attractors, 0 retrieval misses; two rows reached the deliberately small 180-token cap
• Redline capture route: passed
• change-gate unit routes: Gemma4, dispatch, quantize, and rdna-compute all passed

Binary hashes for the final validation build:

• daemon SHA-256: 97a5742796368c8a3b33a06adac42d24d8b7429ed46444b6a282d8f24b25bb2e
• hipfire SHA-256: deccac587ed4d3cbffc43cbd1ceace486f26a267498f68dabc56a8daae34cc70

Known gate noise / incomplete routes

scripts/no-gpu-ci.sh passed Rust and 836/841 Python tests. The five failures are all in the unchanged tests/test_mq4c_repack.py; current origin/master tests require HfqmError, main, and parse_hfqm_index, while the current upstream tool does not expose those APIs.

The selected speed gate compares against a stale gfx1100 decode floor: this branch measured 141.2 tok/s, while a clean origin/master@61633693 worktree measured 141.1 tok/s with the same model and GPU. This is no branch regression (+0.07% relative), although both are below the checked-in 178 tok/s floor.

The change gate's generated Qwen serve command was rejected in preflight because it combined thinking=med (2,048-token thinking cap) with max_tokens=180. The valid thinking-off battery described above was run separately.

The full 30-row LongBench and complete long-decode matrices will continue after PR creation; this initial PR evidence intentionally includes one long-decode case and one full-length rollover case first.

[HUSRCF]

We've noticed a serious issue introduced by b482d95, which lead to:

• When encountering regular body text, hold == 0 incorrectly triggers a break, causing the body text to continually accumulate.
• gemma_is_marker_prefix() uses s.starts_with(marker), which mistakenly misidentifies "marker + body text" as a control marker as a whole.
• During EOS, flush() unconditionally clears the AwaitingThought buffer.

A third commit would be added to solved this issue!

[HUSRCF]

The issue had been resolved

[Kaden-Schutt]

The source/host-side blockers are fixed and final-head CI is green at ff368ab63, but this cannot merge without final-head GPU proof. The review fixes changed the HFQ4-G128 row-tail producer, Gemma scratch sizing, and the hd512 slot-descriptor kernel ABI; the PR body's earlier run predates those changes.

Please rerun on gemma-4-26b-a4b-it.hfq4g128-tailfix.hf4 (SHA-256 45da... fixture from the PR):

1. scripts/redline_daemon_harness.py for the kernel/dispatch path, preserving the JSON report and HIP/PM4 parity.
2. scripts/serve_harness.py battery on the same artifact/settings, with decoded text inspected.
3. Include the full artifact digest, final commit, GPU/ROCm identity, and reports in the PR.

I checked the available hipx and hiptrx model stores; that fixture is not present, so I cannot honestly produce this proof from the maintainer hardware pool.

[HUSRCF]

Got it, PR would reopen once full GPU test done! : )

[HUSRCF]

Thank you for the precise validation checklist. I rebased the follow-up work onto current master@8cd15a62b while preserving the newer device-mesh and CK-sidecar changes, and refreshed this existing PR rather than opening a separate one. The validated final head is 17abd04df.

The additional fix is correctness-critical for retained replay. On gfx1201, the first PM4 divergence occurred at the reverse copy_f32_buffer transition; after repairing that boundary, the next divergence was the MoE gate/up producer feeding gelu_tanh_f32. The AQL path exposed the analogous rope_f32 to kv_cache_write_q8_0_ring transition. The retained barriers ordered dispatch execution but did not provide the cache ownership/acquire semantics required for these VMEM producer-consumer boundaries. The proposed delta preserves the conservative dependency acquire on gfx12 and gives the PM4-IB vendor AQL boundary system acquire/release scope. It does not remove dependency waits or broaden automatic Redline admission.

Validation fixture:

• Source model: google/gemma-4-26B-A4B-it, downloaded from ModelScope and converted locally with the corrected Hipfire HFQ4-G128 quantizer
• Artifact: gemma-4-26b-a4b-it.hfq4g128-tailfix.hf4
• Size: 15,632,064,572 bytes
• SHA-256: 45da530b43e0e8ea2cffd77fe17c69bfb8dce3ec4dde2a26a9b68ad428d01f1c
• KV mode: Q8
• Topology: TP=1, PP=1

Final-head build identity:

• HIP/ROCm: 7.14.60850
• daemon SHA-256: b32bc90ad1ad63c601af72349cd34b3342836638c8849e675f7f29b92a8419ff
• CLI SHA-256: 0c1f2e5d2354bfd7f111cec392ca7e14ae1ad2c9862db17657eaba641bc83252

Final-head no-GPU checks completed on the rebased tree:

• cargo check -p hipfire-generate: PASS
• cargo test -p redline-rocr pm4_ib_packet_matches_aqlprofile_vendor_layout -- --nocapture: PASS
• cargo test -p rdna-compute gfx12_rotated_vmem_writers_require_a_pre_dispatch_acquire -- --nocapture: PASS
• Gemma4/dispatch/generate/rdna-compute crate-map drift checks: PASS
• python3 -m py_compile scripts/eval_gemma4_eseries.py: PASS
• git diff --check: PASS
• cargo build --release: PASS

./scripts/no-gpu-ci.sh completed all Rust checks and reported 839 passed, 5 failed in Python. The five failures are the unchanged upstream tests/test_mq4c_repack.py expectations for HfqmError, main, and parse_hfqm_index, which are absent from the current tools/mq4c_repack.py; this series does not touch that tool or test.

Final-head retained-replay validation on the exact binary and artifact above:

• W7900 / gfx1100: scripts/redline_daemon_harness.py PASS; stable 1,263-dispatch / 24-kernel sequence (80b0adea50f18383); PM4-IB vs HIP bit exact across logits, sliding/full KV, recurrent state, full blob/scratch surfaces, and GDN frame.
• R9700 / gfx1201: PASS with the same 1,263-dispatch / 24-kernel shape (9e0848791cec3ee9); PM4-IB vs HIP bit exact on the same surfaces, including GDN frame.
• AQL was also bit exact during diagnosis on both architectures, but the manual shadow timings are diagnostic only and are not being promoted as a product performance claim.

Final-head user-facing validation used scripts/serve_harness.py battery with Q8 KV, speculation off, greedy sampling, native Gemma no-think framing, max_tokens=512, and max_seq=65536:

• W7900 / gfx1100: 5/5 turns, runaway=0, empty=0, attractor=0, retrieval_miss=0, and zero non-empty reasoning channels.
• R9700 / gfx1201: 5/5 turns with the same clean counters and zero non-empty reasoning channels.
• I inspected all ten decoded answers. Code, arithmetic reasoning, factual explanation, prose continuation, and instruction-following outputs were coherent on both GPUs; the expected small wording differences between architectures did not change meaning or expose hidden reasoning.

User-facing long-context coverage on the same artifact:

• W7900 / gfx1100 Gemma-native no-think LongBench-v2 hard30: 30/30 valid, 0 runtime errors, 0 non-empty reasoning events, 0 no-think contract violations, 29 scored, 10/29 correct (34.48%). Median prefill was 71.64 tok/s and median decode was 66.90 tok/s with max_seq=65536, Q8 KV, and greedy sampling.
• One sample (longbench-13) repeated in the visible answer channel until the 16,384-token cap and was left unscored. It did not re-enter the reasoning channel; this is reported rather than hidden or truncated.
• Thinking-enabled 16K-output stress runs completed 30/30 without runtime errors on both gfx1100 and gfx1201, although most rows exhausted the output budget inside reasoning and are not useful as an accuracy result.

The complete JSON and logs are preserved locally under tmp/pr667-final-reports/{gfx11,gfx12}/ in the validation checkout. They include the full per-dispatch AQL contracts, HIP/PM4 state hashes, request/prompt MD5s, and decoded response bodies. I can upload a compact archive if useful; the reports are deliberately not added to the repository because each Redline JSON contains tens of thousands of per-dispatch records.

This closes the requested final-head GPU rerun. The manual shadow timings remain labelled diagnostic evidence rather than a complete product timed-arm route-proof claim.

[HUSRCF]

Small final-head correction: CI's dispatch-bypass ratchet identified two newly introduced direct HFQ4/HFQ6 indexed MoE gate/up calls in the Gemma4 architecture crate. I moved those calls through the existing run_uniform_moe_gate_up dispatch route in 7b4e93f68 instead of raising the debt threshold.

The relevant gates now pass locally with bypass_regressions=0 and bypass_total=237; scripts/ci-rustfmt-changed.sh, scripts/leanup-ratchets.sh, scripts/ratchet-diff.sh origin/master, cargo check -p hipfire-arch-gemma4, and git diff --check all pass.

I rebuilt release artifacts at 7b4e93f68. They are byte-identical to the artifacts used for the gfx1100/gfx1201 Redline and serve-harness runs reported above:

• daemon SHA-256: b32bc90ad1ad63c601af72349cd34b3342836638c8849e675f7f29b92a8419ff
• CLI SHA-256: 0c1f2e5d2354bfd7f111cec392ca7e14ae1ad2c9862db17657eaba641bc83252

Thus the preserved two-GPU reports are final-binary evidence for the current PR head as well: the source-level routing cleanup did not change either executable.

[HUSRCF]

Final-head update (supersedes the binary-identity note immediately above): the release binaries embed commit identity, so the crate-map-only commit changed their digests. I therefore rebuilt and reran both requested GPU routes on the actual final executable rather than relying on the earlier reports.

Current PR head: e08991af0

Final artifacts:

• daemon SHA-256: a7a08547a1bc92a70461fb66d91e836a7850f51e2d78cf5e2f1a9cac7a64badb
• daemon MD5 (reported by serve harness): 6074e76500da8d37c3998d143f76aef2
• CLI SHA-256: d4d92de7799f5800a8f21f3a79de9025d8de382909cfa8fd0788938d0d84825a
• model SHA-256: 45da530b43e0e8ea2cffd77fe17c69bfb8dce3ec4dde2a26a9b68ad428d01f1c

The final small source correction teaches the shared uniform-MoE dispatcher that native HFQ4G256/HFQ6G256 tags use the same indexed launchers as their layout-equivalent MQ4G256/MQ6G256 forms. This was caught by the first final-head GPU rerun; the corrected binary then passed on both architectures.

Final-head results:

• W7900 / gfx1100 Redline: PASS; stable 1,263 launches / 24 kernels; sequence 80b0adea50f18383; 78.2 tok/s diagnostic median; PM4-IB vs HIP exact for logits, KV, recurrent state, blob/scratch, and GDN frame.
• R9700 / gfx1201 Redline: PASS; stable 1,263 launches / 24 kernels; sequence 9e0848791cec3ee9; 72.2 tok/s diagnostic median; the same PM4-IB vs HIP surfaces are exact, including GDN frame.
• W7900 serve battery: 5/5 coherent answers, runaway=0, empty=0, attractor=0, retrieval_miss=0, reasoning channel empty for all turns.
• R9700 serve battery: the same 5/5 clean result. I inspected all ten visible answers across code, arithmetic, factual, prose, and instruction prompts.

Structural checks at this head pass locally: full crate-map check (42/42), dispatch-bypass ratchet (regressions=0, total=237), ratchet diff, changed-file rustfmt, cargo check -p hipfire-dispatch -p hipfire-arch-gemma4, release build, and git diff --check.

Durable report filenames are redline-daemon-e08991af0.json and serve-battery-e08991af0.json under the corresponding local tmp/pr667-final-reports/gfx11/ and gfx12/ directories. The manual tok/s values above remain diagnostic harness observations, not product timed-arm performance claims.

[Kaden-Schutt]

Thanks for pushing this through. We're promoting beta → master as v0.3.1 and this is on the blocker list, so I'd like to get it landed — but it needs a rebase onto current beta (5dad66ef5) rather than master, because beta made two changes that this PR runs straight into:

1. 05908ea1f moved the lowered/MoE refusal to admission time (crates/hipfire-arch-gemma4/src/carrier.rs::gemma4_lowered_refusal, LOWERED_GENERATE_REFUSAL in ar.rs). Your PR enables exactly that path, so the reconciliation is: delete the refusal gate + constant (or narrow it to whatever combinations you still don't serve), keep your generate_gemma4_lowered dispatch in ar.rs, keep your scratch/max_seq + KV layer-count + HFQ4-G128/Q8-ring stack.
2. beta migrated std::env::var("HIPFIRE_*") reads to hipfire_config::developer_var(...) in lowered.rs / dense.rs / ar.rs. Please re-apply that in the rebased code (and drop the HIPFIRE_KV_SEQ read — beta removed it on purpose).

Crate maps (crates/*/map.md) will conflict; take beta's and regenerate with python3 scripts/check-crate-maps.py <crate>.

Once rebased, the GPU proof needs re-running on the rebased tip (your current evidence is against 8cd15a62b). We don't have a pinned Gemma4 lowered fixture on the gate hosts — if you can name the exact HF artifact + sha256 you validated with, I'll add it to scripts/hw-gate/fixtures.json so the gate can exercise it.

[HUSRCF]

Got it, all conflicts should be resolved in a day. And the Gemma Model information should also be added!

[HUSRCF]

@Kaden-Schutt, while completing the requested final-head dual-architecture E2E and retained-replay validation for this PR, we found a shared-kernel correctness prerequisite that might be important to land before or together with #667: #734.

The indexed HFQ4-G256 MoE gate/up kernel retained an A3B-specific tail = 0 assumption. That is valid for A3B K=2048 (8 groups), but Gemma4 uses K=2816 (11 groups), so groups 8..10—768 of 2816 input dimensions—were silently omitted from each affected routed gate/up projection. This explains the token-attractor/repetitive output seen during the final E2E pass; it is not a model-artifact or sampling issue.

A direct runtime-tail fix restored the math but introduced 16 bytes of compiler-generated private scratch on both gfx1100 and gfx1201, which retained PM4 correctly rejected. #734 instead adds uniquely named, compile-time K2816 variants, preserves the gfx1100 GLC/SLC/DLC policy selection, keeps the A3B K2048 route unchanged, and registers the new symbols with replay metadata.

The pinned Gemma4-26B-A4B artifact (sha256 45da530b43e0e8ea2cffd77fe17c69bfb8dce3ec4dde2a26a9b68ad428d01f1c) now passes on both W7900/gfx1100 and R9700/gfx1201: stable 1263-launch captures, HIP↔PM4 exact shadow parity, GDN-frame exactness, and coherent greedy E2E output with no attractor/runaway/empty response. The decoded assistant output is byte-identical across the two GPUs (md5 20cc5cc7be723d16b871d5813c8ea8b7).

Would you be open to merging #734 first, or cherry-picking it into whichever integration branch is used for #667? Since #667 enables the lowered Gemma4 path that reaches this shared kernel, merging only this PR without the tail fix might expose a severe correctness failure. No model artifact re-upload is needed.