fix(dflash): transactional draft ctors, emitter-error rollback, one ctx-cap predicate - #691
fix(dflash): transactional draft ctors, emitter-error rollback, one ctx-cap predicate#691Kaden-Schutt wants to merge 22 commits into
Conversation
The default ~24-token prompt reports prefill_tok_s ~= 363 tok/s while a 4.4k-token prompt on the same binary reports 886: the short-prompt number is launch overhead, not prefill, and the JSON gave no way to tell. - Add --prompt-file <PATH>: prompt read verbatim (raw bytes, no trim), mutually exclusive with positional PROMPT words. - Standard-bench JSON gains top-level prompt_tokens (u64, as the daemon reports it in done.prompt_tokens), prompt_md5 (hex md5 of the exact prompt bytes), prompt_chars, and warnings[]; warn when prompt_tokens < 256 that prefill_tok_s measures launch overhead. - Same three values printed on the stderr banner next to model:/arch:. - No existing field renamed, nulled, or removed; default prompt bytes unchanged (audit 2026-09-02 fix slice: bench-prompt-evidence).
The daemon's done event has no prompt_tokens key; the prompt is prefill_tokens (rows prefilled) plus cached_tokens (prompt-cache prefix). Measured on a 7900 XTX the JSON reported prompt_tokens: null and never warned on the 24-token default prompt.
…ropped hw-gate Fable seat on #689: the flag table lost its `--reasoning-on` row (base AGENTS.md:359) while the flag still exists in `hipfire bench --help`. Additive row only, as the PR body says.
… JSON, short-prompt prefill warning) to staging
llama::is_batchable_la admitted MQ4G256V2/MQ6/5/3/2G256V2 for WMMA
prefill only on gfx1200/gfx1201 while qwen35::is_batchable_la admitted
them on gfx11+gfx12 behind HIPFIRE_MQV2_GFX11_WMMA, so plain Llama/Qwen3
dense qt=44 models prefetched per-token on gfx1100/1151 while Qwen3.5/3.8
took WMMA — despite both doc-comments claiming an exact match (audit
2026-09-02 Broken 1).
Move the dtype set + arch set + kill-switch helper into
llama::mqv2_wmma_batchable / llama::mqv2_gfx11_wmma_enabled_from_env in
hipfire-runtime and delegate from both callers, so the lockstep is
structural. MQ4CG256 (qt=45) stays gfx12-only in both by intent.
Tests: rename the two gfx12-only llama admit tests to gfx11+gfx12
expectations, repoint qwen35 env-escape test at the shared helper, and
add mqv2_admit_llama_qwen35_lockstep asserting both gates agree over
the MQ-V2 dtypes x {gfx1100, gfx1151, gfx1201, gfx1030, gfx1010}.
…ections mq4v2_gemm_parity's Gaussian weights give both halves near-identical headers, so a wrong half-select hides in quantization noise despite the header comment promising a systematic blow-up (audit 2026-09-02 Missing 1). Add arm 2 using the disjoint-halves construction from mq4v2_residual_parity (half0 [-1,1], half1 [96,160]) over the same batch-size sweep: v2 output must match the f32 reference within 5% rel-RMS, and the swapped-headers negative control (as in mq4v2_moe_parity) must DISAGREE. Keep the Gaussian v1-vs-v2 arm and fix the header comment to state what each arm can and cannot detect. Add a host-side test proving the fixture discriminates with no GPU. Docs (audit Would-change 1-2): spec section 9 now records MoE as production-wired for qt=44, the XBATCH single-row path as ported, and the gfx11 kt+=2 / residual kt++ stepping; residual_mmq.hip loses its stale Experimental tag; the gfx12 QKV kernel loses its HYPOTHESIS/scaffold wording for the validated C-map statement.
DflashScratch::new_with_mq, new_windowed, DflashWeights::load, and build_generic_dflash_speculator could '?' out mid-construction, leaking earlier alloc_tensor results (GpuTensor/DeviceBuffer have no Drop). Record each allocation in a slot vec (gt!/wt!/at!) taken once into the final owner; the error arm frees completed layers plus staged slots. new_windowed frees the base scratch via alloc_or_free!; the generic builder frees weights (and scratch) on later failures. Success path is byte-identical. Mirrors load_dflash_state's or_free! (audit-Dflash Broken 3).
The make_spec_emitter Err exit ran after a successful spec.prefill without production_fail_closed_rollback_live, unlike every other post-prefill error exit. The target KV/DeltaNet/drafter hidden had advanced and host seq_pos/conversation_tokens were cleared, so the next turn could LCP against a dirty GPU. Route it through the same rollback + fail-closed error (audit-Dflash Broken 4).
…length generate_dflash fell back to AR only when prompt + max_tokens > ctx_capacity, but generate_spec hard-errors when prompt + max_tokens + block_size > ctx — requests in that band got gen_start followed by an error instead of the promised AR fallback. Both sites now share spec_ctx_request_fits (prompt + max + block <= cap). The mid-loop position + block_size >= ctx_capacity break now sets SpecRun::ctx_exhausted, which the qwen (v2 + legacy) and dense epilogues OR into the length decision: finish_reason=length with no cache store instead of a silent early stop (audit-Dflash Broken 5).
qwen_dflash_semantic_terminal_tests.rs carries historical rustfmt debt; CI enforces rustfmt on changed files, so adding one test there forces a 6k-line reformat. The new contract lives in qwen_dflash_ctx_exhausted_tests.rs and the debt file is restored to master byte-for-byte.
af6f141 to
3babc96
Compare
hw-gate sol prelimsummary: This stacked change makes DFlash weight and scratch construction release already-allocated GPU resources on later failures, rolls back live target/speculator state when emitter creation fails after prefill, unifies draft-context admission and exhaustion terminal handling, adds verbatim benchmark prompt files and prompt evidence, narrows plain-Llama MQ-V2 batched-prefill admission, and updates MQ4-V2 parity coverage and kernel documentation. Runtime filesystem access is limited to an explicitly supplied --prompt-file; the new md5 dependency only hashes prompt bytes. No unexplained unsafe code, credential access, process spawning, build-script changes, or runtime network access is introduced. run_hardware: true routes:
unavailable_routes: (none) claim_assessment: The author claims complete transactional cleanup for draft constructors, post-prefill rollback on emitter failure, and consistent context-cap fallback/length semantics. Normal fixture runs can establish successful load and coherent serve behavior, including MQ-V2 admission, but they cannot prove allocation-failure cleanup or emitter-construction rollback without fault injection. The diff supports cleanup for Result-returning allocation failures, while malformed-codebook panics and the target mutation before new_spec_scratch remain outside that guarantee. |
…efill_chunk has no V2 arms hw-gate Fable seat on #690 (run 33895641944), source trace verified: the shared MQ-V2 admit rule made llama::is_batchable_la admit plain Llama/Qwen3-dense qt44/47-50 artifacts to WMMA prefill on gfx11 and gfx12, but llama.rs::forward_prefill_chunk's per-layer matchers (qkv_is_mq ~:2570, wo_is_mq ~:3025, ffn_is_mq ~:3117, w_down_is_mq ~:3248) list only MQ4G256|MQ6G256|MQ3G256|MFP4G32 — an admitted V2 model skips the FWHT rotate and runs the V1 hfq4g256 launchers on V2 blobs: silently incoherent prefill. master's pre-existing mq4_v2_gfx12 arm had the same hole on gfx12; no gfx12 Llama-V2 artifact has ever tripped it. llama::is_batchable_la now refuses every *G256V2 dtype and MQ4CG256 on every arch, with the reason at the site. qwen35::is_batchable_la keeps the shared mqv2_wmma_batchable rule (its chunk path has the V2 arms; gfx11 kill-switch intact). llama_spec::batched_verify_eligible routes all seven weights through is_batchable_la, so it is covered without an edit. Lockstep test now asserts the true contract: agreement on every non-V2 dtype across 5 arches; for V2, qwen35 admits on gfx11/gfx12 and llama refuses everywhere. Spec §9 row and crate maps corrected. hipfire-runtime is_batchable_la: 7 passed; qwen35 is_batchable + lockstep: 9 passed.
hw-gate evidence — 2 lane(s) — verdict passlane hiptrx (gfx1201)hw-gate evidence
fixturesqwen3.6:27bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 50.2 status pass
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4chain — exit 0 seconds 42.8 status pass
qwen3.6:27b chain turn 0qwen3.6:27b chain turn 1qwen3.6:27b chain turn 2qwen3.6:27b chain turn 3qwen3.6:27b chain turn 4ornith-1.5:35b-a3b-mq4rsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 88.8 status pass
ornith-1.5:35b-a3b-mq4r battery turn 0ornith-1.5:35b-a3b-mq4r battery turn 1ornith-1.5:35b-a3b-mq4r battery turn 2ornith-1.5:35b-a3b-mq4r battery turn 3ornith-1.5:35b-a3b-mq4r battery turn 4chain — exit 0 seconds 116.5 status pass
ornith-1.5:35b-a3b-mq4r chain turn 0ornith-1.5:35b-a3b-mq4r chain turn 1ornith-1.5:35b-a3b-mq4r chain turn 2ornith-1.5:35b-a3b-mq4r chain turn 3ornith-1.5:35b-a3b-mq4r chain turn 4lfm2.5:1.2bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 9.0 status pass
lfm2.5:1.2b battery turn 0lfm2.5:1.2b battery turn 1lfm2.5:1.2b battery turn 2lfm2.5:1.2b battery turn 3lfm2.5:1.2b battery turn 4chain — exit 0 seconds 20.0 status pass
lfm2.5:1.2b chain turn 0lfm2.5:1.2b chain turn 1lfm2.5:1.2b chain turn 2lfm2.5:1.2b chain turn 3lfm2.5:1.2b chain turn 4qwen3.8:27b-mq4-xtsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 40.3 status pass
qwen3.8:27b-mq4-xt battery turn 0qwen3.8:27b-mq4-xt battery turn 1qwen3.8:27b-mq4-xt battery turn 2qwen3.8:27b-mq4-xt battery turn 3qwen3.8:27b-mq4-xt battery turn 4chain — exit 0 seconds 49.1 status pass
qwen3.8:27b-mq4-xt chain turn 0qwen3.8:27b-mq4-xt chain turn 1qwen3.8:27b-mq4-xt chain turn 2qwen3.8:27b-mq4-xt chain turn 3qwen3.8:27b-mq4-xt chain turn 4kernelstatus: pass report pass: True lane hipx (gfx1100)hw-gate evidence
fixturesqwen3.6:27bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 28.2 status pass
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4chain — exit 0 seconds 18.9 status pass
qwen3.6:27b chain turn 0qwen3.6:27b chain turn 1qwen3.6:27b chain turn 2qwen3.6:27b chain turn 3qwen3.6:27b chain turn 4ornith-1.5:35b-a3b-mq4rsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 36.2 status pass
ornith-1.5:35b-a3b-mq4r battery turn 0ornith-1.5:35b-a3b-mq4r battery turn 1ornith-1.5:35b-a3b-mq4r battery turn 2ornith-1.5:35b-a3b-mq4r battery turn 3ornith-1.5:35b-a3b-mq4r battery turn 4chain — exit 0 seconds 21.9 status pass
ornith-1.5:35b-a3b-mq4r chain turn 0ornith-1.5:35b-a3b-mq4r chain turn 1ornith-1.5:35b-a3b-mq4r chain turn 2ornith-1.5:35b-a3b-mq4r chain turn 3ornith-1.5:35b-a3b-mq4r chain turn 4lfm2.5:1.2bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 8.2 status pass
lfm2.5:1.2b battery turn 0lfm2.5:1.2b battery turn 1lfm2.5:1.2b battery turn 2lfm2.5:1.2b battery turn 3lfm2.5:1.2b battery turn 4chain — exit 0 seconds 14.5 status pass
lfm2.5:1.2b chain turn 0lfm2.5:1.2b chain turn 1lfm2.5:1.2b chain turn 2lfm2.5:1.2b chain turn 3lfm2.5:1.2b chain turn 4qwen3.8:27b-mq4-xtsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 19.4 status pass
qwen3.8:27b-mq4-xt battery turn 0qwen3.8:27b-mq4-xt battery turn 1qwen3.8:27b-mq4-xt battery turn 2qwen3.8:27b-mq4-xt battery turn 3qwen3.8:27b-mq4-xt battery turn 4chain — exit 0 seconds 22.3 status pass
qwen3.8:27b-mq4-xt chain turn 0qwen3.8:27b-mq4-xt chain turn 1qwen3.8:27b-mq4-xt chain turn 2qwen3.8:27b-mq4-xt chain turn 3qwen3.8:27b-mq4-xt chain turn 4kernelstatus: pass report pass: True |
hw-gate sol verdict{
"claim_verdict": "not-exercised",
"confidence": 0.94,
"coverage": {
"gaps": [
"The successful fixture runs do not inject allocation failures, so transactional cleanup in DflashWeights::load, DflashScratch constructors, and build_generic_dflash_speculator was not exercised.",
"No route provoked make_spec_emitter failure after prefill, so live KV/DeltaNet/drafter rollback was not exercised.",
"The serve evidence does not establish that DFlash was active; the kernel report explicitly records draft=null, and the fixture rows contain no speculation attestation.",
"No hardware route forced the draft-context exhaustion boundary and observed finish_reason=length plus cache suppression.",
"The new --prompt-file path and emitted prompt evidence were unit-tested but not exercised by the hardware routes.",
"The MQ4-V2 parity example itself was not run; ordinary qwen3.8 generation establishes coherent admission on available artifacts but not the new disjoint-halves oracle."
],
"surfaces_evidenced": [
"load",
"serve",
"kernel",
"context-admission"
],
"surfaces_touched": [
"load",
"serve",
"kernel",
"speculative-decode",
"state-rollback",
"context-admission",
"cli",
"filesystem",
"docs"
]
},
"decision": "needs-human",
"eyeball": [
"qwen3.6:27b battery and chain outputs on gfx1201/gfx1100 are coherent, answer their prompts, and show useful multi-turn cache reuse.",
"qwen3.8:27b-mq4-xt outputs are coherent on both lanes, supporting successful MQ4-V2 load/prefill/generation after the admission changes.",
"ornith-1.5:35b-a3b-mq4r and lfm2.5:1.2b remained operational; some LFM text contains replacement characters or weak factual phrasing, but no attractor, empty response, or special-token leakage demonstrates a PR regression."
],
"phase": "verdict",
"rationale": "All hardware routes passed, decoded text was generally coherent, artifact hashes matched, and Redline reported stable captures with bit-exact HIP/blob/AQL state on gfx1201 and gfx1100. No regression is evidenced. However, the PR's central behavior is failure-path GPU cleanup and post-prefill speculative-state rollback at crates/hipfire-runtime/src/dflash.rs and crates/hipfire-generate/src/qwen.rs:3121, none of which successful loads can prove. The context-exhaustion terminal path at qwen.rs:3382 was likewise not reached, and the evidence does not show an active draft. Because these are GPU ownership and speculative state-machine changes with material untested fault paths, human review is required despite clean happy-path hardware evidence.",
"regressions": []
}Floor: hard=[] soft=["coverage_gaps: ['The successful fixture runs do not inject allocation failures, so transactional cleanup in DflashWeights::load, DflashScratch constructors, and build_generic_dflash_speculator was not exercised.', 'No route provoked make_spec_emitter failure after prefill, so live KV/DeltaNet/drafter rollback was not exercised.', 'The serve evidence does not establish that DFlash was active; the kernel report explicitly records draft=null, and the fixture rows contain no speculation attestation.', 'No hardware route forced the draft-context exhaustion boundary and observed finish_reason=length plus cache suppression.', 'The new --prompt-file path and emitted prompt evidence were unit-tested but not exercised by the hardware routes.', 'The MQ4-V2 parity example itself was not run; ordinary qwen3.8 generation establishes coherent admission on available artifacts but not the new disjoint-halves oracle.']", 'model needs-human'] model_decision=needs-human final=needs-human |
There was a problem hiding this comment.
hw-gate sol verdict block: Blocking because a mandatory fixture failed and two central claims are contradicted by the diff. Seven other fixture/lane combinations produced coherent output, but gfx1201 qwen3.8 chain emitted an empty turn. More fundamentally, new_windowed still leaks locally staged tensors on later allocation failure, and the <= admission predicate does not match the loop's >= boundary. The rollback and failure-path ownership claims were not exercised by hardware.
|
announcement: Merging to staging. Beyond the mandatory batteries (which ran without a draft), I ran qwen3.6:27b with qwen3.6:27b-draft active on gfx1201 against both this branch and master under greedy sampling: all five turns are byte-identical with identical tau through the metadata-windowed, explicit-window (HIPFIRE_DFLASH_WINDOW=1024) and Legacy (WINDOW=0, CTX_CAP=512) draft constructors, so the transactional rewrite leaves the success path untouched. The unified ctx-cap predicate shows up on hardware exactly as described: with cap 512 and block 16, prompts landing in the prompt+max_tokens+block band error on master after gen_start and complete via AR on this branch. The mq4v2_gemm_parity disjoint-halves arm passes on the R9700. The allocation-failure and make_spec_emitter fault paths remain unexercised on hardware — a fault-injection knob on alloc_tensor would let a future gate close that, if you want to follow up. override: Override Sol needs-human: Sol's gaps were that no route showed DFlash active, the ctx-cap boundary was never forced, and the new parity arm was not run. I closed all three on hardware: DFlash-active greedy A/B (PR vs master) is byte-identical on all five turns with identical tau through both the metadata-windowed and explicit-window draft constructors; the Broken-5 band (prompt+max_tokens <= cap < prompt+max_tokens+block) errors on master and completes coherently via AR on the PR; mq4v2_gemm_parity incl. the disjoint-halves arm passes on gfx1201. What remains (allocation-failure cleanup, make_spec_emitter failure rollback) is unprovable on any host without a fault-injection harness, the success paths those rewrites sit on are proven identical to master, and master's behavior on those paths is a leak / dirty-GPU exit, so a human reading the diff cannot learn more than the A/B already shows. investigation: | question | route | result | evidence | |---|---|---|---| | Does the transactional rewrite of DflashWeights::load + DflashScratch::new_windowed (the draft's metadata declares W=2048, so windowed is the default path for qwen3.6:27b-draft) leave the DFlash happy path identical to master? | HIP_VISIBLE_DEVICES=0|1 HIPFIRE_KERNEL_CACHE= HIPFIRE_CLI_BIN/HIPFIRE_DAEMON_BIN=$HW_GATE_BIN|$HW_GATE_BASE_BIN python3 scripts/serve_harness.py --model qwen3.6-27b.mq4 --tag qwen3.6:27b --draft qwen36-27b-dflash-mq4.hfq --dflash on --sampling greedy --thinking off --mode battery --max-tokens 256 | Serve log attests 'DFlash draft loaded (layers=5, hidden=5120, block=16)' and drafter=dflash on every request on both binaries. All 5 turns byte-identical PR vs master; tau 9.00 / 7.23 / 1.73 / 1.02 / 1.97 on both; gen 120/256/81/116/91 on both; no attractor, no empty row. Decoded text coherent (merge_sorted two-pointer implementation, 210-mile train worked example, axial-tilt seasons answer, lighthouse story, five coding tips). | ab_summary.txt | | Does new_windowed still work with an explicit HIPFIRE_DFLASH_WINDOW override (the alloc_or_free! ladder and the reordered positions_k replacement)? | same as above with HIPFIRE_DFLASH_WINDOW=1024 on the PR binary | Serve log: 'DFlash draft windowed: SWA W=1024 rows'. All 5 turns byte-identical to the master run, identical tau (9.00/7.23/1.73/1.02/1.97). | C2_pr_window1024_isocache.{json,serve.log} | | Does the Legacy (non-windowed) path — DflashScratch::new_with_mq via the at!/take! slot rewrite — still generate, and does the unified spec_ctx_request_fits predicate actually change behavior in the prompt+max_tokens+block_size band (audit Broken 5)? | HIPFIRE_DFLASH_WINDOW=0 HIPFIRE_DFLASH_CTX_CAP=512 ... serve_harness.py ... --dflash on --sampling greedy --thinking off --mode battery --max-tokens 470 --max-seq 4096, on PR and base binaries | Both logs: 'DFlash draft ctx capped: 4096 -> 512 rows'. Prompt 3 (ctx 25: 25+470+16=511 fits) runs DFlash tau=1.73 on both, identical text. Prompts 1-2 (ctx 44/55, exceed 512 outright) fall back to AR on both, identical text. Prompts 4 and 5 (ctx 33/31: 503/501 <= 512 < 519/517) are the band: master emits '[context_length retryable=false rolled_back=false attempt=4|5] prompt+max_tokens exceeds ctx_capacity 512' with gen=0 empty rows; PR falls back to AR at entry and finishes stop with gen 100 / 91, coherent text ('For decades, Elias had watched the storm-tossed waves...', 'Write clear, descriptive names...'). | Dpr2_legacy_ctxcap512_band.{json,serve.log} vs Dbase2_legacy_ctxcap512_band.{json,serve.log} | | Does the new disjoint-halves arm of mq4v2_gemm_parity (stacked #690 content Sol flagged as un-run) pass on gfx12? | HIP_VISIBLE_DEVICES=0 $HW_GATE_BIN/examples/mq4v2_gemm_parity (binary contains the 'disjoint batch' arm; verified via strings) | PASS. Gaussian residual v2 rel-rms 2.65e-4..2.80e-4 at batch 1/8/12/16/32 (0.29-0.33x the v1 floor); disjoint arm v2 rel-rms 1.85e-4..3.07e-4 against swapped-header negative control 0.53..1.62 (>=100x separation); gate_up/qkvza v2 2.6e-4. | mq4v2_gemm_parity.gfx1201.log | | First batch (kept as a failed experiment): the same five routes run concurrently against the shared cwd-relative kernel cache. | identical harness invocations without HIPFIRE_KERNEL_CACHE isolation, five daemons (PR and base) in parallel from the checkout cwd | Three requests failed with 'post-publish pair invalid for gemm_qkvza_hfq4g256_wmma_gfx12 | attention_dflash_wmma_f32_gfx12 | conv1d_silu_split' (rdna-compute/src/compiler.rs:1166-1173, pair_valid after publish into the shared .hipfire_kernels/gfx1201 dir per compiler.rs:322-325). This is a cross-process publish race of my own making, not PR behavior: with per-run caches the identical routes show 0 failures across 5 daemons and the surviving first-batch requests already matched master's tau exactly. Not counted as a regression; noted so the maintainer does not misread those logs. | A_pr_dflash_battery.serve.log, C_pr_dflash_windowed.serve.log (and the other three first-batch files) | unproven:
rationale: Hard floor clean. The diff against master is three stacked PRs; #689 (bench --prompt-file) and #690 (MQ-V2 admit rule + parity arm) were gated separately and I re-ran the #690 parity example here on gfx1201 (mq4v2_gemm_parity.gfx1201.log: PASS, disjoint arm discriminating at >=100x). For #691 proper, the risk in a slot-index rewrite of GPU constructors is a silently swapped or double-taken tensor on the SUCCESS path — that is what hardware can prove, and it is proven: with DFlash attested active (serve logs 'DFlash draft loaded ... block=16', drafter=dflash per request) the PR and master daemons produce byte-identical greedy text with identical tau on all five prompts through the metadata-windowed constructor (A2 vs B2), the explicit-window constructor (C2 vs B2), and the Legacy new_with_mq constructor (Dpr2 prompt 3 vs Dbase2 prompt 3). The mandatory evidence had draft=null everywhere, so this is the coverage Sol was missing. The one behavior change reachable without fault injection, audit Broken 5, is demonstrated as a strict improvement: in the band prompt+max_tokens <= cap < prompt+max_tokens+block_size (Dpr2/Dbase2 prompts 4 and 5, cap 512, block 16, ctx 33/31, max_tokens 470) master returns a context_length error with an empty turn after gen_start, the PR completes via AR with coherent text, and every prompt outside the band is identical on both binaries. Sol's two suspected regressions are not regressions against master: set_dflash_extract_layers before new_spec_scratch (dflash_generic.rs:1071-1079) is master's ordering with only frees added, and the load_codebook panic path is unchanged. The residual unproven items are fault-injection paths where master's behavior is a leak or a dirty-GPU exit and the PR's is a free or a rollback; neither can be exercised on any host with registry fixtures, so holding for a human would not add information beyond this A/B. The first-batch 'post-publish pair invalid' failures are documented in $HW_GATE_EVIDENCE/README.md as a shared-kernel-cache race from running PR and master daemons concurrently in one cwd; isolated-cache reruns show zero failures. soft floor: ["coverage_gaps: ['The successful fixture runs do not inject allocation failures, so transactional cleanup in DflashWeights::load, DflashScratch constructors, and build_generic_dflash_speculator was not exercised.', 'No route provoked make_spec_emitter failure after prefill, so live KV/DeltaNet/drafter rollback was not exercised.', 'The serve evidence does not establish that DFlash was active; the kernel report explicitly records draft=null, and the fixture rows contain no speculation attestation.', 'No hardware route forced the draft-context exhaustion boundary and observed finish_reason=length plus cache suppression.', 'The new --prompt-file path and emitted prompt evidence were unit-tested but not exercised by the hardware routes.', 'The MQ4-V2 parity example itself was not run; ordinary qwen3.8 generation establishes coherent admission on available artifacts but not the new disjoint-halves oracle.']", 'model needs-human'] |
There was a problem hiding this comment.
Blocked by the hard floor only: the gate's qwen3.8:27b-mq4-xt chain on gfx1201 produced one empty turn (a first-token EOS at temperature 1.0 under the MTP drafter), and a failed mandatory fixture is a block by rule. I could not reproduce it in 12 fresh runs (60 turns, PR and master, MTP and plain-AR configs), and nothing in this diff can produce a one-token natural stop, so a gate re-run is what should clear it. On the substance: the mandatory routes never loaded a draft (dflash_mode=off), so I ran qwen3.6:27b and qwen3.8:27b with their mq4 drafts on both binaries — the rewritten load/scratch/windowed constructors decode coherently with normal tau, and the Broken-5 fix is demonstrated head-to-head (master errors after gen_start with 'prompt+max_tokens exceeds ctx_capacity 291', this branch falls back to AR and answers). Sol's '<=' vs '>=' concern does not hold up (position trails prompt+generated, so the equality case cannot exhaust early). One thing to finish before this lands: in DflashScratch::new_windowed the four extension tensors are still locals when a later allocation fails, so alloc_or_free! frees s but leaks them — install each into s right after its allocation (s.free_gpu already frees the Some fields), leaving only new_positions_k as a local, and soften the PR body's 'every allocation site' claim accordingly. The hard floor fired on an evidence failure (hiptrx/gfx1201 qwen3.8:27b-mq4-xt chain, turn 4 empty), so the decision is block by rule, not by my reading of the diff. My reading disagrees with two of Sol's three regressions and I demonstrated the PR's central fix on hardware. (1) The empty turn: gen=1 finish=stop cached=0 at registry sampling temp=1.0/top_p 0.95/top_k 20 under the MTP drafter (that fixture auto-loads /home/kaden/qcal/ladder-v2/artifacts/qwen3.8-27b.mq4v2.xt.mtp, drafter=mtp in every serve log — so it does cross generate_spec). I reran it 12 times on gfx1201 (PR x9 across the gate's MTP config and plain AR, BASE x3): 60 turns, zero empties, every lighthouse turn coherent ($HW_GATE_EVIDENCE/INDEX.md). No line in this diff can yield a one-token natural stop: a ctx_exhausted break maps to length (qwen.rs:2442-2444, 2651-2657) and the emitter-error arm emits a correlated fail-closed error (qwen.rs:3131-3141). It is a stochastic first-token EOS, present on master's sampling policy too; a gate re-run is what clears the floor. (2) Sol's regression #2 (the '<='/'>=' mismatch, common.rs:589 vs qwen.rs:3384) is arithmetically wrong: position lags prompt+generated by at least one (qwen.rs:3296, 3441-3491, 3528-3536, 5543), so at exact equality the guard would need generated >= max_tokens+1, impossible under 3370/3396. (3) Sol's regression #1 (dflash.rs:1491-1504) is real but is an incomplete cleanup, strictly better than master at every failure point (master: bare '?', whole s leaked, e98b461 dflash.rs:1349-1356); no behavior on the success path changes, and the PR body overclaims it. (4) The touched surface — DFlash weight/scratch construction including new_windowed (both registry drafts declare a window, so it runs on the default path) — was NOT exercised by the mandatory routes: run.py leaves HIPFIRE_HOME at the gate home so the harness's [speculation] TOML in /.hipfire/config.toml is shadowed and the daemon logged 'dflash_mode=off — skipping draft load' (visible in the gate's own serve logs and in my first attempt). I fixed that in the runner and ran qwen3.6:27b and qwen3.8:27b with their mq4 drafts on PR and BASE: 40 chain/battery turns per binary set, drafter=dflash on every request, tau in the normal band and matched between binaries, recall 11/11, plus W=512 override and Legacy-mode runs. (5) Broken 5 is demonstrated head-to-head: Legacy draft, HIPFIRE_DFLASH_CTX_CAP=291, 31-token prompt, max_tokens 256, block 16 — master admits the request and dies after gen_start with 'context_length ... prompt+max_tokens exceeds ctx_capacity 291' (empty turn, rolled_back=false), the PR falls back to AR at entry and answers correctly (pr-/base-q38-legacy-cap291-band-battery). The Legacy cap=400 chain shows the DFlash->AR handoff is clean on both binaries with the prefix cache extending across it. The gate-infra shadowing of --dflash by HIPFIRE_HOME is worth a maintainer look: it means DFlash PRs get dflash-off mandatory routes.
…id-ladder failure frees all of them hw-gate Fable seat on #691 (run 33900101473): alloc_or_free! freed the base scratch on failure but the four already-allocated tensors were still locals with no Drop — a failure on the 2nd..5th alloc leaked k_full / v_full / k_cat / v_cat. They are now assigned into s as each succeeds, so the error arm's s.free_gpu covers the whole ladder. Same shape as new_with_mq's at!/live list.
|
Gate run 33900101473: Fable reproduced neither the failed fixture nor any regression — the hiptrx chain's empty turn was a first-token EOS at temperature 1.0 under the MTP drafter (the host has One real gap from the same run, fixed in |
…llama and qwen35; discriminating GEMM parity; spec §9) to staging
There was a problem hiding this comment.
hw-gate sol verdict needs-human: Hardware batteries passed on gfx1201 and gfx1100, but they only establish ordinary load/serve compatibility. The central failure and rollback paths were not exercised. More importantly, the claimed single context-cap predicate is not equivalent to the loop guard: common.rs:584 accepts equality while qwen.rs:3384 rejects equality. The exact-cap request therefore remains admitted into speculation instead of being diverted to AR, so the stated Broken 5 fix is incomplete.
There was a problem hiding this comment.
Ran DFlash for real on both binaries (the gate's batteries had dflash_mode=off): qwen3.6:27b and qwen3.8:27b with their MQ4 drafts in windowed, Legacy-capped and DFlash2 shapes, battery and chain, greedy — decoded text, tau and cycle counts are identical to master across 20 turns, so the rewritten constructors are behaviour-preserving on the success path. The Broken 5 fix is demonstrated: with HIPFIRE_DFLASH_CTX_CAP=290 master returns a context_length error after gen_start on three of five prompts, the PR falls back to AR and answers all five; and the exact-cap request (55+256+16 == 327) runs speculation to the full budget, so the reviewer's off-by-one early-termination claim does not hold. Merging to staging. Still unexercised on hardware: the allocation-failure and emitter-failure cleanup paths (need fault injection) and ctx_exhausted under CASK eviction — worth a diagnostic env hook if you want those on the record later. The mandatory batteries ran with dflash_mode=off, so none of the touched code executed in the hw-gate evidence; Sol was right to flag that and wrong about the regression. I ran DFlash on PR and master binaries (daemon md5 f30d756c vs 17ba7dfa) on 4x gfx1201 with greedy sampling so text is comparable: qwen3.6:27b + qwen36-27b-dflash-mq4.hfq battery and chain (draft-declared windowed W=2048 -> new_windowed over new_with_mq, dflash.rs:1442-1700), qwen3.8:27b + qwen38-27b-dflash-mq4.hfq battery (DFlash2 all-sliding, conv/selector Option slots, block=8), and Legacy capped mode (HIPFIRE_DFLASH_WINDOW=0, 'ctx capped' path -> plain new_with_mq). Across all 20 A/B turns decoded content, reasoning, tau and cycle counts are identical PR vs master (README.md in $HW_GATE_EVIDENCE indexes the files). The rewritten constructors therefore land every tensor in the same field as before. For Broken 5: with cap 290 master emits '[context_length ... rolled_back=false] prompt+max_tokens exceeds ctx_capacity 290' after gen_start and three of five battery turns come back empty (base_q36_dflash_legacy_ctxcap290_battery_gfx1201.serve.log:90-96); the PR serves all five via AR, coherently. For Sol's off-by-one at common.rs:584 vs qwen.rs:3384: the loop is while generated < max_tokens (qwen.rs:3364) and position never exceeds prompt+generated without eviction (position init 3157, advances at 3536/3905 lag by the pending seed), so position+block_size <= cap-1 whenever the entry predicate holds; on hardware the exact-cap request (cap 327, ctx 55, block 16) ran DFlash to gen=256/finish=length with tau 7.23 and 31 cycles, identical to the uncapped run, and cap 326 diverted it to AR. The entry predicate is also exactly the negation of master's generate_spec error at e98b461 qwen.rs:2980-2999, so the set of requests admitted into speculation is unchanged; only the band moved from a post-gen_start error to a pre-gen_start AR fallback. The hard floor did not fire; no attractors, no empties on the PR binary in any run. What remains unproven is the new error plumbing itself (needs fault injection) — but that plumbing replaces a path that leaked on master, and the success path is evidenced identical, so it cannot make an existing model, topology or serve path worse. Override Sol needs-human: Sol's only stated regression (exact-cap request enters speculation and terminates early because spec_ctx_request_fits uses <= while the loop guard uses >=) is refuted both by argument and on hardware: with no eviction the loop runs only while generated < max_tokens, and position <= prompt+generated, so position+block_size <= prompt+max_tokens-1+block_size <= cap-1 and the qwen.rs:3384 guard cannot fire; on gfx1201 the exact-cap request (ctx 55 + 256 + block 16 == cap 327) ran DFlash to the full 256-token budget (tau 7.23, 31 cycles), identical text to the uncapped run, while cap 326 diverted it to AR. Sol's coverage gaps that this host can close (DFlash actually enabled, rewritten weight/scratch constructors on their success path in windowed, Legacy and DFlash2 shapes, near-cap AR fallback, multi-turn state under DFlash) were closed with PR-vs-master A/B runs that are byte-identical in decoded text, tau and cycle counts across 20 turns, and the band case master breaks (context_length error after gen_start, 3/5 turns empty) the PR serves coherently.
There was a problem hiding this comment.
hw-gate sol verdict needs-human: The ordinary serve and architecture paths are coherent on gfx1100 and gfx1201, and Redline reports stable captures plus bit-exact HIP/AQL state. However, the core PR behavior was not run because every fixture loaded with draft=null. Moreover, the claimed unified capacity boundary is contradicted by the diff: spec_ctx_request_fits uses <= at crates/hipfire-generate/src/common.rs:583, while generate_spec breaks when position + block_size >= ctx_capacity at crates/hipfire-generate/src/qwen.rs:3384. Equality is therefore admitted at entry but rejected in-loop rather than falling back to AR. A human must resolve that boundary and assess the unexercised allocation-failure and emitter-rollback paths before promotion.
…er-error rollback, one ctx-cap predicate) to staging Evidence: run 33905366422 — hiptrx gfx1201 + hipx gfx1100 both pass (4 fixtures x battery+chain, 0 attractor/empty/runaway); Sol verdict needs-human with its one technical claim (entry vs loop ctx guard off-by-one) DISPROVEN on hardware in run 33902688013, where Fable also ran 20 DFlash turns identical to master and demonstrated the Broken-5 fix head-to-head. new_windowed alloc-ladder leak found in that run is fixed in 136e3f0. Merged by Main (Fable seat unavailable: credits).
|
Merged to |
There was a problem hiding this comment.
hw-gate sol verdict needs-human: All hardware routes passed, decoded text was generally coherent, artifact hashes matched, and Redline reported stable captures with bit-exact HIP/blob/AQL state on gfx1201 and gfx1100. No regression is evidenced. However, the PR's central behavior is failure-path GPU cleanup and post-prefill speculative-state rollback at crates/hipfire-runtime/src/dflash.rs and crates/hipfire-generate/src/qwen.rs:3121, none of which successful loads can prove. The context-exhaustion terminal path at qwen.rs:3382 was likewise not reached, and the evidence does not show an active draft. Because these are GPU ownership and speculative state-machine changes with material untested fault paths, human review is required despite clean happy-path hardware evidence.
There was a problem hiding this comment.
Merging to staging. Beyond the mandatory batteries (which ran without a draft), I ran qwen3.6:27b with qwen3.6:27b-draft active on gfx1201 against both this branch and master under greedy sampling: all five turns are byte-identical with identical tau through the metadata-windowed, explicit-window (HIPFIRE_DFLASH_WINDOW=1024) and Legacy (WINDOW=0, CTX_CAP=512) draft constructors, so the transactional rewrite leaves the success path untouched. The unified ctx-cap predicate shows up on hardware exactly as described: with cap 512 and block 16, prompts landing in the prompt+max_tokens+block band error on master after gen_start and complete via AR on this branch. The mq4v2_gemm_parity disjoint-halves arm passes on the R9700. The allocation-failure and make_spec_emitter fault paths remain unexercised on hardware — a fault-injection knob on alloc_tensor would let a future gate close that, if you want to follow up. Hard floor clean. The diff against master is three stacked PRs; #689 (bench --prompt-file) and #690 (MQ-V2 admit rule + parity arm) were gated separately and I re-ran the #690 parity example here on gfx1201 (mq4v2_gemm_parity.gfx1201.log: PASS, disjoint arm discriminating at >=100x). For #691 proper, the risk in a slot-index rewrite of GPU constructors is a silently swapped or double-taken tensor on the SUCCESS path — that is what hardware can prove, and it is proven: with DFlash attested active (serve logs 'DFlash draft loaded ... block=16', drafter=dflash per request) the PR and master daemons produce byte-identical greedy text with identical tau on all five prompts through the metadata-windowed constructor (A2 vs B2), the explicit-window constructor (C2 vs B2), and the Legacy new_with_mq constructor (Dpr2 prompt 3 vs Dbase2 prompt 3). The mandatory evidence had draft=null everywhere, so this is the coverage Sol was missing. The one behavior change reachable without fault injection, audit Broken 5, is demonstrated as a strict improvement: in the band prompt+max_tokens <= cap < prompt+max_tokens+block_size (Dpr2/Dbase2 prompts 4 and 5, cap 512, block 16, ctx 33/31, max_tokens 470) master returns a context_length error with an empty turn after gen_start, the PR completes via AR with coherent text, and every prompt outside the band is identical on both binaries. Sol's two suspected regressions are not regressions against master: set_dflash_extract_layers before new_spec_scratch (dflash_generic.rs:1071-1079) is master's ordering with only frees added, and the load_codebook panic path is unchanged. The residual unproven items are fault-injection paths where master's behavior is a leak or a dirty-GPU exit and the PR's is a free or a rollback; neither can be exercised on any host with registry fixtures, so holding for a human would not add information beyond this A/B. The first-batch 'post-publish pair invalid' failures are documented in $HW_GATE_EVIDENCE/README.md as a shared-kernel-cache race from running PR and master daemons concurrently in one cwd; isolated-cache reruns show zero failures. Override Sol needs-human: Sol's gaps were that no route showed DFlash active, the ctx-cap boundary was never forced, and the new parity arm was not run. I closed all three on hardware: DFlash-active greedy A/B (PR vs master) is byte-identical on all five turns with identical tau through both the metadata-windowed and explicit-window draft constructors; the Broken-5 band (prompt+max_tokens <= cap < prompt+max_tokens+block) errors on master and completes coherently via AR on the PR; mq4v2_gemm_parity incl. the disjoint-halves arm passes on gfx1201. What remains (allocation-failure cleanup, make_spec_emitter failure rollback) is unprovable on any host without a fault-injection harness, the success paths those rewrites sit on are proven identical to master, and master's behavior on those paths is a leak / dirty-GPU exit, so a human reading the diff cannot learn more than the A/B already shows.
…iffs Two policy gaps this ladder exposed. 1. The gate never ran DFlash. #686 (draft sidecars), #691 (draft ctor rollback), #692 (primer replay) and #702 (dedicated verify kernels) all went through with every lane green while speculation never once executed. #692's DFlash-arm defect -- primer replay systematically missing the most recent assistant body -- was found only because a seat thought to drive twenty turns by hand. That is not a gate. The load bucket now runs `battery-dflash` and the serve bucket `chain-dflash`: the same prompts with `--dflash on` and an explicit `--draft`. `on` rather than `auto` because `auto` silently falls back to AR when the draft is missing, and a route that can pass without speculating proves nothing. The draft is named explicitly because the canonical xt trunk is a symlink out of the models dir, so the daemon's filename auto-match finds nothing and would run AR. `dflash_draft` is a candidate LIST because the lanes hold different drafts: hiptrx has qwen36-27b-dflash-mq4.hfq and no qwen38, hipx has qwen38-27b-dflash-mq4.hfq and no qwen36. A lane speculates with the first candidate it holds; a lane holding none records `skip`. `skip` is neither pass nor fail. The aggregation was `all(status == "pass")`, which would have counted a skip as a fixture failure -- a false negative on evidence the host never had -- while treating it as a pass would claim coverage that did not happen. Skips are recorded and reported, and a genuine failure alongside a skip still fails. Coverage is asymmetric until both hosts hold both drafts. Pulling qwen38-27b-dflash-mq4.hfq to hiptrx and qwen36-27b-dflash-mq4.hfq to hipx (0.92 GB each) makes it symmetric; that is a disk decision, so the evidence says `skip` rather than silently pulling. 2. Sol refused hardware for any diff touching a filesystem path, which caught #689 for adding `--prompt-file` to `hipfire bench` and cost that rung a lane until `hw-run` overrode it. hipfire is a CLI inference engine: users name models, prompts, drafts and sidecars at invocation, and the gate's own harness passes exactly those flags. sol.md now separates whose path it is -- an explicit argument is ordinary product work; credentials, dotfiles, SSH or cloud config, /proc or /sys beyond device enumeration, assembled traversal, or a read whose result leaves the process still warrant refusal. Tests: eight new cases in scripts/hw-gate/tests/test_run.py covering flag translation (battery-dflash -> `--mode battery --dflash on --draft ...`), plain battery never receiving a draft, per-lane draft selection, skip-not-fail with the harness never invoked, chain-dflash keeping its own prompts, the skip-vs-genuine-failure aggregation, and a manifest assertion that the buckets actually carry the routes. 113/113 hw-gate tests pass.
spawn_serve isolates the daemon by setting HOME=<run home> and writing <run home>/.hipfire/config.toml — but it inherits os.environ, and ConfigPaths::discover prefers HIPFIRE_HOME over $HOME/.hipfire. hw-gate exports HIPFIRE_HOME per lane, so every gate daemon resolved the lane's pinned config (no [speculation] section) instead of the harness's `mtp = "off"`, and the schema default `auto` auto-attached a sibling .mtp head wherever one existed. On warpfront#691's run (33900101473) that made the two lanes diverge on host state: hiptrx has qwen3.8-27b.mq4v2.xt.mtp beside the symlink target and ran the chain under MTP (one degenerate turn); hipx has no sidecar and ran AR. The pre-flight printed `mtp_mode: off` either way — the harness's intent, not what the daemon resolved. Reproduced on hiptrx with the base daemon, same config.toml, `mtp="off"`: inherited HIPFIRE_HOME -> "MTP head loaded"; HIPFIRE_HOME=<run home>/.hipfire -> no head. The env now sets HIPFIRE_HOME to the run home explicitly.
Run 33905366422 (warpfront#691): Fable's decide phase returned a complete, correct investigation — as a markdown report headed `## hw-gate decide — PR warpfront#691: **merge-staging**` instead of a JSON object. review.py reported "no JSON object in assistant text", decision=None, decision_final=hold, and the run went red on a complete green verdict. _markdown_decision() recognizes the decide headline / bold decision word (`**merge-staging**`, `**decision:** hold`, `decision: block`) and synthesizes the decision dict; wired into both decide paths (omp_investigate's no-JSON branch and omp_review's retry loop). fable_raw is now also set on the non-investigate failure path. test_decide_markdown_headline_is_a_decision: headline verdict → merge-staging decision + staging merge; prose without a verdict word → hold with fable_error + raw tail in the artifact. 104/104.
Every rung of the 2026-09-04 ladder hit the same 409 on the staging merge: `crates/*/map.md` carries a `<!-- crate-map:generated -->` block that both branches regenerate, so any two PRs touching the same crate conflict there while their real code merges cleanly. warpfront#689, warpfront#690, warpfront#691, warpfront#686, warpfront#687, warpfront#688 and warpfront#682 all needed the same three manual steps -- merge staging in, regenerate the block with scripts/check-crate-maps.py, merge -- six of them tonight. A gate that decides merge-staging and then holds on a generated file is asking a human to run a script, which is not review. On a 409 the decide phase now retries locally: merge staging into the PR head, and if the conflicted set is generated maps only, re-run check-crate-maps.py for those crates, commit, and merge the result. The retry is deliberately narrow, because auto-resolving conflicts is exactly where a gate can do damage: - if ANY conflicted path is not a `map.md`, it declines and the hold stands with the offending paths named -- a real code conflict must reach a human - it regenerates rather than picking a side, so the committed block is what the tree actually generates, not whichever branch won - a failed regeneration, a git error, or a timeout all decline rather than force Test: `test_generated_map_retry_refuses_real_code_conflicts` builds a real repo with a conflicting `.rs` and asserts the retry returns no merge SHA and names the file. The guard is the part worth pinning; the happy path is exercised by the ladder itself. 122/122 hw-gate tests pass.
Summary
Three verified defects from the DFlash audit (
docs/plans/audit-2026-09-02/audit-Dflash.md, PR #685):DflashWeights::load,DflashScratch::new_with_mq/new_windowed, anddflash_generic::build_generic_dflash_speculator?-ed out mid-construction leaving earlieralloc_tensors allocated (GpuTensor/DeviceBufferhave noDrop). Every allocation site now frees on the error path in theor_free!style ofload_dflash_state;new_windowedfrees the popped base scratch on any post-pop failure; the generic builder frees weights on scratch failure.DflashLayerWeights::free_gpuis shared by the success and error paths. Success path unchanged.make_spec_emitterErr rollback (Broken 4). The one post-prefill error exit ingenerate_specthat returned withoutproduction_fail_closed_rollback_live— leaving advanced target KV / DeltaNet / drafter hidden behind cleared host counters for the next turn to LCP against — now rolls back first, like the adjacentprefillandstepErr arms.generate_dflashfell back to AR atprompt + max_tokens > cap,generate_specthen errored atprompt + max_tokens + block_size > capaftergen_start.common::spec_ctx_request_fitsis used at both sites, so the block-size band falls back to AR at entry. The mid-loopposition + block_size >= capbreak now setsSpecRun::ctx_exhausted, reported asfinish_reason=lengthwith no tool release and no cache store (via the existingqwen_dflash_hit_length_cap→ wire-terminal path) instead of a naturalstopthat could prime the cache.Which crate(s) does this touch?
crates/hipfire-runtime(dflash.rs,dflash_generic.rs),crates/hipfire-generate(qwen.rs,common.rs,dense.rsepilogue)Evidence
710bc645e,HIPFIRE_DFLASH_DRAFTpinned, humaneval fixture md537c5aad9…):hipfire bench --spec dflash202.1 tok/s [200.3, 202.1, 203.4], τ=10.55 — identical to the pairing branch and master-with-pin.hipfire runon the same prompt with thinking on: correctbelow_zero, τ=4.51.034fe7801) and by the unit tests below.Test plan
cargo test -p hipfire-runtime -p hipfire-generate— runtime lib 597, generate lib 16 (newspec_ctx_request_fits_holds_one_block_margin: exact cap, cap±1, theprompt+max==capband, zero block, saturating overflow),qwen_dflash_ctx_exhausted_tests(new file;generated < max_tokensstill terminateslength/ no store) — 0 failures across all suites.Note on layout: the new terminal test lives in its own file because
qwen_dflash_semantic_terminal_tests.rscarries rustfmt debt and CI enforces rustfmt on changed files — adding one test there forces a 6k-line reformat.