fix(prompt-cache): template-aware assistant-primer splice — Qwen3.8 prefix cache hits again - #692
fix(prompt-cache): template-aware assistant-primer splice — Qwen3.8 prefix cache hits again#692Kaden-Schutt wants to merge 69 commits into
Conversation
…down (G2) Classify a retained source once and decide one effective topology before any destructive side effect, so a refused load leaves the prior model usable (issue #666 G2). - admission::admit_source (read-only): open the source, classify arch_id + vision (tower-tensor decides; contract 179a20d), decide the effective topology (single/pp/ep), and refuse no-carrier / ambiguous / VMM allowlist / VMM+pp / carrier-pp / EP-arch / lfm2 vision-no-config / DFlash lm-head quant — all before any GPU/VMM/teardown work. - Carrier::admit_topology (default + qwen35/cohere2moe/maple/gemma4 overrides) mirrors each carrier's load-time pp refusal. - Split the load entries: path wrappers classify-then-load via load_admitted_with_gemma4_drafter / load_model_ep_admitted, which consume the retained SourceAdmission (no re-open, no re-classify). - Reorder the daemon load handler: daemon topology refusals + admission run BEFORE prior-model teardown; on refusal the prior model stays loaded. RATCHET-RAISE: daemon_lines 4155 -> 4176, traded for the ~21-line G2 source-aware admission block inserted before prior-model teardown. Verification: workspace build clean; full workspace test suite passes; qwen3.6:27b (fresh-daemon VMM commit) + qwen3.6-35b-a3b load and generate coherently; a bad-path load and a vmm+pp>1 load both refuse at admission with the prior model still generating afterward.
admit_source refused kv_backend=vmm for every EP load, so a DeepSeek V4 EP + vmm load that master serves (load_model_ep_with_kv_mode arch 9 arm) was refused at admission. Gate the refusal on matches!(arch_id, 5|6|10), mirroring master's per-arch dispatch, and keep the DS4 (9) arm vmm-capable. Also correct the no-reopen claim on the EP path: the per-arch EP loaders re-open path per rank, so the retained SourceAdmission.source is dropped there rather than consumed (single/pp route is unchanged).
Regenerate the hipfire-loader generated map block after the per-arch VMM refusal fix (admission.rs 274 -> 298 lines, 5 -> 6 tests; lib.rs 4879 -> 4881 lines). Keeps scripts/check-crate-maps.py --check green in CI.
The default ~24-token prompt reports prefill_tok_s ~= 363 tok/s while a 4.4k-token prompt on the same binary reports 886: the short-prompt number is launch overhead, not prefill, and the JSON gave no way to tell. - Add --prompt-file <PATH>: prompt read verbatim (raw bytes, no trim), mutually exclusive with positional PROMPT words. - Standard-bench JSON gains top-level prompt_tokens (u64, as the daemon reports it in done.prompt_tokens), prompt_md5 (hex md5 of the exact prompt bytes), prompt_chars, and warnings[]; warn when prompt_tokens < 256 that prefill_tok_s measures launch overhead. - Same three values printed on the stderr banner next to model:/arch:. - No existing field renamed, nulled, or removed; default prompt bytes unchanged (audit 2026-09-02 fix slice: bench-prompt-evidence).
The daemon's done event has no prompt_tokens key; the prompt is prefill_tokens (rows prefilled) plus cached_tokens (prompt-cache prefix). Measured on a 7900 XTX the JSON reported prompt_tokens: null and never warned on the 24-token default prompt.
…ropped hw-gate Fable seat on #689: the flag table lost its `--reasoning-on` row (base AGENTS.md:359) while the flag still exists in `hipfire bench --help`. Additive row only, as the PR body says.
… JSON, short-prompt prefill warning) to staging
llama::is_batchable_la admitted MQ4G256V2/MQ6/5/3/2G256V2 for WMMA
prefill only on gfx1200/gfx1201 while qwen35::is_batchable_la admitted
them on gfx11+gfx12 behind HIPFIRE_MQV2_GFX11_WMMA, so plain Llama/Qwen3
dense qt=44 models prefetched per-token on gfx1100/1151 while Qwen3.5/3.8
took WMMA — despite both doc-comments claiming an exact match (audit
2026-09-02 Broken 1).
Move the dtype set + arch set + kill-switch helper into
llama::mqv2_wmma_batchable / llama::mqv2_gfx11_wmma_enabled_from_env in
hipfire-runtime and delegate from both callers, so the lockstep is
structural. MQ4CG256 (qt=45) stays gfx12-only in both by intent.
Tests: rename the two gfx12-only llama admit tests to gfx11+gfx12
expectations, repoint qwen35 env-escape test at the shared helper, and
add mqv2_admit_llama_qwen35_lockstep asserting both gates agree over
the MQ-V2 dtypes x {gfx1100, gfx1151, gfx1201, gfx1030, gfx1010}.
…ections mq4v2_gemm_parity's Gaussian weights give both halves near-identical headers, so a wrong half-select hides in quantization noise despite the header comment promising a systematic blow-up (audit 2026-09-02 Missing 1). Add arm 2 using the disjoint-halves construction from mq4v2_residual_parity (half0 [-1,1], half1 [96,160]) over the same batch-size sweep: v2 output must match the f32 reference within 5% rel-RMS, and the swapped-headers negative control (as in mq4v2_moe_parity) must DISAGREE. Keep the Gaussian v1-vs-v2 arm and fix the header comment to state what each arm can and cannot detect. Add a host-side test proving the fixture discriminates with no GPU. Docs (audit Would-change 1-2): spec section 9 now records MoE as production-wired for qt=44, the XBATCH single-row path as ported, and the gfx11 kt+=2 / residual kt++ stepping; residual_mmq.hip loses its stale Experimental tag; the gfx12 QKV kernel loses its HYPOTHESIS/scaffold wording for the validated C-map statement.
DflashScratch::new_with_mq, new_windowed, DflashWeights::load, and build_generic_dflash_speculator could '?' out mid-construction, leaking earlier alloc_tensor results (GpuTensor/DeviceBuffer have no Drop). Record each allocation in a slot vec (gt!/wt!/at!) taken once into the final owner; the error arm frees completed layers plus staged slots. new_windowed frees the base scratch via alloc_or_free!; the generic builder frees weights (and scratch) on later failures. Success path is byte-identical. Mirrors load_dflash_state's or_free! (audit-Dflash Broken 3).
The make_spec_emitter Err exit ran after a successful spec.prefill without production_fail_closed_rollback_live, unlike every other post-prefill error exit. The target KV/DeltaNet/drafter hidden had advanced and host seq_pos/conversation_tokens were cleared, so the next turn could LCP against a dirty GPU. Route it through the same rollback + fail-closed error (audit-Dflash Broken 4).
…length generate_dflash fell back to AR only when prompt + max_tokens > ctx_capacity, but generate_spec hard-errors when prompt + max_tokens + block_size > ctx — requests in that band got gen_start followed by an error instead of the promised AR fallback. Both sites now share spec_ctx_request_fits (prompt + max + block <= cap). The mid-loop position + block_size >= ctx_capacity break now sets SpecRun::ctx_exhausted, which the qwen (v2 + legacy) and dense epilogues OR into the length decision: finish_reason=length with no cache store instead of a silent early stop (audit-Dflash Broken 5).
qwen_dflash_semantic_terminal_tests.rs carries historical rustfmt debt; CI enforces rustfmt on changed files, so adding one test there forces a 6k-line reformat. The new contract lives in qwen_dflash_ctx_exhausted_tests.rs and the debt file is restored to master byte-for-byte.
…efill_chunk has no V2 arms hw-gate Fable seat on #690 (run 33895641944), source trace verified: the shared MQ-V2 admit rule made llama::is_batchable_la admit plain Llama/Qwen3-dense qt44/47-50 artifacts to WMMA prefill on gfx11 and gfx12, but llama.rs::forward_prefill_chunk's per-layer matchers (qkv_is_mq ~:2570, wo_is_mq ~:3025, ffn_is_mq ~:3117, w_down_is_mq ~:3248) list only MQ4G256|MQ6G256|MQ3G256|MFP4G32 — an admitted V2 model skips the FWHT rotate and runs the V1 hfq4g256 launchers on V2 blobs: silently incoherent prefill. master's pre-existing mq4_v2_gfx12 arm had the same hole on gfx12; no gfx12 Llama-V2 artifact has ever tripped it. llama::is_batchable_la now refuses every *G256V2 dtype and MQ4CG256 on every arch, with the reason at the site. qwen35::is_batchable_la keeps the shared mqv2_wmma_batchable rule (its chunk path has the V2 arms; gfx11 kill-switch intact). llama_spec::batched_verify_eligible routes all seven weights through is_batchable_la, so it is covered without an edit. Lockstep test now asserts the true contract: agreement on every non-V2 dtype across 5 arches; for V2, qwen35 admits on gfx11/gfx12 and llama refuses everywhere. Spec §9 row and crate maps corrected. hipfire-runtime is_batchable_la: 7 passed; qwen35 is_batchable + lockstep: 9 passed.
…id-ladder failure frees all of them hw-gate Fable seat on #691 (run 33900101473): alloc_or_free! freed the base scratch on failure but the four already-allocated tensors were still locals with no Drop — a failure on the 2nd..5th alloc leaked k_full / v_full / k_cat / v_cat. They are now assigned into s as each succeeds, so the error arm's s.free_gpu covers the whole ladder. Same shape as new_with_mq's at!/live list.
…llama and qwen35; discriminating GEMM parity; spec §9) to staging
…e does not re-emit it Both jinja cache lookups (ar.rs, qwen.rs dflash) prepended the live turn's generation primer to every cached assistant body, on the assumption that the template renders history assistant turns bare. Qwen3.5's does; Qwen3.8's re-emits <think>\n\n</think>\n\n on history turns when thinking is off, so the spliced render carried the primer twice and the LCP died at the first assistant turn of every conversation: every turn re-prefilled from the last checkpoint (2-4k tokens/turn on the coding session; the '2 minutes to first token' complaint). Measured on a 7900 XTX, qwen3.8-27b.mq4, AR, HIPFIRE_QWEN_CACHE_TRACE=1: prior_past[26..] = ```python... rend_past[26..] = <think>\n\n</think>\n\n```python... lcp=26 prior_len=118 rendered_len=148 template_emits_history_primer probes the template with a one-exchange history and decides per template; both lookups use it.
4bc750e to
5b130d1
Compare
hw-gate sol prelimsummary: This head is a large stacked change, not only the advertised prompt-cache fix: it makes assistant-primer replay template-aware, adds registry-managed DFlash sidecars and prompt-file benchmark evidence, introduces source-aware pre-teardown admission and EP mesh labeling, changes DFlash allocation/rollback/context-exhaustion behavior, adds sticky GPU-fault handling, changes LFM batching eligibility and Llama MQ-V2 prefill admission, and updates MQ4-V2 documentation/tests. The two kernel files contain comment-only changes, but runtime dispatch and load/serve state machines materially change. run_hardware: true routes:
unavailable_routes: (none) claim_assessment: The author claims Qwen3.8 thinking-off history templates already emit the empty-think primer, so suppressing a second primer restores full multi-turn LCP/cache hits without changing generated bytes, while Qwen3.5-style behavior remains unchanged. Proof requires a real multi-turn Qwen3.8 chain showing coherent byte-stable outputs and cached-prefix growth after turn one, plus a Qwen3.5/3.6-family control showing no cache regression. The supplied PR-body timings and test counts are claims, not gate evidence. questions_for_author:
|
hw-gate evidence — 2 lane(s) — verdict faillane hiptrx (unknown)hw-gate evidence
fixturesqwen3.6:27bsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: harness exit 1; chain-dflash: harness exit 1 battery — exit 1 seconds 199.3 status failreason: harness exit 1
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4chain — exit 1 seconds 198.9 status failreason: harness exit 1
qwen3.6:27b chain turn 0qwen3.6:27b chain turn 1qwen3.6:27b chain turn 2qwen3.6:27b chain turn 3qwen3.6:27b chain turn 4battery-dflash — exit 1 seconds 198.6 status failreason: harness exit 1 no rows chain-dflash — exit 1 seconds 198.7 status failreason: harness exit 1 no rows ornith-1.5:35b-a3b-mq4rsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft); chain-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft) battery — exit 1 seconds 198.5 status failreason: harness exit 1
ornith-1.5:35b-a3b-mq4r battery turn 0ornith-1.5:35b-a3b-mq4r battery turn 1ornith-1.5:35b-a3b-mq4r battery turn 2ornith-1.5:35b-a3b-mq4r battery turn 3ornith-1.5:35b-a3b-mq4r battery turn 4chain — exit 1 seconds 198.8 status failreason: harness exit 1
ornith-1.5:35b-a3b-mq4r chain turn 0ornith-1.5:35b-a3b-mq4r chain turn 1ornith-1.5:35b-a3b-mq4r chain turn 2ornith-1.5:35b-a3b-mq4r chain turn 3ornith-1.5:35b-a3b-mq4r chain turn 4battery-dflash — exit 0 seconds 0.0 status skipreason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft no rows chain-dflash — exit 0 seconds 0.0 status skipreason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft no rows lfm2.5:1.2bsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft); chain-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft) battery — exit 1 seconds 198.5 status failreason: harness exit 1
lfm2.5:1.2b battery turn 0lfm2.5:1.2b battery turn 1lfm2.5:1.2b battery turn 2lfm2.5:1.2b battery turn 3lfm2.5:1.2b battery turn 4chain — exit 1 seconds 198.3 status failreason: harness exit 1
lfm2.5:1.2b chain turn 0lfm2.5:1.2b chain turn 1lfm2.5:1.2b chain turn 2lfm2.5:1.2b chain turn 3lfm2.5:1.2b chain turn 4battery-dflash — exit 0 seconds 0.0 status skipreason: fixture lfm2.5:1.2b declares no dflash_draft no rows chain-dflash — exit 0 seconds 0.0 status skipreason: fixture lfm2.5:1.2b declares no dflash_draft no rows qwen3.8:27b-mq4-xtsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: harness exit 1; chain-dflash: harness exit 1 battery — exit 1 seconds 198.4 status failreason: harness exit 1
qwen3.8:27b-mq4-xt battery turn 0qwen3.8:27b-mq4-xt battery turn 1qwen3.8:27b-mq4-xt battery turn 2qwen3.8:27b-mq4-xt battery turn 3qwen3.8:27b-mq4-xt battery turn 4chain — exit 1 seconds 198.7 status failreason: harness exit 1
qwen3.8:27b-mq4-xt chain turn 0qwen3.8:27b-mq4-xt chain turn 1qwen3.8:27b-mq4-xt chain turn 2qwen3.8:27b-mq4-xt chain turn 3qwen3.8:27b-mq4-xt chain turn 4battery-dflash — exit 1 seconds 198.4 status failreason: harness exit 1 no rows chain-dflash — exit 1 seconds 198.6 status failreason: harness exit 1 no rows kernelstatus: fail report pass: True lane hipx (gfx1100)hw-gate evidence
fixturesqwen3.6:27bsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery-dflash: harness exit 1; chain-dflash: harness exit 1 battery — exit 0 seconds 30.1 status pass
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4chain — exit 0 seconds 23.9 status pass
qwen3.6:27b chain turn 0qwen3.6:27b chain turn 1qwen3.6:27b chain turn 2qwen3.6:27b chain turn 3qwen3.6:27b chain turn 4battery-dflash — exit 1 seconds 7.1 status failreason: harness exit 1 no rows chain-dflash — exit 1 seconds 7.1 status failreason: harness exit 1 no rows ornith-1.5:35b-a3b-mq4rsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft); chain-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft) battery — exit 0 seconds 34.2 status pass
ornith-1.5:35b-a3b-mq4r battery turn 0ornith-1.5:35b-a3b-mq4r battery turn 1ornith-1.5:35b-a3b-mq4r battery turn 2ornith-1.5:35b-a3b-mq4r battery turn 3ornith-1.5:35b-a3b-mq4r battery turn 4chain — exit 0 seconds 23.4 status pass
ornith-1.5:35b-a3b-mq4r chain turn 0ornith-1.5:35b-a3b-mq4r chain turn 1ornith-1.5:35b-a3b-mq4r chain turn 2ornith-1.5:35b-a3b-mq4r chain turn 3ornith-1.5:35b-a3b-mq4r chain turn 4battery-dflash — exit 0 seconds 0.0 status skipreason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft no rows chain-dflash — exit 0 seconds 0.0 status skipreason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft no rows lfm2.5:1.2bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft); chain-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft) battery — exit 0 seconds 8.0 status pass
lfm2.5:1.2b battery turn 0lfm2.5:1.2b battery turn 1lfm2.5:1.2b battery turn 2lfm2.5:1.2b battery turn 3lfm2.5:1.2b battery turn 4chain — exit 0 seconds 13.5 status pass
lfm2.5:1.2b chain turn 0lfm2.5:1.2b chain turn 1lfm2.5:1.2b chain turn 2lfm2.5:1.2b chain turn 3lfm2.5:1.2b chain turn 4battery-dflash — exit 0 seconds 0.0 status skipreason: fixture lfm2.5:1.2b declares no dflash_draft no rows chain-dflash — exit 0 seconds 0.0 status skipreason: fixture lfm2.5:1.2b declares no dflash_draft no rows qwen3.8:27b-mq4-xtsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery-dflash: harness exit 1; chain-dflash: harness exit 1 battery — exit 0 seconds 19.6 status pass
qwen3.8:27b-mq4-xt battery turn 0qwen3.8:27b-mq4-xt battery turn 1qwen3.8:27b-mq4-xt battery turn 2qwen3.8:27b-mq4-xt battery turn 3qwen3.8:27b-mq4-xt battery turn 4chain — exit 0 seconds 21.2 status pass
qwen3.8:27b-mq4-xt chain turn 0qwen3.8:27b-mq4-xt chain turn 1qwen3.8:27b-mq4-xt chain turn 2qwen3.8:27b-mq4-xt chain turn 3qwen3.8:27b-mq4-xt chain turn 4battery-dflash — exit 1 seconds 7.1 status failreason: harness exit 1 no rows chain-dflash — exit 1 seconds 7.1 status failreason: harness exit 1 no rows kernelstatus: pass report pass: True |
hw-gate sol verdict{
"claim_verdict": "disproven",
"confidence": 0.98,
"coverage": {
"gaps": [
"No successful DFlash generation route evidenced speculative decode, context-exhaustion handling, or emitter-error rollback.",
"No fault-injection evidence covered transactional GPU allocation rollback or sticky 700/719 poisoning.",
"No multi-GPU EP/PP route exercised the new admission and DeviceMesh topology behavior.",
"Registry pull/rm sidecar lifecycle and policy changes were not exercised end to end.",
"The gfx12/hiptrx lane timed out or failed every fixture route after about 198 seconds, and its Redline run timed out on a subsequent load despite an internally passing first report.",
"No successful MQ4-V2 parity run covered the changed gfx12 dispatch-sensitive surface."
],
"surfaces_evidenced": [
"kernel",
"load",
"serve",
"prompt-cache"
],
"surfaces_touched": [
"kernel",
"load",
"serve",
"prompt-cache",
"speculative-decode",
"rollback",
"multi-gpu-topology",
"registry-policy",
"config",
"filesystem",
"docs"
]
},
"decision": "block",
"eyeball": [
"qwen3.8:27b-mq4-xt gfx1100 chain: responses are semantically coherent, but all five turns report cached=0, directly disproving the central cache-hit claim.",
"qwen3.6:27b gfx1100 chain: later turns show substantial cache hits, demonstrating that the harness can observe working prefix caching.",
"lfm2.5:1.2b gfx1100 chain turn 2: inspect the repeated `Distance\ufffd\ufffd` replacement characters.",
"All hiptrx fixture routes should be inspected as infrastructure or device-selection failures: coherent rows were produced, but every route ended with exit 1 after roughly 198 seconds.",
"The paired-Qwen DFlash logs need inspection because both models failed before emitting a single decoded row."
],
"phase": "verdict",
"rationale": "Block: the central Qwen3.8 claim is directly disproven by the gfx1100 chain, where cached remained zero after turn one. Both Qwen DFlash routes also failed with no output, and the mandatory kernel lane is globally failed because hiptrx timed out during load. Although the gfx1100 Redline report showed stable sequences and exact HIP/AQL/PM4 parity, substantial rollback, topology, policy, DFlash, and gfx12 coverage remains absent.",
"regressions": [
{
"beta_behavior": "The advertised template-aware primer probe did not restore caching on the tested Qwen3.8 fixture: every turn in the gfx1100 qwen3.8:27b-mq4-xt chain reported cached=0.",
"evidence": "hipx/gfx1100 qwen3.8:27b-mq4-xt chain: cached values were 0,0,0,0,0, while the qwen3.6:27b control produced later-turn cache hits of 160/44/579/720. This directly contradicts the claimed Qwen3.8 cache restoration.",
"file": "crates/hipfire-runtime/src/prompt_frame.rs",
"line": 1498,
"master_behavior": "Qwen3.8 multi-turn Jinja replay should extend the prior prompt-cache prefix after turn one.",
"severity": "high"
},
{
"beta_behavior": "Both paired Qwen fixtures failed before producing any DFlash rows.",
"evidence": "hipx qwen3.6:27b battery-dflash and chain-dflash exited 1 after about 7.1 seconds with zero rows; qwen3.8:27b-mq4-xt did the same. The ordinary AR battery and chain routes passed on the same lane.",
"file": "crates/hipfire-generate/src/qwen.rs",
"line": 2204,
"master_behavior": "A registry-paired DFlash model should load and complete battery and chain generation when explicitly tested with DFlash.",
"severity": "high"
},
{
"beta_behavior": "The LFM chain emitted Unicode replacement characters inside otherwise ordinary mathematical labels.",
"evidence": "hipx lfm2.5:1.2b chain turn 2 contains repeated `Distance\ufffd\ufffd` text. The harness marked the route pass, but decoded replacement characters are a coherence failure under the hardware-gate eyeball rule.",
"file": "crates/hipfire-runtime/src/prompt_frame.rs",
"line": 1498,
"master_behavior": "Decoded multi-turn output should remain valid text.",
"severity": "medium"
}
]
}Floor: hard=['hw_run_result=failure', "evidence verdict='fail'", 'kernel status != pass', 'policy_paths: registry/models.json,registry/v1.json,scripts/leanup-thresholds.txt'] soft=["coverage_gaps: ['No successful DFlash generation route evidenced speculative decode, context-exhaustion handling, or emitter-error rollback.', 'No fault-injection evidence covered transactional GPU allocation rollback or sticky 700/719 poisoning.', 'No multi-GPU EP/PP route exercised the new admission and DeviceMesh topology behavior.', 'Registry pull/rm sidecar lifecycle and policy changes were not exercised end to end.', 'The gfx12/hiptrx lane timed out or failed every fixture route after about 198 seconds, and its Redline run timed out on a subsequent load despite an internally passing first report.', 'No successful MQ4-V2 parity run covered the changed gfx12 dispatch-sensitive surface.']"] model_decision=block final=block |
Resolving a path-form model to its registry entry changed /health.model from the requested path to the tag. serve_harness's warm probe compares health.model to the launched path by realpath, so it never saw the serve as warm and killed/respawned it every 180 s (measured: two spawn attempts, zero turns). Keep the entry lookup for sidecars and policy; name the served model the way it was requested.
Measured on a 7900 XTX (serve_harness session_coding, greedy, thinking off, q8 KV): qwen3.8:27b-mq5 + its mq5 draft completes turns 1-2 (tau 3.6/3.5) then dies at turn 3, ctx ~4.9k, with spec_step hipMemCreate out of memory; every later turn is an empty response. The same session under AR passes all 8 turns (13.4k ctx, 38.1 -> 34.4 tok/s). 18.7 GB weights + ~5 GB fixed residency + 1.7 GB draft leaves no room for KV growth. Drop the sidecar from the 27B mq5/mq6 tiers (and qwen3.5:27b-mq6); mq4-tier and below keep theirs (measured 202 tok/s on qwen3.8:27b).
…ll declares hw-gate Fable seat on #686 (hardware probe): `hipfire rm qwen3.8:27b-mq4-pro` deleted qwen38-27b-dflash-mq4.hfq while qwen3.8:27b and qwen3.8:27b-mq4-xt — both declaring that sidecar — were still on disk; those siblings then ran AR under dflash_mode=auto or refused under `on`. Same shape for the mq3 draft (3 targets) and the 9B mq4 draft (3 targets). rm now removes a declared sidecar only when no OTHER registry entry declaring the same file still has its own target present in the models dir; otherwise it prints `keeping DFlash sidecar <file>: still declared by <tags>`. rm_command is a thin wrapper over rm_with_registry(&RegistryV1) so the rule is unit-testable without env or network. Tests: rm_keeps_shared_dflash_sidecar_while_sibling_target_present, rm_removes_dflash_sidecar_with_last_declaring_target, rm_without_dflash_declaration_leaves_draft_file_alone.
hw-gate Fable seat on #686: a daemon that went through one refused dflash_mode=on load held ~5.17 GB more VRAM with the next model resident than a clean daemon, compounding under serve's lazy retry. Cause: free_qwen35_bundle returns every buffer to the Gpu pool (free_tensor has no size cap, dispatch.rs:3261) and only unload_model drains it (lib.rs:3871-3872); a load that fails in finish_qwen35_load never reaches unload_model, so the whole target stayed pooled and the next load reused only the same-sized buckets. rollback_unfinished_qwen35 now mirrors unload_model: invalidate_graph_state + drain_pool after the frees. Covers all three callers: CASK eviction failure, dflash_mode=on draft failure, and the pre-existing mtp=on head failure (same leak, older than this PR).
… artifact hw-gate Fable seat on #686 (run 33889233321): resolve_tag matched any path by its final component, so `hipfire rm /elsewhere/qwen3.6-27b.mq4` — a different file sharing the basename — resolved to the qwen3.6:27b entry and deleted the installed model's triattn and DFlash sidecars while the model itself stayed; a same-basename foreign file loaded by path inherited the entry's sidecar and kv/max_seq policy. registry: resolve_tag drops the file_name() arm (a path is not a tag); bare entry.file names still resolve; new entry_for_file for exact matches. cli: registry_entry_for_path(paths, registry, input) — a path-form input resolves only when canonicalize(input) == canonicalize(models_dir/entry.file) for some entry, so the -xt symlink into ~/qcal still matches by target and a lookalike elsewhere never does. rm, run, bench, and serve (incl. the pre-warm thread) route through it. dflash_mode=on on a path with no entry and no explicit draft now fails closed: "DFlash draft required (dflash_mode=on) but <path> is not a registry-managed artifact; pass developer.dflash_draft or use the registry tag" (auto still serves it as a bare artifact) — closes Fable's earlier note that `on` ran AR silently there. The daemon only consumes CLI-lowered params and needs no change. cargo test -p hipfire-registry: 21 passed (2 new); -p hipfire-cli: 220 passed (4 new).
# Conflicts: # crates/hipfire-cli/map.md # crates/hipfire-runtime/map.md
…, pool drain on refused load, path-identity fail-closed) to staging Policy floor (registry/models.json, registry/v1.json) means the decide seat can never land this; merged by Main on the gate's hardware evidence after four rounds of Fable findings, all fixed and measured: - pool drain on refused load (d233d2a): 18,950 MB retained -> 281 MB, and the same daemon then serves qwen3.8+draft at -20 KB vs a clean daemon (hipx gfx1100) - shared-sidecar rm guard (def19e3): rm no longer deletes a sidecar another target still declares - path-basename identity (876cf28): registry_entry_for_path requires canonicalize equality, so a foreign /elsewhere/qwen3.6-27b.mq4 gets no sidecar and dflash_mode=on fails closed instead of running AR silently Enablement is unchanged: dflash_mode default stays off and the sidecar is resolved only under auto/on, so a paired draft on disk still never drafts until the user opts in. The only user-visible delta is pull size: +0.55 GB (9B) / +0.92-0.98 GB (27B). 249 tests pass in hipfire-registry + hipfire-cli at 397a366 (includes the beta merge with regenerated cli/runtime crate maps).
# Conflicts: # crates/hipfire-generate/map.md # crates/hipfire-loader/map.md
…ogy combinations before allocation) to staging
Fable's verdict was merge-staging; the only thing that stopped the seat was the recurring generated-map 409 (staging_merge_conflict hard floor on crates/hipfire-{generate,loader}/map.md). Merged by Main after merging beta in and regenerating those blocks with scripts/check-crate-maps.py. Run 33914516085: both lanes pass, and the decide phase ran the #683 repro on the 5x gfx1201 host rather than reading the diff:
- ornith-1.5:35b-a3b --tp 4 is refused at pre-warm with all four cards still at 32548 MB free, where master loads four ranks (~7.86 GB each) and only fails on the first request with 'EP arch mismatch'
- the one path the PR closes that master could reach — the batch-only EP route (batch_staging.rs:231-331) — segfaults inside libamdhip64 on master, 2/2 runs, so refusing it removes a crash rather than a serve path
- dense qwen3.5/3.6 TP through the same entry still loads two ranks and decodes coherently at 20.1 tok/s on the PR build
- LFM2 --continuous-batch-size 2 is byte-identical on both builds while master allocated an Lfm2DecodeBatchState nothing ever drove (batch.rs:180 returns false unconditionally); LFM2 --tp 2 refuses identically on both
Sol was needs-human on coverage, not on a regression: no Gemma4 or DeepSeek-V4/MiniMax fixture exists on this host, and those arms are arch/config-only code before device init. Recorded as unproven in the decision artifact, not as a claim. 70 tests pass in hipfire-loader + hipfire-generate at 5dbe4a9.
# Conflicts: # crates/hipfire-runtime/map.md
… axis) to staging Fable's verdict was merge-staging; blocked only by the recurring generated-map 409 (staging_merge_conflict on crates/hipfire-runtime/map.md). Merged by Main after merging beta in and regenerating that block. Run 33914554146: both lanes pass, and the decide phase measured the claims on the 5x gfx1201 host: - a 4x gfx1201 EP load of ornith-1.5:35b-a3b-mq4r on the PR daemon logged 'EP load: tp=4 arch=qwen35' after init_ep, completed on 4 ranks, then unloaded/reloaded/unloaded with rank-0 vram_free_mb 32548/25512/32400/25510/32398 -- the identical five numbers the master daemon produced from the same stdin, so the Ep mesh axis is a zero-runtime-delta rename as claimed - Gpus::init_ep is field-for-field init_tp except mesh: DeviceMesh::rect(Ep, n), and the mesh field is read only by constructors and tests Sol's needs-human was coverage, not regression: no AWQ/PARO Llama artifact was cycled with free-VRAM diagnostics, and the Qwen2 free_all path has no fixture on either lane. Both recorded as unproven rather than as claims. 21 tests pass in hipfire-runtime + hipfire-loader at a0aa126.
# Conflicts: # crates/hipfire-daemon/map.md # crates/hipfire-loader/map.md # crates/hipfire-loader/src/lib.rs
…fy and admit before teardown) to staging Fork PR (fivetide), so it is staged in this repo as staging/pr-682: PR head 95e0e65 merged with beta at a9f4ca8. That merge had a REAL semantic conflict in crates/hipfire-loader/src/lib.rs between #687's ep_admission and #682's admit_source; resolved by keeping #682's classify-once split (load_model_ep_admitted) and #687's shared ep_unsupported_arch_message(id) as the backstop arm so the message cannot drift. The resolution is proven on hardware (hiptrx gfx1201, daemon md5 99457c71e4e400793d3efede7278ac3f), not just compiled: - battery on the canonical qwen3.8-27b.mq4-xt (mq4v2) trunk: 5/5 turns, attractor 0, empty 0, runaway 0, recall satisfied on every turn, avg decode 29.5 tok/s - LFM2 (arch_id=11) --tp 2 refuses with exactly the shared constructor's text, 'EP not supported for arch_id=11 (expected 5|6 for Qwen3.5, 9 for DeepSeek V4 or 10 for MiniMax)', at [validation retryable=false rolled_back=false] with the serve process still alive - ornith-1.5-35b-a3b (arch_id=6) --tp 2 refuses through #687's rule at 32548 MB free / 32624 MB total, i.e. #682's restructure did NOT bypass the refusal that #687 added - the precise risk of that conflict - 48 tests pass in hipfire-loader + hipfire-daemon Fable's investigation (run 33921475093) closed every one of Sol's coverage gaps on hardware: six refusal classes all fire before teardown with the prior model still generating, while the BASE daemon answers 'no model loaded' after four of the six. Two floors, both human calls, both made: scripts/leanup-thresholds.txt is a policy path, and the ratchet raise daemon_lines <= 4155 -> 4176 is accepted - the logic landed in hipfire_loader::admission and what grew in the daemon is the call site plus emit_uncorrelated_error boilerplate that must live at the protocol boundary. If that ceiling is hit again, collapse the repeated emit blocks into a local helper first. Unproven and recorded as such: DS4 (arch 9) and MiniMax (arch 10) EP admission, and Qwen3.5-VL / LFM2-VL tower classification - no such artifact exists on either lane.
Each of the three EP loaders (ds4, minimax, qwen35) now checks gpus.mesh.size_of(Ep) == n right after the existing devices-vs-tp check, so a future constructor that records the wrong mesh axis fails loudly at load instead of loading silently mislabeled. Proof: cargo check -p hipfire-loader clean (no new warnings); cargo test -p hipfire-loader: 34 passed, 0 failed.
|
Correcting the finding on this PR, because the record as it stands points at the wrong defect. I pulled the serve log from the run that produced it (gate run 33907487315,
So the "primer replay systematically misses the most recent assistant body" reading does not survive its own evidence: the ordering shows the That makes the real work here:
The AR half is unaffected and was already proven on both lanes. An agent is on (1) and (2) now; I'll post what it finds. Config for the record: qwen3.6-27b, DFlash on, ChatML, thinking on with non-empty |
…fills
The DFlash-arm finding on this PR pointed at primer replay; the serve log does
not support that. Gate run 33907487315
(`fable-evidence/dflash-chain-qwen36-serve.log`) shows full-prefix reuse on
every hit -- `prior_len=1528 rendered_len=1582 lcp=1528` then
`prior_len=2266 rendered_len=2290 lcp=2266`, both `no reset`, suffixes 54 and
24 -- so `lcp == prior_len` and there is no store/replay asymmetry. The
`hit=false` lookups that follow are the rollback clearing `asst_turn_cache`
(common.rs:277), i.e. fail-closed behaviour working, not a second defect.
What actually happens is a sticky `HipError(719)` (`hipErrorLaunchFailure`) in
the `spec.prefill` miss path during mid-window `spec_prefix_realign`
(qwen.rs:3591, formatted at :3625). The op label `hipMemcpy H2D offset`
(hip-bridge ffi.rs:1063) points at `draft_seed_backfill`'s ring upload
(dflash.rs:2169) on a windowed run. It is not an overrun: every size check on
that path panics, and the log carries a HipError rather than a panic. The chunk
D2H downloads that precede the backfill would have reported an already-dead
context first, so the context was alive through the seed -- leaving either the
backfill's H2D destination pointer or, more likely, a backfill kernel failing
with the next H2D reporting it stickily. Localising that needs a device-sync
bisection across the backfill chunk ops on hardware, recorded rather than
guessed.
Independent of that root cause, the retry behaviour is wrong on its own terms.
719 kills the context, so `reset_recurrent` cannot recover it -- yet the run
shows three further attempts, each producing another 719 from a memset that
never had a chance. This commit makes the first sticky fault the last:
- reset_core.rs: STICKY set {700, 719}, a first-wins `GpuPoison` latch, and
`note_hip_error` / `note_hip_result` / `gpu_poison` / `clear_gpu_poison`,
with a lifecycle unit test
- qwen35 weights.rs: `DeltaNetState::reset` latches sticky memset failures
- qwen35 dflash_spec.rs: the prefill seed/backfill/logits error sites latch on
the typed `HipError`, no string sniffing
- daemon main.rs: a pre-generate poison check fails fast -- class `gpu`,
non-retryable, "process restart required", mirroring the `batch_poisoned`
shape already in that file
The latch never auto-clears: a model unload/reload does not reset the primary
HIP context, so clearing would just burn one prefill per reload. The CLI
gateway already refuses these errors (validation, non-retryable, max one
retry), so the repeats seen in the log came from above it; they now stop at the
daemon.
Verified: `cargo check` clean on hipfire-runtime, hipfire-arch-qwen35,
hipfire-daemon. `cargo test -p hipfire-runtime --lib` 599 passed;
`-p hipfire-arch-qwen35 --lib` 189 passed; reset_core 10 passed including
`sticky_poison_latch_lifecycle`. Crate maps regenerated for the three crates.
Not fixed here, and recorded rather than invented: the DFlash bake omits the
ChatML newline trailer AR appends (qwen.rs:3895 vs ar.rs:4122), proven
LCP-harmless by test; and the `reasoning_content` guard forces a plain-render
fallback on thinking sessions with echoed reasoning, whose slack (+3/+4) does
not match the hardware signature (+2).
…iffs Two policy gaps this ladder exposed. 1. The gate never ran DFlash. #686 (draft sidecars), #691 (draft ctor rollback), #692 (primer replay) and #702 (dedicated verify kernels) all went through with every lane green while speculation never once executed. #692's DFlash-arm defect -- primer replay systematically missing the most recent assistant body -- was found only because a seat thought to drive twenty turns by hand. That is not a gate. The load bucket now runs `battery-dflash` and the serve bucket `chain-dflash`: the same prompts with `--dflash on` and an explicit `--draft`. `on` rather than `auto` because `auto` silently falls back to AR when the draft is missing, and a route that can pass without speculating proves nothing. The draft is named explicitly because the canonical xt trunk is a symlink out of the models dir, so the daemon's filename auto-match finds nothing and would run AR. `dflash_draft` is a candidate LIST because the lanes hold different drafts: hiptrx has qwen36-27b-dflash-mq4.hfq and no qwen38, hipx has qwen38-27b-dflash-mq4.hfq and no qwen36. A lane speculates with the first candidate it holds; a lane holding none records `skip`. `skip` is neither pass nor fail. The aggregation was `all(status == "pass")`, which would have counted a skip as a fixture failure -- a false negative on evidence the host never had -- while treating it as a pass would claim coverage that did not happen. Skips are recorded and reported, and a genuine failure alongside a skip still fails. Coverage is asymmetric until both hosts hold both drafts. Pulling qwen38-27b-dflash-mq4.hfq to hiptrx and qwen36-27b-dflash-mq4.hfq to hipx (0.92 GB each) makes it symmetric; that is a disk decision, so the evidence says `skip` rather than silently pulling. 2. Sol refused hardware for any diff touching a filesystem path, which caught #689 for adding `--prompt-file` to `hipfire bench` and cost that rung a lane until `hw-run` overrode it. hipfire is a CLI inference engine: users name models, prompts, drafts and sidecars at invocation, and the gate's own harness passes exactly those flags. sol.md now separates whose path it is -- an explicit argument is ordinary product work; credentials, dotfiles, SSH or cloud config, /proc or /sys beyond device enumeration, assembled traversal, or a read whose result leaves the process still warrant refusal. Tests: eight new cases in scripts/hw-gate/tests/test_run.py covering flag translation (battery-dflash -> `--mode battery --dflash on --draft ...`), plain battery never receiving a draft, per-lane draft selection, skip-not-fail with the harness never invoked, chain-dflash keeping its own prompts, the skip-vs-genuine-failure aggregation, and a manifest assertion that the buckets actually carry the routes. 113/113 hw-gate tests pass.
bind(mesh): assert Ep topology on every EP load path
…Reduce Remove the v1.1 stub Gpus::init_vram_weighted (zero callers) and the never-constructed CollectiveHint::AllReduce variant. Scrub matching docs and crate map entries.
overhaul(s6a): delete dead runtime multi_gpu/mesh symbols
There was a problem hiding this comment.
hw-gate sol verdict needs-human: All executed fixtures passed on gfx1201 and gfx1100, decoded text is coherent, qwen3.8 AR chains demonstrate the intended cache extension, and qwen3.6 controls show no observed template regression. Redline also reports stable captures and exact HIP/blob/AQL state parity. However, the branch additionally changes DFlash replay, daemon-wide sticky GPU-fault state, plain-Llama MQ-V2 admission, and benchmark CLI behavior without direct evidence for those paths. The central AR fix is supported, but the author's AR-and-DFlash equivalence claim is not fully exercised, so the state-machine and speculative-decode coverage gaps require human disposition.
# Conflicts: # crates/hipfire-daemon/map.md # crates/hipfire-generate/map.md # crates/hipfire-runtime/map.md
…lay) to staging Run 33934632513: both lanes pass at 370592d (hiptrx gfx1201 + hipx gfx1100, verdict pass, zero non-pass modes); Sol needs-human with zero regressions; Fable unavailable (credits), so the staging merge is Main's, not the seat's. The DFlash-arm finding that held this PR was misdiagnosed, and the log settles it. Gate run 33907487315 (fable-evidence/dflash-chain-qwen36-serve.log) shows prior_len=1528 rendered_len=1582 lcp=1528, then prior_len=2266 rendered_len=2290 lcp=2266 -- full-prefix reuse both times, 'no reset', suffixes 54 and 24. So lcp == prior_len and there is no store/replay asymmetry; the hit=false lookups that follow are the rollback clearing asst_turn_cache (common.rs:277), i.e. fail-closed behaviour working. What actually happened is a sticky HipError(719) in the spec.prefill miss path during mid-window spec_prefix_realign (qwen.rs:3591), whose 'hipMemcpy H2D offset' label points at draft_seed_backfill's ring upload (dflash.rs:2169) on a windowed run. Not an overrun: every size check there panics, and the log carries a HipError. Root-causing the fault itself needs a device-sync bisection across the backfill chunk ops on hardware; that is recorded, not guessed. Independent of the cause, the retry behaviour was wrong: 719 kills the context, so reset_recurrent cannot recover it, yet three further attempts each produced another 719. 370592d latches the first sticky fault (700/719 first-wins GpuPoison in reset_core.rs, latching at the qwen35 reset/prefill error sites) and fails fast at the daemon with class gpu, non-retryable, 'process restart required'. The latch never auto-clears, because a model reload does not reset the primary HIP context. 600 tests pass in hipfire-runtime and 189 in hipfire-arch-qwen35 at fa30b3a, including sticky_poison_latch_lifecycle. Generated crate maps for daemon, generate and runtime regenerated for the beta merge.
|
Merged to Run 33934632513: both lanes pass at The finding that held this PR was misdiagnosed, and its own log settles it — full detail in the comment above. Short version: What landed on top is the fix that is correct regardless of the 719's root cause: the first sticky fault is now the last. 600 tests in Still open and recorded rather than invented: the 719 itself needs a device-sync bisection across the backfill chunk ops, plus a pointer-lifetime trace on |
There was a problem hiding this comment.
hw-gate sol verdict block: Block: the central Qwen3.8 claim is directly disproven by the gfx1100 chain, where cached remained zero after turn one. Both Qwen DFlash routes also failed with no output, and the mandatory kernel lane is globally failed because hiptrx timed out during load. Although the gfx1100 Redline report showed stable sequences and exact HIP/AQL/PM4 parity, substantial rollback, topology, policy, DFlash, and gfx12 coverage remains absent.
A rung that merges to `beta` stays OPEN by design -- promoting beta -> master is the maintainer's call -- so `pull_request.merged` is false and the merged-PR guard from warpfront#712 does not apply. Every later touch of that branch then re-runs the full gate on work that is already staged: warpfront#692 and warpfront#723 both re-ran within minutes of their staging merges, taking the runner from live rungs, and the same pattern accounted for several of the runs cancelled by hand tonight. `select` now asks whether the head is an ancestor of the staging branch. If it is, the evidence exists and the hardware has nothing to add, so `run_hw` is false: the lanes, Sol's verdict and the decide phase all skip, and the recorded decision still governs the status. The PR is not touched and no label changes. Deliberately an ancestor test rather than a SHA equality test: a rung merges as a staging commit whose parent is the head, so equality would never match, and an ancestor test also covers a rung whose branch was merged and then pushed again without new work. `workflow_dispatch` is unaffected, so a manual re-gate of a staged rung still runs -- that is the escape hatch for re-measuring after a gate fix, which is exactly what warpfront#702 needed tonight. 132/132 hw-gate tests pass; the workflow parses and the select job's step list and `run_hw` expression were checked.
Summary
Every conversational turn after the first re-prefilled the whole conversation on Qwen3.8 (measured
lcp=26againstprior_len=118, 2–6.5 s of re-prefill per turn on the coding session) because the jinja prompt-cache replay prepends the live turn's generation primer to every cached assistant body on the assumption the template renders history turns bare. Qwen3.5's template does; Qwen3.8's template re-emits<think>\n\n</think>\n\non history assistant turns when thinking is off, so the spliced render carried the empty-think block twice, diverged 4 tokens into the first answer, and the LCP never hit.Token-level proof (
HIPFIRE_QWEN_CACHE_TRACE=1, qwen3.8:27b.mq4, two-turn probe):The fix probes the template once (
template_emits_history_primer: render a one-exchange history with a sentinel answer, check whether the primer sits between the assistant opener and the sentinel) and both lookups (AR inar.rs, DFlash inqwen.rs) prepend the primer only when the template does not re-emit it. Qwen3.5 behaviour is unchanged; Qwen3.8 gets the extension it was getting only in error paths.Which crate(s) does this touch?
crates/hipfire-runtime(prompt_frame.rs: probe + test),crates/hipfire-generate(ar.rs,qwen.rs: two lookup sites)Evidence (RX 7900 XTX,
4bc750e73, qwen3.8:27b.mq4/.mq5, greedy, thinking off)Two-turn probe:
lcp=26 → lcp=118 = prior_len,cached_tokens=118, and the fixed render is shorter (144 vs 148 — the double primer is gone).Full 8-turn coding session (
session_coding.json, serve_harness, q8 KV), before → after (AR arm shown; the fixed build's two arms are byte-identical turn by turn):22–30× cheaper per-turn prefill; all 8 turns pass with recall 3/3, same gen counts as before (greedy ⇒ output-preserving).
Not caused by this bug, measured in the same sessions:
mq5 + drafton a 24 GB card OOMs at ~5k ctx regardless (#686 dropped the pairing on those tiers and has the details).Test plan
cargo test -p hipfire-runtime --lib prompt_frame— 53 pass incl. newhistory_primer_probe_distinguishes_qwen35_and_qwen38_templates(bare template → prepend, re-emitting template → don't, empty primer → don't)cargo test -p hipfire-generate— 221 pass