Skip to content

fix(prompt-cache): template-aware assistant-primer splice — Qwen3.8 prefix cache hits again - #692

Open
Kaden-Schutt wants to merge 69 commits into
masterfrom
fix/prefix-cache-primer
Open

fix(prompt-cache): template-aware assistant-primer splice — Qwen3.8 prefix cache hits again#692
Kaden-Schutt wants to merge 69 commits into
masterfrom
fix/prefix-cache-primer

Conversation

@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Summary

Every conversational turn after the first re-prefilled the whole conversation on Qwen3.8 (measured lcp=26 against prior_len=118, 2–6.5 s of re-prefill per turn on the coding session) because the jinja prompt-cache replay prepends the live turn's generation primer to every cached assistant body on the assumption the template renders history turns bare. Qwen3.5's template does; Qwen3.8's template re-emits <think>\n\n</think>\n\n on history assistant turns when thinking is off, so the spliced render carried the empty-think block twice, diverged 4 tokens into the first answer, and the LCP never hit.

Token-level proof (HIPFIRE_QWEN_CACHE_TRACE=1, qwen3.8:27b.mq4, two-turn probe):

common[20..26]  "assistant\n<think>\n\n</think>\n\n"
prior_past[26]  "```python\ndef is_prime(n):…"
rend_past[26]   "<think>\n\n</think>\n\n```python…"   ← primer doubled
lcp=26 prior_len=118 rendered_len=148

The fix probes the template once (template_emits_history_primer: render a one-exchange history with a sentinel answer, check whether the primer sits between the assistant opener and the sentinel) and both lookups (AR in ar.rs, DFlash in qwen.rs) prepend the primer only when the template does not re-emit it. Qwen3.5 behaviour is unchanged; Qwen3.8 gets the extension it was getting only in error paths.

Which crate(s) does this touch?

  • crates/hipfire-runtime (prompt_frame.rs: probe + test), crates/hipfire-generate (ar.rs, qwen.rs: two lookup sites)

Evidence (RX 7900 XTX, 4bc750e73, qwen3.8:27b.mq4/.mq5, greedy, thinking off)

Two-turn probe: lcp=26 → lcp=118 = prior_len, cached_tokens=118, and the fixed render is shorter (144 vs 148 — the double primer is gone).

Full 8-turn coding session (session_coding.json, serve_harness, q8 KV), before → after (AR arm shown; the fixed build's two arms are byte-identical turn by turn):

turn ctx cached before prefill before cached after prefill after
t2 3564 0 4654 ms 3514 208 ms
t3 5885 0 4529 ms 5843 189 ms
t4 8603 0 6411 ms 8563 213 ms
t8 15490 0 2083 ms 15456 258 ms

22–30× cheaper per-turn prefill; all 8 turns pass with recall 3/3, same gen counts as before (greedy ⇒ output-preserving).

Not caused by this bug, measured in the same sessions: mq5 + draft on a 24 GB card OOMs at ~5k ctx regardless (#686 dropped the pairing on those tiers and has the details).

Test plan

  • cargo test -p hipfire-runtime --lib prompt_frame — 53 pass incl. new history_primer_probe_distinguishes_qwen35_and_qwen38_templates (bare template → prepend, re-emitting template → don't, empty primer → don't)
  • cargo test -p hipfire-generate — 221 pass
  • Hardware: probe + full session above, both AR and DFlash arms

Bjoern Agent and others added 26 commits September 2, 2026 15:40
…down (G2)

Classify a retained source once and decide one effective topology before
any destructive side effect, so a refused load leaves the prior model
usable (issue #666 G2).

- admission::admit_source (read-only): open the source, classify arch_id +
  vision (tower-tensor decides; contract 179a20d), decide the effective
  topology (single/pp/ep), and refuse no-carrier / ambiguous / VMM
  allowlist / VMM+pp / carrier-pp / EP-arch / lfm2 vision-no-config /
  DFlash lm-head quant — all before any GPU/VMM/teardown work.
- Carrier::admit_topology (default + qwen35/cohere2moe/maple/gemma4
  overrides) mirrors each carrier's load-time pp refusal.
- Split the load entries: path wrappers classify-then-load via
  load_admitted_with_gemma4_drafter / load_model_ep_admitted, which consume
  the retained SourceAdmission (no re-open, no re-classify).
- Reorder the daemon load handler: daemon topology refusals + admission run
  BEFORE prior-model teardown; on refusal the prior model stays loaded.

RATCHET-RAISE: daemon_lines 4155 -> 4176, traded for the ~21-line G2
source-aware admission block inserted before prior-model teardown.

Verification: workspace build clean; full workspace test suite passes;
qwen3.6:27b (fresh-daemon VMM commit) + qwen3.6-35b-a3b load and generate
coherently; a bad-path load and a vmm+pp>1 load both refuse at admission
with the prior model still generating afterward.
admit_source refused kv_backend=vmm for every EP load, so a DeepSeek V4
EP + vmm load that master serves (load_model_ep_with_kv_mode arch 9 arm)
was refused at admission. Gate the refusal on matches!(arch_id, 5|6|10),
mirroring master's per-arch dispatch, and keep the DS4 (9) arm vmm-capable.

Also correct the no-reopen claim on the EP path: the per-arch EP loaders
re-open path per rank, so the retained SourceAdmission.source is dropped
there rather than consumed (single/pp route is unchanged).
Regenerate the hipfire-loader generated map block after the per-arch VMM
refusal fix (admission.rs 274 -> 298 lines, 5 -> 6 tests; lib.rs 4879 ->
4881 lines). Keeps scripts/check-crate-maps.py --check green in CI.
The default ~24-token prompt reports prefill_tok_s ~= 363 tok/s while a
4.4k-token prompt on the same binary reports 886: the short-prompt number
is launch overhead, not prefill, and the JSON gave no way to tell.

- Add --prompt-file <PATH>: prompt read verbatim (raw bytes, no trim),
  mutually exclusive with positional PROMPT words.
- Standard-bench JSON gains top-level prompt_tokens (u64, as the daemon
  reports it in done.prompt_tokens), prompt_md5 (hex md5 of the exact
  prompt bytes), prompt_chars, and warnings[]; warn when prompt_tokens
  < 256 that prefill_tok_s measures launch overhead.
- Same three values printed on the stderr banner next to model:/arch:.
- No existing field renamed, nulled, or removed; default prompt bytes
  unchanged (audit 2026-09-02 fix slice: bench-prompt-evidence).
The daemon's done event has no prompt_tokens key; the prompt is
prefill_tokens (rows prefilled) plus cached_tokens (prompt-cache prefix).
Measured on a 7900 XTX the JSON reported prompt_tokens: null and never
warned on the 24-token default prompt.
…ropped

hw-gate Fable seat on #689: the flag table lost its `--reasoning-on` row (base AGENTS.md:359) while the flag still exists in `hipfire bench --help`. Additive row only, as the PR body says.
… JSON, short-prompt prefill warning) to staging
llama::is_batchable_la admitted MQ4G256V2/MQ6/5/3/2G256V2 for WMMA
prefill only on gfx1200/gfx1201 while qwen35::is_batchable_la admitted
them on gfx11+gfx12 behind HIPFIRE_MQV2_GFX11_WMMA, so plain Llama/Qwen3
dense qt=44 models prefetched per-token on gfx1100/1151 while Qwen3.5/3.8
took WMMA — despite both doc-comments claiming an exact match (audit
2026-09-02 Broken 1).

Move the dtype set + arch set + kill-switch helper into
llama::mqv2_wmma_batchable / llama::mqv2_gfx11_wmma_enabled_from_env in
hipfire-runtime and delegate from both callers, so the lockstep is
structural. MQ4CG256 (qt=45) stays gfx12-only in both by intent.

Tests: rename the two gfx12-only llama admit tests to gfx11+gfx12
expectations, repoint qwen35 env-escape test at the shared helper, and
add mqv2_admit_llama_qwen35_lockstep asserting both gates agree over
the MQ-V2 dtypes x {gfx1100, gfx1151, gfx1201, gfx1030, gfx1010}.
…ections

mq4v2_gemm_parity's Gaussian weights give both halves near-identical
headers, so a wrong half-select hides in quantization noise despite the
header comment promising a systematic blow-up (audit 2026-09-02
Missing 1). Add arm 2 using the disjoint-halves construction from
mq4v2_residual_parity (half0 [-1,1], half1 [96,160]) over the same
batch-size sweep: v2 output must match the f32 reference within 5%
rel-RMS, and the swapped-headers negative control (as in
mq4v2_moe_parity) must DISAGREE. Keep the Gaussian v1-vs-v2 arm and fix
the header comment to state what each arm can and cannot detect. Add a
host-side test proving the fixture discriminates with no GPU.

Docs (audit Would-change 1-2): spec section 9 now records MoE as
production-wired for qt=44, the XBATCH single-row path as ported, and
the gfx11 kt+=2 / residual kt++ stepping; residual_mmq.hip loses its
stale Experimental tag; the gfx12 QKV kernel loses its
HYPOTHESIS/scaffold wording for the validated C-map statement.
DflashScratch::new_with_mq, new_windowed, DflashWeights::load, and
build_generic_dflash_speculator could '?' out mid-construction, leaking
earlier alloc_tensor results (GpuTensor/DeviceBuffer have no Drop).

Record each allocation in a slot vec (gt!/wt!/at!) taken once into the
final owner; the error arm frees completed layers plus staged slots.
new_windowed frees the base scratch via alloc_or_free!; the generic
builder frees weights (and scratch) on later failures. Success path is
byte-identical. Mirrors load_dflash_state's or_free! (audit-Dflash
Broken 3).
The make_spec_emitter Err exit ran after a successful spec.prefill
without production_fail_closed_rollback_live, unlike every other
post-prefill error exit. The target KV/DeltaNet/drafter hidden had
advanced and host seq_pos/conversation_tokens were cleared, so the
next turn could LCP against a dirty GPU. Route it through the same
rollback + fail-closed error (audit-Dflash Broken 4).
…length

generate_dflash fell back to AR only when prompt + max_tokens >
ctx_capacity, but generate_spec hard-errors when prompt + max_tokens
+ block_size > ctx — requests in that band got gen_start followed by
an error instead of the promised AR fallback. Both sites now share
spec_ctx_request_fits (prompt + max + block <= cap).

The mid-loop position + block_size >= ctx_capacity break now sets
SpecRun::ctx_exhausted, which the qwen (v2 + legacy) and dense
epilogues OR into the length decision: finish_reason=length with no
cache store instead of a silent early stop (audit-Dflash Broken 5).
qwen_dflash_semantic_terminal_tests.rs carries historical rustfmt debt; CI
enforces rustfmt on changed files, so adding one test there forces a
6k-line reformat. The new contract lives in qwen_dflash_ctx_exhausted_tests.rs
and the debt file is restored to master byte-for-byte.
…efill_chunk has no V2 arms

hw-gate Fable seat on #690 (run 33895641944), source trace verified: the shared MQ-V2 admit rule made llama::is_batchable_la admit plain Llama/Qwen3-dense qt44/47-50 artifacts to WMMA prefill on gfx11 and gfx12, but llama.rs::forward_prefill_chunk's per-layer matchers (qkv_is_mq ~:2570, wo_is_mq ~:3025, ffn_is_mq ~:3117, w_down_is_mq ~:3248) list only MQ4G256|MQ6G256|MQ3G256|MFP4G32 — an admitted V2 model skips the FWHT rotate and runs the V1 hfq4g256 launchers on V2 blobs: silently incoherent prefill. master's pre-existing mq4_v2_gfx12 arm had the same hole on gfx12; no gfx12 Llama-V2 artifact has ever tripped it.

llama::is_batchable_la now refuses every *G256V2 dtype and MQ4CG256 on every arch, with the reason at the site. qwen35::is_batchable_la keeps the shared mqv2_wmma_batchable rule (its chunk path has the V2 arms; gfx11 kill-switch intact). llama_spec::batched_verify_eligible routes all seven weights through is_batchable_la, so it is covered without an edit. Lockstep test now asserts the true contract: agreement on every non-V2 dtype across 5 arches; for V2, qwen35 admits on gfx11/gfx12 and llama refuses everywhere. Spec §9 row and crate maps corrected.

hipfire-runtime is_batchable_la: 7 passed; qwen35 is_batchable + lockstep: 9 passed.
…id-ladder failure frees all of them

hw-gate Fable seat on #691 (run 33900101473): alloc_or_free! freed the base scratch on failure but the four already-allocated tensors were still locals with no Drop — a failure on the 2nd..5th alloc leaked k_full / v_full / k_cat / v_cat. They are now assigned into s as each succeeds, so the error arm's s.free_gpu covers the whole ladder. Same shape as new_with_mq's at!/live list.
…llama and qwen35; discriminating GEMM parity; spec §9) to staging
…e does not re-emit it

Both jinja cache lookups (ar.rs, qwen.rs dflash) prepended the live turn's
generation primer to every cached assistant body, on the assumption that
the template renders history assistant turns bare. Qwen3.5's does;
Qwen3.8's re-emits <think>\n\n</think>\n\n on history turns when thinking
is off, so the spliced render carried the primer twice and the LCP died at
the first assistant turn of every conversation: every turn re-prefilled
from the last checkpoint (2-4k tokens/turn on the coding session; the
'2 minutes to first token' complaint).

Measured on a 7900 XTX, qwen3.8-27b.mq4, AR, HIPFIRE_QWEN_CACHE_TRACE=1:
  prior_past[26..] = ```python...   rend_past[26..] = <think>\n\n</think>\n\n```python...
  lcp=26 prior_len=118 rendered_len=148

template_emits_history_primer probes the template with a one-exchange
history and decides per template; both lookups use it.
@Kaden-Schutt
Kaden-Schutt force-pushed the fix/prefix-cache-primer branch from 4bc750e to 5b130d1 Compare September 4, 2026 17:58
@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate sol prelim

summary: This head is a large stacked change, not only the advertised prompt-cache fix: it makes assistant-primer replay template-aware, adds registry-managed DFlash sidecars and prompt-file benchmark evidence, introduces source-aware pre-teardown admission and EP mesh labeling, changes DFlash allocation/rollback/context-exhaustion behavior, adds sticky GPU-fault handling, changes LFM batching eligibility and Llama MQ-V2 prefill admission, and updates MQ4-V2 documentation/tests. The two kernel files contain comment-only changes, but runtime dispatch and load/serve state machines materially change.

run_hardware: true
run_hardware_reasons: The Rust, HIP, registry, and documentation changes are ordinary hipfire product work with no obfuscated payloads or unexplained unsafe code.; The new filesystem reads are user-requested model and --prompt-file paths or registry-declared sidecars under the model directory; they do not read credentials or unrelated host state.; scripts/registry_gen.py only extends existing Hugging Face registry generation to annotate DFlash sidecars; it is not invoked by the runtime hardware routes.; Cargo.toml adds the ordinary md5 crate solely to stamp exact benchmark prompt bytes.; Real hardware is required because the change reaches gfx12 MQ-V2 dispatch eligibility, model-source admission, DFlash GPU allocation/rollback, multi-turn recurrent prompt-cache state, and serve terminal behavior.

routes:

mode tag source why
battery qwen3.6:27b bucket bucket kernel,load,serve
chain qwen3.6:27b bucket bucket kernel,load,serve
battery-dflash qwen3.6:27b bucket bucket kernel,load,serve
chain-dflash qwen3.6:27b bucket bucket kernel,load,serve
battery ornith-1.5:35b-a3b-mq4r bucket bucket kernel,load,serve
chain ornith-1.5:35b-a3b-mq4r bucket bucket kernel,load,serve
battery-dflash ornith-1.5:35b-a3b-mq4r bucket bucket kernel,load,serve
chain-dflash ornith-1.5:35b-a3b-mq4r bucket bucket kernel,load,serve
battery lfm2.5:1.2b bucket bucket kernel,load,serve
chain lfm2.5:1.2b bucket bucket kernel,load,serve
battery-dflash lfm2.5:1.2b bucket bucket kernel,load,serve
chain-dflash lfm2.5:1.2b bucket bucket kernel,load,serve
battery qwen3.8:27b-mq4-xt bucket bucket kernel,load,serve
chain qwen3.8:27b-mq4-xt bucket bucket kernel,load,serve
battery-dflash qwen3.8:27b-mq4-xt bucket bucket kernel,load,serve
chain-dflash qwen3.8:27b-mq4-xt bucket bucket kernel,load,serve

unavailable_routes:

(none)

claim_assessment: The author claims Qwen3.8 thinking-off history templates already emit the empty-think primer, so suppressing a second primer restores full multi-turn LCP/cache hits without changing generated bytes, while Qwen3.5-style behavior remains unchanged. Proof requires a real multi-turn Qwen3.8 chain showing coherent byte-stable outputs and cached-prefix growth after turn one, plus a Qwen3.5/3.6-family control showing no cache regression. The supplied PR-body timings and test counts are claims, not gate evidence.

questions_for_author:

@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate evidence — 2 lane(s) — verdict fail

lane hiptrx (unknown)

hw-gate evidence

field value
base 3bd2914bac09bb6d48b03cf5b0155473ff1bc922
head fa30b3ae247a64f9acc44ddeeca28678c0ba9bf0
buckets kernel,load,serve
host gfx unknown
host rocm 7.15.26333-0000000
device 1
runner hiptrx
daemon_md5 2f6b10605a61dcb45c86d8769ef3e06c
hipfire_md5 594eedc165bc975b7bc6d99a0217c258
build_seconds 337.46435594558716
verdict fail
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: harness exit 1; chain-dflash: harness exit 1

battery — exit 1 seconds 199.3 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 186 90 3.1 18.2 False False False True
battery 1 stop 30 0 86 78 502.2 36.0 False False False True
battery 2 stop 31 0 19 16 520.7 36.2 False False False True
battery 3 stop 47 0 31 8 561.8 36.1 False False False True
battery 4 stop 47 0 6 2 564.2 36.2 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating_chars(s: str) -> str:
    """
    Returns the longest substring of a string without repeating characters.
    
    Args:
        s: Input string
        
    Returns:
        The longest substring without repeating characters
    """
    if not s:
        return ""
    
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        
        char_index[char] = end
        
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.6:27b battery turn 1
The sky appears blue during the day because sunlight interacts with Earth's atmosphere, scattering shorter blue wavelengths more widely than longer red wavelengths. At sunset, sunlight travels through a much thicker layer of atmosphere to reach the observer, which filters out most of the blue light and allows the longer red and orange wavelengths to dominate. This shift in scattered light composition is why the sky transitions from blue to red as the sun dips below the horizon.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
Answer: 43

chain — exit 1 seconds 198.9 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 44 0 124 54 512.3 35.8 False False False True
chain 1 stop 224 169 204 109 413.5 35.7 False False False True
chain 2 stop 454 429 76 62 299.5 35.7 False False False True
chain 3 stop 564 531 87 71 305.5 35.6 False False False True
chain 4 stop 683 652 65 46 359.4 35.6 False False False True
qwen3.6:27b chain turn 0
```python
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list without using sorted()."""
    result = []
    i, j = 0, 0
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1
    # Append remaining elements
    result.extend(a[i:])
    result.extend(b[j:])
    return result
```
qwen3.6:27b chain turn 1
To find the total distance traveled, we calculate the distance for each segment of the trip using the formula:

**Distance = Speed × Time**

### Step 1: Calculate the distance for the first segment
*   **Speed:** 60 mph
*   **Time:** 2.5 hours
*   **Distance:** $60 \times 2.5 = 150$ miles

### Step 2: Calculate the distance for the second segment
*   **Speed:** 40 mph
*   **Time:** 1.5 hours
*   **Distance:** $40 \times 1.5 = 60$ miles

### Step 3: Calculate the total distance
Add the distances from both segments together:
$$150 \text{ miles} + 60 \text{ miles} = 210 \text{ miles}$$

The train traveled a total of **210** miles.
qwen3.6:27b chain turn 2
The primary cause of the seasons is the tilt of Earth’s rotational axis, which is inclined approximately 23.5 degrees relative to its orbital plane around the Sun. As Earth revolves, this constant axial tilt causes different hemispheres to receive varying intensities and durations of direct sunlight throughout the year. The resulting change in solar heating creates the distinct seasonal patterns we experience.
qwen3.6:27b chain turn 3
For forty years, Elias had watched the waves beat against the jagged cliffs with predictable monotony. One gray morning, however, he noticed a glint of unnatural color tangled among the dark rocks near the waterline. As he carefully cleared away the seaweed, he uncovered a brass time capsule, perfectly preserved from a ship lost decades ago. The rusted seal on the lid promised secrets from a world he had long forgotten.
qwen3.6:27b chain turn 4
1. Write meaningful, descriptive variable and function names.
2. Keep functions small and focused on a single responsibility.
3. Write comprehensive tests to catch regressions early.
4. Document complex logic with clear comments and documentation.
5. Consistently adhere to a shared style guide and formatting rules.

battery-dflash — exit 1 seconds 198.6 status fail

reason: harness exit 1

no rows

chain-dflash — exit 1 seconds 198.7 status fail

reason: harness exit 1

no rows

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft); chain-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft)

battery — exit 1 seconds 198.5 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 127 62 23.3 15.0 False False False True
battery 1 stop 30 0 108 98 462.5 58.3 False False False True
battery 2 stop 31 0 16 13 478.7 64.1 False False False True
battery 3 stop 47 0 22 6 686.1 67.6 False False False True
battery 4 stop 47 0 17 7 687.6 46.2 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    start = 0
    max_start = 0
    max_length = 0
    seen = {}

    for end, char in enumerate(s):
        if char in seen and seen[char] >= start:
            start = seen[char] + 1
        seen[char] = end

        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start

    return s[max_start:max_start + max_length]
```
ornith-1.5:35b-a3b-mq4r battery turn 1
The sky appears blue during the day because sunlight interacts with the atmosphere through a process called Rayleigh scattering, in which shorter blue wavelengths of light scatter off gas molecules much more effectively than longer wavelengths, spreading blue light across the entire sky in every direction. At sunset, the Sun sits lower on the horizon, meaning its light must travel through a significantly thicker layer of atmosphere to reach your eyes. This longer path filters out most of the scattered blue light, allowing the longer red and orange wavelengths to dominate and paint the sky with warm hues.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the Seine River runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{"name": "Alice", "age": 34, "city": "Lisbon"}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

chain — exit 1 seconds 198.8 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 length 44 0 256 153 719.7 71.0 False False True True
chain 1 stop 353 0 144 75 178.5 61.7 False False False True
chain 2 stop 523 497 90 74 443.8 53.1 False False False True
chain 3 stop 647 613 132 107 549.5 44.5 False False False True
chain 4 stop 811 779 84 62 461.6 49.2 False False False True
ornith-1.5:35b-a3b-mq4r chain turn 0
```python
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list.

    Uses a two-pointer approach that walks through both input lists in a
    single pass, always appending the smaller of the two current elements.
    Runs in O(n + m) time where n and m are the lengths of the inputs.
    """
    result = []
    i, j = 0, 0

    # Walk through both lists simultaneously.
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1

    # Append whatever remains from either list.
    result.extend(a[i:])
    result.extend(b[j:])

    return result
```

**How it works:**

The function uses two pointers (`i` for `a`, `j` for `b`) that advance through the lists in parallel. At each step it compares the current elements and appends the smaller one, keeping the result sorted. Since the inputs are already sorted, you never need to look back — every comparison
ornith-1.5:35b-a3b-mq4r chain turn 1
# Total Distance Calculation

For each leg of the trip, I use the formula: **Distance = Speed × Time**

## Leg 1
- Speed: 60 mph
- Time: 2.5 hours
- Distance = 60 × 2.5 = **150 miles**

## Leg 2
- Speed: 40 mph
- Time: 1.5 hours
- Distance = 40 × 1.5 = **60 miles**

## Total
$$150 + 60 = \boxed{210 \text{ miles}}$$

The train traveled **210 miles** in total.
ornith-1.5:35b-a3b-mq4r chain turn 2
The seasons are caused by Earth's axial tilt of about 23.5 degrees relative to its orbit around the Sun. As Earth revolves around the Sun over the course of a year, different parts of the planet receive varying amounts of direct sunlight depending on their angle toward or away from it. This changing angle and duration of sunlight—not our distance from the Sun—is what produces the cyclical pattern of spring, summer, autumn, and winter.
ornith-1.5:35b-a3b-mq4r chain turn 3
Every dawn, Mara climbed the lighthouse tower to trace the sea's restless line with her gaze, never expecting anything but the usual drift of kelp and stone. That morning, however, a glint of glass caught the pale light near the jagged rocks, and she hurried down to investigate. Half-buried in the wet pebbles lay a sealed bottle, its letter inside so faded she could barely make out the words, yet they seemed written straight for her. She clutched it against her coat and looked out at the horizon, wondering if the ocean had finally carried her an answer to a question she had never dared to ask.
ornith-1.5:35b-a3b-mq4r chain turn 4
1. Name things clearly and consistently so their intent is obvious at a glance.
2. Keep functions small and focused on a single responsibility.
3. Avoid duplicating logic by extracting reusable helpers when patterns repeat.
4. Comment on "why," not "what," to explain reasoning behind non-obvious decisions.
5. Keep changes minimal and commit frequently so code stays easy to review and revert.

battery-dflash — exit 0 seconds 0.0 status skip

reason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft

no rows

chain-dflash — exit 0 seconds 0.0 status skip

reason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft

no rows

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft); chain-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft)

battery — exit 1 seconds 198.5 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 122 57 None 208.55 False False False True
battery 1 stop 0 0 67 58 None 209.38 False False False True
battery 2 stop 0 0 15 13 None 208.33 False False False True
battery 3 stop 0 0 26 8 None 209.68 False False False True
battery 4 stop 0 0 31 19 None 209.46 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_sub(s):
    char_index = {}
    start = 0
    max_length = 0
    max_sub = ""

    for i, char in enumerate(s):
        if char in char_index and char_index[char] > start:
            start = char_index[char] + 1
        char_index[char] = i
        if i - start + 1 > max_length:
            max_length = i - start + 1
            max_sub = s[start:i+1]

    return max_sub
```
lfm2.5:1.2b battery turn 1
The sky appears blue during the day because blue light from the sun is scattered more effectively by small atmospheric particles. At sunset, the sun is lower in the sky, so the red light travels through more atmosphere, dispersing and making the sky look red. This interplay of scattering and the sun's position creates the colors we see.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the Seine River runs through it.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
Let's add the numbers: 17 + 26.  
17 + 26 = 43  
So the final answer is: Answer: 43

chain — exit 1 seconds 198.3 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 0 0 243 133 None 202.84 False False False True
chain 1 stop 0 0 164 105 None 188.51 False False False True
chain 2 stop 0 0 69 58 None 180.63 False False False True
chain 3 stop 0 0 95 73 None 174.63 False False False True
chain 4 stop 0 0 66 48 None 170.54 False False False True
lfm2.5:1.2b chain turn 0
```python
def merge_sorted(a, b):
    """
    Merges two already-sorted lists into a single sorted list.

    Parameters:
    a (list): First sorted list
    b (list): Second sorted list

    Returns:
    list: A new sorted list containing all elements from both input lists.
    """
    i, j = 0, 0
    merged = []
    
    # Traverse both lists and merge them in sorted order
    while i < len(a) and j < len(b):
        if a[i] < b[j]:
            merged.append(a[i])
            i += 1
        else:
            merged.append(b[j])
            j += 1

    # Append remaining elements from list a, if any
    while i < len(a):
        merged.append(a[i])
        i += 1

    # Append remaining elements from list b, if any
    while j < len(b):
        merged.append(b[j])
        j += 1

    return merged
```

This function efficiently merges two sorted lists into a single sorted list using the `i` and `j` indexes.
lfm2.5:1.2b chain turn 1
Here's how to solve the problem step by step:

1. **First part of the journey:**
   - Speed = 60 mph
   - Time = 2.5 hours
   - Distance = Speed × Time = 60 × 2.5 = 150 miles

2. **Second part of the journey:**
   - Speed = 40 mph
   - Time = 1.5 hours
   - Distance = Speed × Time = 40 × 1.5 = 60 miles

3. **Total distance:**
   - Total Distance = Distance in first part + Distance in second part
   - Total Distance = 150 + 60 = 210 miles

**Final Answer:** 210 miles

The train traveled a total of **210 miles**.
lfm2.5:1.2b chain turn 2
The seasons on Earth are caused by the Earth's axial tilt of approximately 23.5 degrees relative to its orbit around the Sun. As the Earth revolves, different parts of the planet receive varying amounts of sunlight throughout the year, leading to seasonal changes. These shifts in sunlight intensity and angles create the cyclical patterns known as the seasons.
lfm2.5:1.2b chain turn 3
On a crisp winter morning, the lighthouse keeper noticed a small, weathered box washed up on the rocky shore, its contents mysterious. Curiosity piqued, he carefully opened the box to reveal an ancient map etched in faded ink. As he examined the map, it hinted at a long-lost treasure hidden near the coast. With a thrill of adventure, the keeper decided to follow the clues into the next phase of his daily duty.
lfm2.5:1.2b chain turn 4
1. Use clear and descriptive variable names to improve readability.  
2. Break your code into modular functions for better organization.  
3. Write comments to explain complex sections for future readers.  
4. Keep a consistent coding style to enhance maintainability.  
5. Regularly review and refactor code to eliminate redundancies.

battery-dflash — exit 0 seconds 0.0 status skip

reason: fixture lfm2.5:1.2b declares no dflash_draft

no rows

chain-dflash — exit 0 seconds 0.0 status skip

reason: fixture lfm2.5:1.2b declares no dflash_draft

no rows

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1; chain: harness exit 1; battery-dflash: harness exit 1; chain-dflash: harness exit 1

battery — exit 1 seconds 198.4 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 129 58 64.3 43.9 False False False True
battery 1 stop 30 0 100 88 359.4 49.5 False False False True
battery 2 stop 31 0 16 13 378.6 62.5 False False False True
battery 3 stop 47 0 31 8 454.8 58.2 False False False True
battery 4 stop 47 0 9 3 455.1 31.7 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    best_start = 0
    best_end = 0

    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start > best_end - best_start:
            best_start = start
            best_end = end

    return s[best_start:best_end + 1]
```
qwen3.8:27b-mq4-xt battery turn 1
During the day, sunlight scatters off the molecules in the atmosphere, with shorter blue wavelengths scattering more strongly in all directions, which makes the sky appear blue when viewed away from the sun. At sunset, the sun is lower on the horizon, so its light travels through a much thicker layer of the atmosphere, causing the blue light to scatter out of the direct line of sight. This leaves the longer red and orange wavelengths to dominate the view, resulting in the warm colors characteristic of a sunset.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the River Seine runs through it.
qwen3.8:27b-mq4-xt battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.8:27b-mq4-xt battery turn 4
43

Answer: 43

chain — exit 1 seconds 198.7 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 44 0 120 51 375.0 62.0 False False False True
chain 1 stop 224 0 210 116 614.5 62.5 False False False True
chain 2 stop 464 0 71 59 751.2 35.8 False False False True
chain 3 stop 573 0 95 77 727.4 34.1 False False False True
chain 4 stop 704 0 73 53 708.8 29.3 False False False True
qwen3.8:27b-mq4-xt chain turn 0
```python
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list."""
    result = []
    i, j = 0, 0
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1
    # Append remaining elements
    result.extend(a[i:])
    result.extend(b[j:])
    return result
```
qwen3.8:27b-mq4-xt chain turn 1
To find the total distance traveled, we calculate the distance for each leg of the trip separately and then add them together. The formula for distance is:

$$ \text{Distance} = \text{Speed} \times \text{Time} $$

**Step 1: Calculate the distance for the first leg.**
*   Speed = 60 mph
*   Time = 2.5 hours
*   Distance = $60 \times 2.5 = 150$ miles

**Step 2: Calculate the distance for the second leg.**
*   Speed = 40 mph
*   Time = 1.5 hours
*   Distance = $40 \times 1.5 = 60$ miles

**Step 3: Add the distances together.**
*   Total Distance = $150 + 60 = 210$ miles

**Final Answer:**
The train traveled a total of **210** miles.
qwen3.8:27b-mq4-xt chain turn 2
The seasons are primarily caused by the tilt of Earth's axis relative to its orbital plane around the Sun. As Earth orbits, this tilt results in varying angles of sunlight and day lengths across different latitudes throughout the year. These changes determine the amount of solar energy received, leading to the cyclical pattern of winter, spring, summer, and fall.
qwen3.8:27b-mq4-xt chain turn 3
Elias brushed the brine from his coat and stepped onto the jagged rocks, expecting only seaweed or driftwood. Instead, his lantern light caught the glint of a brass pocket watch, perfectly intact and ticking softly in the foam. He pulled it from the cold water, recognizing the intricate engraving on the back as his late father's initials. For the first time in decades, the silence of the island felt less like solitude and more like a conversation.
qwen3.8:27b-mq4-xt chain turn 4
1. Write descriptive, single-purpose variable and function names.
2. Keep functions small and focused on one specific task.
3. Avoid code duplication by reusing logic through functions or modules.
4. Add concise comments explaining why code exists, not just what it does.
5. Write unit tests to verify behavior and enable safe refactoring.

battery-dflash — exit 1 seconds 198.4 status fail

reason: harness exit 1

no rows

chain-dflash — exit 1 seconds 198.6 status fail

reason: harness exit 1

no rows

kernel

status: fail
reason: harness exit 1

report pass: True

lane hipx (gfx1100)

hw-gate evidence

field value
base 3bd2914bac09bb6d48b03cf5b0155473ff1bc922
head fa30b3ae247a64f9acc44ddeeca28678c0ba9bf0
buckets kernel,load,serve
host gfx gfx1100
host rocm 7.15.26333-0000000
device 0
runner hipx
daemon_md5 2f6b10605a61dcb45c86d8769ef3e06c
hipfire_md5 594eedc165bc975b7bc6d99a0217c258
build_seconds 47.34608697891235
verdict fail
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery-dflash: harness exit 1; chain-dflash: harness exit 1

battery — exit 0 seconds 30.1 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 132 62 3.3 17.6 False False False True
battery 1 stop 30 0 76 65 439.9 49.8 False False False True
battery 2 stop 31 0 19 16 451.0 49.9 False False False True
battery 3 stop 47 0 31 8 462.2 49.8 False False False True
battery 4 stop 47 0 6 2 466.0 50.0 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for i, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = i
        if i - start + 1 > max_length:
            max_length = i - start + 1
            max_start = start
            
    return s[max_start:max_start + max_length]
```
qwen3.6:27b battery turn 1
During the day, the atmosphere scatters shorter blue wavelengths of sunlight in all directions, making the sky appear blue to our eyes. At sunset, sunlight travels through a thicker layer of atmosphere, which scatters away most of the blue and green light before it reaches the observer. This leaves the longer red and orange wavelengths to dominate the view, creating the characteristic colors of sunset.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
Answer: 43

chain — exit 0 seconds 23.9 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 44 0 115 46 390.7 49.6 False False False True
chain 1 length 215 160 256 125 392.8 49.7 False False True True
chain 2 stop 493 44 85 72 285.8 49.6 False False False True
chain 3 stop 612 579 107 80 294.6 49.6 False False False True
chain 4 stop 751 720 70 50 351.0 49.5 False False False True
qwen3.6:27b chain turn 0
```python
def merge_sorted(a, b):
    """Merge two already-sorted lists into one sorted list."""
    result = []
    i, j = 0, 0
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1
    result.extend(a[i:])
    result.extend(b[j:])
    return result
```
qwen3.6:27b chain turn 1
To find the total distance traveled, we calculate the distance for each segment of the trip and then add them together.

**Step 1: Calculate the distance for the first segment.**
The train travels at 60 mph for 2.5 hours.
$$ \text{Distance}_1 = \text{Speed} \times \text{Time} $$
$$ \text{Distance}_1 = 60 \text{ mph} \times 2.5 \text{ hours} = 150 \text{ miles} $$

**Step 2: Calculate the distance for the second segment.**
The train travels at 40 mph for 1.5 hours.
$$ \text{Distance}_2 = \text{Speed} \times \text{Time} $$
$$ \text{Distance}_2 = 40 \text{ mph} \times 1.5 \text{ hours} = 60 \text{ miles} $$

**Step 3: Calculate the total distance.**
Add the distances from both segments.
$$ \text{Total Distance} = \text{Distance}_1 + \text{Distance}_2 $$
$$ \text{Total Distance} = 
qwen3.6:27b chain turn 2
The seasons on Earth are primarily caused by the planet's axial tilt of approximately 23.5 degrees relative to its orbital plane around the Sun. This tilt means that different hemispheres receive varying amounts of direct sunlight and daylight hours throughout the year as Earth revolves around the Sun. When a hemisphere is tilted toward the Sun, it experiences summer due to more direct solar radiation, while the hemisphere tilted away experiences winter.
qwen3.6:27b chain turn 3
Elias had watched the storm-tossed coast for thirty years, expecting nothing but driftwood and kelp. That morning, however, a glint of unnatural green light caught his eye among the jagged black stones. He knelt beside a small, glass bottle that hummed with a low, rhythmic vibration, far older than any shipwreck he had ever known. As he uncorked it, a whisper of salt air rushed out, carrying the voice of the first keeper who had tended this light centuries ago.
qwen3.6:27b chain turn 4
1. Keep functions small and focused on a single responsibility.
2. Use clear, descriptive names for variables, functions, and classes.
3. Write comprehensive tests to catch regressions early.
4. Document complex logic and decisions with concise comments or docstrings.
5. Follow consistent coding standards and formatting rules across the project.

battery-dflash — exit 1 seconds 7.1 status fail

reason: harness exit 1

no rows

chain-dflash — exit 1 seconds 7.1 status fail

reason: harness exit 1

no rows

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft); chain-dflash: skipped (fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft)

battery — exit 0 seconds 34.2 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 160 79 412.9 23.0 False False False True
battery 1 stop 30 0 101 90 485.0 92.7 False False False True
battery 2 stop 31 0 16 13 441.4 131.3 False False False True
battery 3 stop 47 0 31 8 628.4 123.4 False False False True
battery 4 stop 47 0 17 7 647.2 75.4 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring_without_repeating_chars(s: str) -> str:
    if not s:
        return ""
    
    char_index = {}  # character to its latest index
    max_length = 0
    start = 0  # start index of current window
    best_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        
        if end - start + 1 > max_length:
            max_length = end - start + 1
            best_start = start
    
    return s[best_start:best_start + max_length]
```
ornith-1.5:35b-a3b-mq4r battery turn 1
The sky appears blue during the day because sunlight interacts with Earth's atmosphere through a process called Rayleigh scattering, in which shorter blue wavelengths of light scatter more easily off air molecules than longer wavelengths. At sunset, the sun sits lower on the horizon, meaning its light must travel through a much greater thickness of atmosphere, causing most of the blue light to be scattered away before reaching your eyes. This leaves primarily the longer red and orange wavelengths to reach your eyes directly, producing the characteristic warm sunset colors.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the River Seine runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

chain — exit 0 seconds 23.4 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 length 44 0 256 145 536.7 109.3 False False True True
chain 1 stop 352 0 150 62 131.7 62.7 False False False True
chain 2 stop 528 502 74 59 427.3 41.3 False False False True
chain 3 stop 636 602 102 82 612.7 79.0 False False False True
chain 4 stop 770 738 99 73 536.9 65.4 False False False True
ornith-1.5:35b-a3b-mq4r chain turn 0
```python
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list.

    Uses a two-pointer approach to combine `a` and `b` in O(n + m) time.
    Neither input is modified, and no built-in sort is used.
    """
    result = []
    i, j = 0, 0

    # Advance through both lists, appending the smaller element each time.
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1

    # Copy whatever remains (at most one of the two sublists is non-empty).
    result.extend(a[i:])
    result.extend(b[j:])
    return result
```

**How it works:**
- Two pointers (`i`, `j`) track the current position in each list.
- At each step, the smaller of the two current elements is appended to `result`, and the corresponding pointer advances.
- When one list is exhausted, the remainder of the other is appended directly.


ornith-1.5:35b-a3b-mq4r chain turn 1
**Step 1: Distance for the first leg**

$$\text{Distance} = \text{Speed} \times \text{Time} = 60 \text{ mph} \times 2.5 \text{ hours} = 150 \text{ miles}$$

**Step 2: Distance for the second leg**

$$40 \text{ mph} \times 1.5 \text{ hours} = 60 \text{ miles}$$

**Step 3: Add the two legs together**

$$150 + 60 = 210 \text{ miles}$$

**Total distance traveled: 210 miles** ✅
ornith-1.5:35b-a3b-mq4r chain turn 2
The seasons are caused by the 23.5-degree tilt of Earth's axis relative to its orbit around the Sun. As Earth orbits the Sun over the course of a year, this tilt causes different hemispheres to receive varying amounts of direct sunlight. When one hemisphere is tilted toward the Sun, it experiences summer, while the opposite hemisphere simultaneously experiences winter.
ornith-1.5:35b-a3b-mq4r chain turn 3
Elias had tended the lighthouse for thirty years and thought he'd seen every secret the ocean chose to return. On a fog-chilled morning, he found a small wooden box wedged in the rocks, its surface carved with symbols he didn't recognize. Inside lay a cluster of glass vials, each holding a tiny, glowing light that flickered like a captured star. As he lifted one free, the water behind him began to boil without wind, and something below the surface began to rise.
ornith-1.5:35b-a3b-mq4r chain turn 4
1. Use clear, descriptive names for variables and functions so the intent is obvious at a glance.
2. Keep functions small and focused, ensuring each one does a single well-defined task.
3. Consistently follow the project's style and formatting conventions so the code looks uniform.
4. Write meaningful comments only where the reasoning isn't self-evident from the code itself.
5. Avoid duplicating logic by extracting shared behavior into reusable helpers when patterns repeat.

battery-dflash — exit 0 seconds 0.0 status skip

reason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft

no rows

chain-dflash — exit 0 seconds 0.0 status skip

reason: fixture ornith-1.5:35b-a3b-mq4r declares no dflash_draft

no rows

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft); chain-dflash: skipped (fixture lfm2.5:1.2b declares no dflash_draft)

battery — exit 0 seconds 8.0 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 119 58 None 224.11 False False False True
battery 1 stop 0 0 62 55 None 227.11 False False False True
battery 2 stop 0 0 18 16 None 230.77 False False False True
battery 3 stop 0 0 26 8 None 230.09 False False False True
battery 4 stop 0 0 21 13 None 228.26 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_sub(s):
    seen = {}
    start = 0
    max_length = 0
    max_sub = ""
    for i, char in enumerate(s):
        if char in seen and start <= seen[char]:
            start = seen[char] + 1
        else:
            if i - start + 1 > max_length:
                max_length = i - start + 1
                max_sub = s[start:i+1]
        seen[char] = i
    return max_sub
```
lfm2.5:1.2b battery turn 1
The sky appears blue during the day because sunlight interacts with gases in the atmosphere, scattering shorter blue wavelengths more effectively. At sunset, the sunlight travels through a longer path of the Earth's atmosphere, causing more scattering of red and orange wavelengths. This results in the vibrant red colors we see as the sun sets.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
We add the two numbers together: 17 + 26 = 43.  
Answer: 43

chain — exit 0 seconds 13.5 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 0 0 188 96 None 160.14 False False False True
chain 1 stop 0 0 139 87 None 165.48 False False False True
chain 2 stop 0 0 63 55 None 158.69 False False False True
chain 3 stop 0 0 91 70 None 153.2 False False False True
chain 4 stop 0 0 71 54 None 154.01 False False False True
lfm2.5:1.2b chain turn 0
```python
def merge_sorted(a, b):
    """
    Merges two already-sorted lists into a single sorted list without using sorted().

    Args:
        a (list): First sorted list.
        b (list): Second sorted list.

    Returns:
        list: A new list containing elements from both a and b in sorted order.
    """
    merged = []
    i, j = 0, 0

    # Traverse both lists and append smaller element to merged
    while i < len(a) and j < len(b):
        if a[i] < b[j]:
            merged.append(a[i])
            i += 1
        else:
            merged.append(b[j])
            j += 1

    # Append remaining elements from a or b
    merged.extend(a[i:])
    merged.extend(b[j:])

    return merged
```
lfm2.5:1.2b chain turn 1
Let's break down the train's journey step by step.

### Step 1: Train travels at 60 mph for 2.5 hours
Distance = Speed × Time  
First segment:  
Distance�� = 60 mph × 2.5 hours = 150 miles

### Step 2: Train travels at 40 mph for 1.5 hours
Distance�� = 40 mph × 1.5 hours = 60 miles

### Total Distance
Total distance = Distance�� + Distance��  
= 150 miles + 60 miles = **210 miles**

### Final Answer:
The train traveled a total of **210 miles**.
lfm2.5:1.2b chain turn 2
The seasons on Earth are caused by the relative movement of the Earth and the Sun, as well as the tilt of the Earth's axis. When the Northern Hemisphere tilts toward the Sun, it experiences summer, while the Southern Hemisphere experiences winter. This changing angle of sunlight affects the length and intensity of the seasons.
lfm2.5:1.2b chain turn 3
One foggy evening, a young lighthouse keeper notices a strange, weathered object bobbing in the crashing waves near the rocks. Curious, he carefully collects the item, which turns out to be a beautifully preserved mermaid tail. As he examines it, he feels a sudden chill in the air and hears a faint whisper of sea stories. That night, he dreams of the sea's secrets, forever changed by the mysterious encounter.
lfm2.5:1.2b chain turn 4
1. Use clear and descriptive variable names to improve readability.  
2. Break your code into small, reusable functions for better organization.  
3. Write comments to explain complex logic for other developers.  
4. Regularly test your code to catch errors early and maintain stability.  
5. Follow consistent naming conventions and style guidelines throughout your project.

battery-dflash — exit 0 seconds 0.0 status skip

reason: fixture lfm2.5:1.2b declares no dflash_draft

no rows

chain-dflash — exit 0 seconds 0.0 status skip

reason: fixture lfm2.5:1.2b declares no dflash_draft

no rows

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery-dflash: harness exit 1; chain-dflash: harness exit 1

battery — exit 0 seconds 19.6 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 132 62 301.5 28.2 False False False True
battery 1 stop 30 0 100 80 417.9 48.3 False False False True
battery 2 stop 31 0 19 16 430.6 48.5 False False False True
battery 3 stop 47 0 22 6 478.1 48.4 False False False True
battery 4 stop 47 0 17 7 479.7 48.5 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.8:27b-mq4-xt battery turn 1
Sunlight, which contains all visible colors, enters Earth's atmosphere where gas molecules scatter shorter-wavelength blue light much more effectively than longer-wavelength red light, a process known as Rayleigh scattering. During the day, when the sun is high, we see this scattered blue light coming from all directions in the sky. At sunset, sunlight travels through a much longer path of atmosphere, scattering away almost all the blue light and leaving the less-scattered red and orange wavelengths to dominate the view.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.8:27b-mq4-xt battery turn 3
{"name": "Alice", "age": 34, "city": "Lisbon"}
qwen3.8:27b-mq4-xt battery turn 4
17 + 26 = 43

Answer: 43

chain — exit 0 seconds 21.2 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 44 0 116 47 386.9 48.3 False False False True
chain 1 stop 216 161 234 122 392.8 48.3 False False False True
chain 2 stop 476 451 97 84 276.9 48.3 False False False True
chain 3 stop 607 574 133 105 295.1 48.2 False False False True
chain 4 stop 772 741 73 54 336.3 48.1 False False False True
qwen3.8:27b-mq4-xt chain turn 0
```python
def merge_sorted(a, b):
    """Merge two already-sorted lists into one sorted list."""
    i = 0
    j = 0
    result = []
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1
    result.extend(a[i:])
    result.extend(b[j:])
    return result
```
qwen3.8:27b-mq4-xt chain turn 1
To find the total distance traveled, we need to calculate the distance for each segment of the trip and then add them together. The formula for distance is:

$$ \text{Distance} = \text{Speed} \times \text{Time} $$

**Step 1: Calculate the distance for the first segment**
*   Speed: 60 mph
*   Time: 2.5 hours
*   Distance $d_1 = 60 \times 2.5 = 150$ miles

**Step 2: Calculate the distance for the second segment**
*   Speed: 40 mph
*   Time: 1.5 hours
*   Distance $d_2 = 40 \times 1.5 = 60$ miles

**Step 3: Calculate the total distance**
*   Total Distance $= d_1 + d_2$
*   Total Distance $= 150 + 60 = 210$ miles

**Final Answer:**
The train traveled a total of **210 miles**.
qwen3.8:27b-mq4-xt chain turn 2
The seasons on Earth are primarily caused by the planet's axial tilt of approximately 23.5 degrees relative to its orbital plane around the Sun. As Earth orbits the Sun, this tilt causes different hemispheres to receive varying angles and durations of sunlight throughout the year, leading to longer days and higher solar intensity in one hemisphere while the other experiences shorter days and lower intensity. This differential exposure to solar radiation creates the alternating patterns of warming and cooling that we recognize as seasons.
qwen3.8:27b-mq4-xt chain turn 3
Elias wiped the salt spray from the lantern glass when he heard the rhythmic thud of debris hitting the jagged shore below. He hurried down the spiral stairs, his heart pounding not from exertion but from the peculiar glint he caught in the moonlight on the black rocks. There, wedged between the barnacles and seaweed, was not a broken plank or a rusted buoy, but a pristine, unopened letter bearing his own name in his late wife’s handwriting. He stood frozen in the brine-soaked cold, tears mixing with the salt on his face, knowing with terrifying certainty that she had passed away forty years prior.
qwen3.8:27b-mq4-xt chain turn 4
1. Write clear, descriptive variable and function names that explain their purpose.
2. Keep functions small and focused on a single responsibility.
3. Use consistent formatting and adhere to established style guides.
4. Add concise comments to explain complex logic or non-obvious decisions.
5. Write automated unit tests to verify functionality and prevent regressions.

battery-dflash — exit 1 seconds 7.1 status fail

reason: harness exit 1

no rows

chain-dflash — exit 1 seconds 7.1 status fail

reason: harness exit 1

no rows

kernel

status: pass

report pass: True

@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate sol verdict

{
  "claim_verdict": "disproven",
  "confidence": 0.98,
  "coverage": {
    "gaps": [
      "No successful DFlash generation route evidenced speculative decode, context-exhaustion handling, or emitter-error rollback.",
      "No fault-injection evidence covered transactional GPU allocation rollback or sticky 700/719 poisoning.",
      "No multi-GPU EP/PP route exercised the new admission and DeviceMesh topology behavior.",
      "Registry pull/rm sidecar lifecycle and policy changes were not exercised end to end.",
      "The gfx12/hiptrx lane timed out or failed every fixture route after about 198 seconds, and its Redline run timed out on a subsequent load despite an internally passing first report.",
      "No successful MQ4-V2 parity run covered the changed gfx12 dispatch-sensitive surface."
    ],
    "surfaces_evidenced": [
      "kernel",
      "load",
      "serve",
      "prompt-cache"
    ],
    "surfaces_touched": [
      "kernel",
      "load",
      "serve",
      "prompt-cache",
      "speculative-decode",
      "rollback",
      "multi-gpu-topology",
      "registry-policy",
      "config",
      "filesystem",
      "docs"
    ]
  },
  "decision": "block",
  "eyeball": [
    "qwen3.8:27b-mq4-xt gfx1100 chain: responses are semantically coherent, but all five turns report cached=0, directly disproving the central cache-hit claim.",
    "qwen3.6:27b gfx1100 chain: later turns show substantial cache hits, demonstrating that the harness can observe working prefix caching.",
    "lfm2.5:1.2b gfx1100 chain turn 2: inspect the repeated `Distance\ufffd\ufffd` replacement characters.",
    "All hiptrx fixture routes should be inspected as infrastructure or device-selection failures: coherent rows were produced, but every route ended with exit 1 after roughly 198 seconds.",
    "The paired-Qwen DFlash logs need inspection because both models failed before emitting a single decoded row."
  ],
  "phase": "verdict",
  "rationale": "Block: the central Qwen3.8 claim is directly disproven by the gfx1100 chain, where cached remained zero after turn one. Both Qwen DFlash routes also failed with no output, and the mandatory kernel lane is globally failed because hiptrx timed out during load. Although the gfx1100 Redline report showed stable sequences and exact HIP/AQL/PM4 parity, substantial rollback, topology, policy, DFlash, and gfx12 coverage remains absent.",
  "regressions": [
    {
      "beta_behavior": "The advertised template-aware primer probe did not restore caching on the tested Qwen3.8 fixture: every turn in the gfx1100 qwen3.8:27b-mq4-xt chain reported cached=0.",
      "evidence": "hipx/gfx1100 qwen3.8:27b-mq4-xt chain: cached values were 0,0,0,0,0, while the qwen3.6:27b control produced later-turn cache hits of 160/44/579/720. This directly contradicts the claimed Qwen3.8 cache restoration.",
      "file": "crates/hipfire-runtime/src/prompt_frame.rs",
      "line": 1498,
      "master_behavior": "Qwen3.8 multi-turn Jinja replay should extend the prior prompt-cache prefix after turn one.",
      "severity": "high"
    },
    {
      "beta_behavior": "Both paired Qwen fixtures failed before producing any DFlash rows.",
      "evidence": "hipx qwen3.6:27b battery-dflash and chain-dflash exited 1 after about 7.1 seconds with zero rows; qwen3.8:27b-mq4-xt did the same. The ordinary AR battery and chain routes passed on the same lane.",
      "file": "crates/hipfire-generate/src/qwen.rs",
      "line": 2204,
      "master_behavior": "A registry-paired DFlash model should load and complete battery and chain generation when explicitly tested with DFlash.",
      "severity": "high"
    },
    {
      "beta_behavior": "The LFM chain emitted Unicode replacement characters inside otherwise ordinary mathematical labels.",
      "evidence": "hipx lfm2.5:1.2b chain turn 2 contains repeated `Distance\ufffd\ufffd` text. The harness marked the route pass, but decoded replacement characters are a coherence failure under the hardware-gate eyeball rule.",
      "file": "crates/hipfire-runtime/src/prompt_frame.rs",
      "line": 1498,
      "master_behavior": "Decoded multi-turn output should remain valid text.",
      "severity": "medium"
    }
  ]
}

Floor: hard=['hw_run_result=failure', "evidence verdict='fail'", 'kernel status != pass', 'policy_paths: registry/models.json,registry/v1.json,scripts/leanup-thresholds.txt'] soft=["coverage_gaps: ['No successful DFlash generation route evidenced speculative decode, context-exhaustion handling, or emitter-error rollback.', 'No fault-injection evidence covered transactional GPU allocation rollback or sticky 700/719 poisoning.', 'No multi-GPU EP/PP route exercised the new admission and DeviceMesh topology behavior.', 'Registry pull/rm sidecar lifecycle and policy changes were not exercised end to end.', 'The gfx12/hiptrx lane timed out or failed every fixture route after about 198 seconds, and its Redline run timed out on a subsequent load despite an internally passing first report.', 'No successful MQ4-V2 parity run covered the changed gfx12 dispatch-sensitive surface.']"] model_decision=block final=block

Kaden-Schutt and others added 14 commits September 4, 2026 20:19
Resolving a path-form model to its registry entry changed /health.model
from the requested path to the tag. serve_harness's warm probe compares
health.model to the launched path by realpath, so it never saw the serve as
warm and killed/respawned it every 180 s (measured: two spawn attempts,
zero turns). Keep the entry lookup for sidecars and policy; name the served
model the way it was requested.
Measured on a 7900 XTX (serve_harness session_coding, greedy, thinking off,
q8 KV): qwen3.8:27b-mq5 + its mq5 draft completes turns 1-2 (tau 3.6/3.5)
then dies at turn 3, ctx ~4.9k, with spec_step hipMemCreate out of memory;
every later turn is an empty response. The same session under AR passes all
8 turns (13.4k ctx, 38.1 -> 34.4 tok/s). 18.7 GB weights + ~5 GB fixed
residency + 1.7 GB draft leaves no room for KV growth. Drop the sidecar
from the 27B mq5/mq6 tiers (and qwen3.5:27b-mq6); mq4-tier and below keep
theirs (measured 202 tok/s on qwen3.8:27b).
…ll declares

hw-gate Fable seat on #686 (hardware probe): `hipfire rm qwen3.8:27b-mq4-pro` deleted qwen38-27b-dflash-mq4.hfq while qwen3.8:27b and qwen3.8:27b-mq4-xt — both declaring that sidecar — were still on disk; those siblings then ran AR under dflash_mode=auto or refused under `on`. Same shape for the mq3 draft (3 targets) and the 9B mq4 draft (3 targets).

rm now removes a declared sidecar only when no OTHER registry entry declaring the same file still has its own target present in the models dir; otherwise it prints `keeping DFlash sidecar <file>: still declared by <tags>`. rm_command is a thin wrapper over rm_with_registry(&RegistryV1) so the rule is unit-testable without env or network.

Tests: rm_keeps_shared_dflash_sidecar_while_sibling_target_present, rm_removes_dflash_sidecar_with_last_declaring_target, rm_without_dflash_declaration_leaves_draft_file_alone.
hw-gate Fable seat on #686: a daemon that went through one refused dflash_mode=on load held ~5.17 GB more VRAM with the next model resident than a clean daemon, compounding under serve's lazy retry. Cause: free_qwen35_bundle returns every buffer to the Gpu pool (free_tensor has no size cap, dispatch.rs:3261) and only unload_model drains it (lib.rs:3871-3872); a load that fails in finish_qwen35_load never reaches unload_model, so the whole target stayed pooled and the next load reused only the same-sized buckets.

rollback_unfinished_qwen35 now mirrors unload_model: invalidate_graph_state + drain_pool after the frees. Covers all three callers: CASK eviction failure, dflash_mode=on draft failure, and the pre-existing mtp=on head failure (same leak, older than this PR).
… artifact

hw-gate Fable seat on #686 (run 33889233321): resolve_tag matched any path by its final component, so `hipfire rm /elsewhere/qwen3.6-27b.mq4` — a different file sharing the basename — resolved to the qwen3.6:27b entry and deleted the installed model's triattn and DFlash sidecars while the model itself stayed; a same-basename foreign file loaded by path inherited the entry's sidecar and kv/max_seq policy.

registry: resolve_tag drops the file_name() arm (a path is not a tag); bare entry.file names still resolve; new entry_for_file for exact matches.

cli: registry_entry_for_path(paths, registry, input) — a path-form input resolves only when canonicalize(input) == canonicalize(models_dir/entry.file) for some entry, so the -xt symlink into ~/qcal still matches by target and a lookalike elsewhere never does. rm, run, bench, and serve (incl. the pre-warm thread) route through it. dflash_mode=on on a path with no entry and no explicit draft now fails closed: "DFlash draft required (dflash_mode=on) but <path> is not a registry-managed artifact; pass developer.dflash_draft or use the registry tag" (auto still serves it as a bare artifact) — closes Fable's earlier note that `on` ran AR silently there. The daemon only consumes CLI-lowered params and needs no change.

cargo test -p hipfire-registry: 21 passed (2 new); -p hipfire-cli: 220 passed (4 new).
# Conflicts:
#	crates/hipfire-cli/map.md
#	crates/hipfire-runtime/map.md
…, pool drain on refused load, path-identity fail-closed) to staging

Policy floor (registry/models.json, registry/v1.json) means the decide seat can never land this; merged by Main on the gate's hardware evidence after four rounds of Fable findings, all fixed and measured:
- pool drain on refused load (d233d2a): 18,950 MB retained -> 281 MB, and the same daemon then serves qwen3.8+draft at -20 KB vs a clean daemon (hipx gfx1100)
- shared-sidecar rm guard (def19e3): rm no longer deletes a sidecar another target still declares
- path-basename identity (876cf28): registry_entry_for_path requires canonicalize equality, so a foreign /elsewhere/qwen3.6-27b.mq4 gets no sidecar and dflash_mode=on fails closed instead of running AR silently

Enablement is unchanged: dflash_mode default stays off and the sidecar is resolved only under auto/on, so a paired draft on disk still never drafts until the user opts in. The only user-visible delta is pull size: +0.55 GB (9B) / +0.92-0.98 GB (27B). 249 tests pass in hipfire-registry + hipfire-cli at 397a366 (includes the beta merge with regenerated cli/runtime crate maps).
# Conflicts:
#	crates/hipfire-generate/map.md
#	crates/hipfire-loader/map.md
…ogy combinations before allocation) to staging

Fable's verdict was merge-staging; the only thing that stopped the seat was the recurring generated-map 409 (staging_merge_conflict hard floor on crates/hipfire-{generate,loader}/map.md). Merged by Main after merging beta in and regenerating those blocks with scripts/check-crate-maps.py. Run 33914516085: both lanes pass, and the decide phase ran the #683 repro on the 5x gfx1201 host rather than reading the diff:
- ornith-1.5:35b-a3b --tp 4 is refused at pre-warm with all four cards still at 32548 MB free, where master loads four ranks (~7.86 GB each) and only fails on the first request with 'EP arch mismatch'
- the one path the PR closes that master could reach — the batch-only EP route (batch_staging.rs:231-331) — segfaults inside libamdhip64 on master, 2/2 runs, so refusing it removes a crash rather than a serve path
- dense qwen3.5/3.6 TP through the same entry still loads two ranks and decodes coherently at 20.1 tok/s on the PR build
- LFM2 --continuous-batch-size 2 is byte-identical on both builds while master allocated an Lfm2DecodeBatchState nothing ever drove (batch.rs:180 returns false unconditionally); LFM2 --tp 2 refuses identically on both

Sol was needs-human on coverage, not on a regression: no Gemma4 or DeepSeek-V4/MiniMax fixture exists on this host, and those arms are arch/config-only code before device init. Recorded as unproven in the decision artifact, not as a claim. 70 tests pass in hipfire-loader + hipfire-generate at 5dbe4a9.
# Conflicts:
#	crates/hipfire-runtime/map.md
… axis) to staging

Fable's verdict was merge-staging; blocked only by the recurring generated-map 409 (staging_merge_conflict on crates/hipfire-runtime/map.md). Merged by Main after merging beta in and regenerating that block. Run 33914554146: both lanes pass, and the decide phase measured the claims on the 5x gfx1201 host:
- a 4x gfx1201 EP load of ornith-1.5:35b-a3b-mq4r on the PR daemon logged 'EP load: tp=4 arch=qwen35' after init_ep, completed on 4 ranks, then unloaded/reloaded/unloaded with rank-0 vram_free_mb 32548/25512/32400/25510/32398 -- the identical five numbers the master daemon produced from the same stdin, so the Ep mesh axis is a zero-runtime-delta rename as claimed
- Gpus::init_ep is field-for-field init_tp except mesh: DeviceMesh::rect(Ep, n), and the mesh field is read only by constructors and tests

Sol's needs-human was coverage, not regression: no AWQ/PARO Llama artifact was cycled with free-VRAM diagnostics, and the Qwen2 free_all path has no fixture on either lane. Both recorded as unproven rather than as claims. 21 tests pass in hipfire-runtime + hipfire-loader at a0aa126.
# Conflicts:
#	crates/hipfire-daemon/map.md
#	crates/hipfire-loader/map.md
#	crates/hipfire-loader/src/lib.rs
…fy and admit before teardown) to staging

Fork PR (fivetide), so it is staged in this repo as staging/pr-682: PR head 95e0e65 merged with beta at a9f4ca8. That merge had a REAL semantic conflict in crates/hipfire-loader/src/lib.rs between #687's ep_admission and #682's admit_source; resolved by keeping #682's classify-once split (load_model_ep_admitted) and #687's shared ep_unsupported_arch_message(id) as the backstop arm so the message cannot drift.

The resolution is proven on hardware (hiptrx gfx1201, daemon md5 99457c71e4e400793d3efede7278ac3f), not just compiled:
- battery on the canonical qwen3.8-27b.mq4-xt (mq4v2) trunk: 5/5 turns, attractor 0, empty 0, runaway 0, recall satisfied on every turn, avg decode 29.5 tok/s
- LFM2 (arch_id=11) --tp 2 refuses with exactly the shared constructor's text, 'EP not supported for arch_id=11 (expected 5|6 for Qwen3.5, 9 for DeepSeek V4 or 10 for MiniMax)', at [validation retryable=false rolled_back=false] with the serve process still alive
- ornith-1.5-35b-a3b (arch_id=6) --tp 2 refuses through #687's rule at 32548 MB free / 32624 MB total, i.e. #682's restructure did NOT bypass the refusal that #687 added - the precise risk of that conflict
- 48 tests pass in hipfire-loader + hipfire-daemon

Fable's investigation (run 33921475093) closed every one of Sol's coverage gaps on hardware: six refusal classes all fire before teardown with the prior model still generating, while the BASE daemon answers 'no model loaded' after four of the six.

Two floors, both human calls, both made: scripts/leanup-thresholds.txt is a policy path, and the ratchet raise daemon_lines <= 4155 -> 4176 is accepted - the logic landed in hipfire_loader::admission and what grew in the daemon is the call site plus emit_uncorrelated_error boilerplate that must live at the protocol boundary. If that ceiling is hit again, collapse the repeated emit blocks into a local helper first.

Unproven and recorded as such: DS4 (arch 9) and MiniMax (arch 10) EP admission, and Qwen3.5-VL / LFM2-VL tower classification - no such artifact exists on either lane.
Each of the three EP loaders (ds4, minimax, qwen35) now checks
gpus.mesh.size_of(Ep) == n right after the existing devices-vs-tp
check, so a future constructor that records the wrong mesh axis fails
loudly at load instead of loading silently mislabeled.

Proof: cargo check -p hipfire-loader clean (no new warnings);
cargo test -p hipfire-loader: 34 passed, 0 failed.
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator Author

Correcting the finding on this PR, because the record as it stands points at the wrong defect.

I pulled the serve log from the run that produced it (gate run 33907487315, fable-evidence/dflash-chain-qwen36-serve.log). In order:

[qwen-cache store dflash]  fp=0xb8df43230469ae1b cached_seq=1485
[qwen-cache lcp dflash-jinja] prior_len=1528 rendered_len=1582 lcp=1528
[qwen-cache HIT dflash] reuse prefix=1528 suffix=54 (no reset)
[qwen-cache store dflash]  fp=0xd7a7d25bf966ba4c cached_seq=683
[qwen-cache lcp dflash-jinja] prior_len=2266 rendered_len=2290 lcp=2266
[qwen-cache HIT dflash] reuse prefix=2266 suffix=24 (no reset)
spec_prefix_realign: HipError(719): hipMemcpy H2D offset: unspecified launch failure
[qwen-cache jinja lookup dflash] fp=0xb8df... hit=false
[qwen-cache jinja lookup dflash] fp=0xd7a7... hit=false
prefill: HipError(719): hipMemsetAsync: unspecified launch failure (reset_recurrent: qwen3…)   ×3

lcp == prior_len on every hit — full-prefix reuse, suffix=54 then suffix=24, no reset. There is no stored + 2 slack in the log, and the later lookups are hit=false outright rather than partial LCPs.

So the "primer replay systematically misses the most recent assistant body" reading does not survive its own evidence: the ordering shows the HipError(719) comes first and the misses follow it. 719 is hipErrorLaunchFailure — sticky and context-killing — so once it fires, every subsequent lookup legitimately misses and no amount of reset_recurrent can recover.

That makes the real work here:

  1. spec_prefix_realign's H2D copy faulting. The usual cause of a sticky 719 is a launch against a pointer that has been freed or reallocated. We hit that exact class twice tonight in sibling PRs — dangling kernargs after a rollback (fix(dflash): transactional draft ctors, emitter-error rollback, one ctx-cap predicate #691) and a VRAM pool never drained on a refused load (feat(dflash): registry-declared draft sidecars — pull fetches, auto uses, on requires #686). Worth asking whether the realign path assumes device state that a cache HIT reusing a prefix (no reset) never re-established.
  2. The retry loop. After a sticky 719 the context is dead, yet the daemon retried three more times and produced three more 719s. Failing the request closed on the first sticky error is a correctness fix independent of the root cause.

The AR half is unaffected and was already proven on both lanes. An agent is on (1) and (2) now; I'll post what it finds. Config for the record: qwen3.6-27b, DFlash on, ChatML, thinking on with non-empty reasoning_content, tool_calls=0.

…fills

The DFlash-arm finding on this PR pointed at primer replay; the serve log does
not support that. Gate run 33907487315
(`fable-evidence/dflash-chain-qwen36-serve.log`) shows full-prefix reuse on
every hit -- `prior_len=1528 rendered_len=1582 lcp=1528` then
`prior_len=2266 rendered_len=2290 lcp=2266`, both `no reset`, suffixes 54 and
24 -- so `lcp == prior_len` and there is no store/replay asymmetry. The
`hit=false` lookups that follow are the rollback clearing `asst_turn_cache`
(common.rs:277), i.e. fail-closed behaviour working, not a second defect.

What actually happens is a sticky `HipError(719)` (`hipErrorLaunchFailure`) in
the `spec.prefill` miss path during mid-window `spec_prefix_realign`
(qwen.rs:3591, formatted at :3625). The op label `hipMemcpy H2D offset`
(hip-bridge ffi.rs:1063) points at `draft_seed_backfill`'s ring upload
(dflash.rs:2169) on a windowed run. It is not an overrun: every size check on
that path panics, and the log carries a HipError rather than a panic. The chunk
D2H downloads that precede the backfill would have reported an already-dead
context first, so the context was alive through the seed -- leaving either the
backfill's H2D destination pointer or, more likely, a backfill kernel failing
with the next H2D reporting it stickily. Localising that needs a device-sync
bisection across the backfill chunk ops on hardware, recorded rather than
guessed.

Independent of that root cause, the retry behaviour is wrong on its own terms.
719 kills the context, so `reset_recurrent` cannot recover it -- yet the run
shows three further attempts, each producing another 719 from a memset that
never had a chance. This commit makes the first sticky fault the last:

- reset_core.rs: STICKY set {700, 719}, a first-wins `GpuPoison` latch, and
  `note_hip_error` / `note_hip_result` / `gpu_poison` / `clear_gpu_poison`,
  with a lifecycle unit test
- qwen35 weights.rs: `DeltaNetState::reset` latches sticky memset failures
- qwen35 dflash_spec.rs: the prefill seed/backfill/logits error sites latch on
  the typed `HipError`, no string sniffing
- daemon main.rs: a pre-generate poison check fails fast -- class `gpu`,
  non-retryable, "process restart required", mirroring the `batch_poisoned`
  shape already in that file

The latch never auto-clears: a model unload/reload does not reset the primary
HIP context, so clearing would just burn one prefill per reload. The CLI
gateway already refuses these errors (validation, non-retryable, max one
retry), so the repeats seen in the log came from above it; they now stop at the
daemon.

Verified: `cargo check` clean on hipfire-runtime, hipfire-arch-qwen35,
hipfire-daemon. `cargo test -p hipfire-runtime --lib` 599 passed;
`-p hipfire-arch-qwen35 --lib` 189 passed; reset_core 10 passed including
`sticky_poison_latch_lifecycle`. Crate maps regenerated for the three crates.

Not fixed here, and recorded rather than invented: the DFlash bake omits the
ChatML newline trailer AR appends (qwen.rs:3895 vs ar.rs:4122), proven
LCP-harmless by test; and the `reasoning_content` guard forces a plain-render
fallback on thinking sessions with echoed reasoning, whose slack (+3/+4) does
not match the hardware signature (+2).
@github-actions github-actions Bot removed the needs-human hw-gate reviewer requests a human decision label Sep 5, 2026
Kaden-Schutt added a commit that referenced this pull request Sep 5, 2026
…iffs

Two policy gaps this ladder exposed.

1. The gate never ran DFlash. #686 (draft sidecars), #691 (draft ctor
   rollback), #692 (primer replay) and #702 (dedicated verify kernels) all went
   through with every lane green while speculation never once executed. #692's
   DFlash-arm defect -- primer replay systematically missing the most recent
   assistant body -- was found only because a seat thought to drive twenty turns
   by hand. That is not a gate.

   The load bucket now runs `battery-dflash` and the serve bucket
   `chain-dflash`: the same prompts with `--dflash on` and an explicit
   `--draft`. `on` rather than `auto` because `auto` silently falls back to AR
   when the draft is missing, and a route that can pass without speculating
   proves nothing. The draft is named explicitly because the canonical xt trunk
   is a symlink out of the models dir, so the daemon's filename auto-match finds
   nothing and would run AR.

   `dflash_draft` is a candidate LIST because the lanes hold different drafts:
   hiptrx has qwen36-27b-dflash-mq4.hfq and no qwen38, hipx has
   qwen38-27b-dflash-mq4.hfq and no qwen36. A lane speculates with the first
   candidate it holds; a lane holding none records `skip`.

   `skip` is neither pass nor fail. The aggregation was
   `all(status == "pass")`, which would have counted a skip as a fixture
   failure -- a false negative on evidence the host never had -- while treating
   it as a pass would claim coverage that did not happen. Skips are recorded and
   reported, and a genuine failure alongside a skip still fails.

   Coverage is asymmetric until both hosts hold both drafts. Pulling
   qwen38-27b-dflash-mq4.hfq to hiptrx and qwen36-27b-dflash-mq4.hfq to hipx
   (0.92 GB each) makes it symmetric; that is a disk decision, so the evidence
   says `skip` rather than silently pulling.

2. Sol refused hardware for any diff touching a filesystem path, which caught
   #689 for adding `--prompt-file` to `hipfire bench` and cost that rung a lane
   until `hw-run` overrode it. hipfire is a CLI inference engine: users name
   models, prompts, drafts and sidecars at invocation, and the gate's own
   harness passes exactly those flags. sol.md now separates whose path it is --
   an explicit argument is ordinary product work; credentials, dotfiles, SSH or
   cloud config, /proc or /sys beyond device enumeration, assembled traversal,
   or a read whose result leaves the process still warrant refusal.

Tests: eight new cases in scripts/hw-gate/tests/test_run.py covering flag
translation (battery-dflash -> `--mode battery --dflash on --draft ...`), plain
battery never receiving a draft, per-lane draft selection, skip-not-fail with
the harness never invoked, chain-dflash keeping its own prompts, the
skip-vs-genuine-failure aggregation, and a manifest assertion that the buckets
actually carry the routes. 113/113 hw-gate tests pass.
bind(mesh): assert Ep topology on every EP load path
…Reduce

Remove the v1.1 stub Gpus::init_vram_weighted (zero callers) and the
never-constructed CollectiveHint::AllReduce variant. Scrub matching docs
and crate map entries.
overhaul(s6a): delete dead runtime multi_gpu/mesh symbols

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict needs-human: All executed fixtures passed on gfx1201 and gfx1100, decoded text is coherent, qwen3.8 AR chains demonstrate the intended cache extension, and qwen3.6 controls show no observed template regression. Redline also reports stable captures and exact HIP/blob/AQL state parity. However, the branch additionally changes DFlash replay, daemon-wide sticky GPU-fault state, plain-Llama MQ-V2 admission, and benchmark CLI behavior without direct evidence for those paths. The central AR fix is supported, but the author's AR-and-DFlash equivalence claim is not fully exercised, so the state-machine and speculative-decode coverage gaps require human disposition.

# Conflicts:
#	crates/hipfire-daemon/map.md
#	crates/hipfire-generate/map.md
#	crates/hipfire-runtime/map.md
Kaden-Schutt added a commit that referenced this pull request Sep 5, 2026
…lay) to staging

Run 33934632513: both lanes pass at 370592d (hiptrx gfx1201 + hipx gfx1100,
verdict pass, zero non-pass modes); Sol needs-human with zero regressions;
Fable unavailable (credits), so the staging merge is Main's, not the seat's.

The DFlash-arm finding that held this PR was misdiagnosed, and the log settles
it. Gate run 33907487315 (fable-evidence/dflash-chain-qwen36-serve.log) shows
prior_len=1528 rendered_len=1582 lcp=1528, then prior_len=2266 rendered_len=2290
lcp=2266 -- full-prefix reuse both times, 'no reset', suffixes 54 and 24. So
lcp == prior_len and there is no store/replay asymmetry; the hit=false lookups
that follow are the rollback clearing asst_turn_cache (common.rs:277), i.e.
fail-closed behaviour working.

What actually happened is a sticky HipError(719) in the spec.prefill miss path
during mid-window spec_prefix_realign (qwen.rs:3591), whose 'hipMemcpy H2D
offset' label points at draft_seed_backfill's ring upload (dflash.rs:2169) on a
windowed run. Not an overrun: every size check there panics, and the log carries
a HipError. Root-causing the fault itself needs a device-sync bisection across
the backfill chunk ops on hardware; that is recorded, not guessed.

Independent of the cause, the retry behaviour was wrong: 719 kills the context,
so reset_recurrent cannot recover it, yet three further attempts each produced
another 719. 370592d latches the first sticky fault (700/719 first-wins
GpuPoison in reset_core.rs, latching at the qwen35 reset/prefill error sites)
and fails fast at the daemon with class gpu, non-retryable, 'process restart
required'. The latch never auto-clears, because a model reload does not reset
the primary HIP context.

600 tests pass in hipfire-runtime and 189 in hipfire-arch-qwen35 at fa30b3a,
including sticky_poison_latch_lifecycle. Generated crate maps for daemon,
generate and runtime regenerated for the beta merge.
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator Author

Merged to beta as acae74dd4 (branch tip fa30b3ae2).

Run 33934632513: both lanes pass at 370592d17 (hiptrx gfx1201 + hipx gfx1100, verdict: pass, zero non-pass modes), Sol needs-human with zero regressions, Fable unavailable (credits) — so this staging merge is mine, not the seat's, and master stays human-promoted.

The finding that held this PR was misdiagnosed, and its own log settles it — full detail in the comment above. Short version: lcp == prior_len on every hit, so primer replay was never missing the assistant body; a sticky HipError(719) in the windowed draft_seed_backfill H2D upload came first, and the hit=false lookups after it are the rollback clearing asst_turn_cache working as designed.

What landed on top is the fix that is correct regardless of the 719's root cause: the first sticky fault is now the last. 700/719 latch a first-wins GpuPoison in reset_core.rs, the qwen35 reset/prefill error sites latch on the typed HipError (no string sniffing), and the daemon fails fast pre-generate with class gpu, non-retryable, "process restart required". The latch never auto-clears, because a model reload does not reset the primary HIP context — clearing it would just burn one prefill per reload.

600 tests in hipfire-runtime and 189 in hipfire-arch-qwen35, including sticky_poison_latch_lifecycle.

Still open and recorded rather than invented: the 719 itself needs a device-sync bisection across the backfill chunk ops, plus a pointer-lifetime trace on draft_scratch.{target_hidden,k_full,v_full,positions_k} across a HIT → realign. That is hardware work; I have the hosts when you want it.

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict block: Block: the central Qwen3.8 claim is directly disproven by the gfx1100 chain, where cached remained zero after turn one. Both Qwen DFlash routes also failed with no output, and the mandatory kernel lane is globally failed because hiptrx timed out during load. Although the gfx1100 Redline report showed stable sequences and exact HIP/AQL/PM4 parity, substantial rollback, topology, policy, DFlash, and gfx12 coverage remains absent.

ghazni101 pushed a commit to ghazni101/hipfire that referenced this pull request Sep 5, 2026
A rung that merges to `beta` stays OPEN by design -- promoting beta -> master
is the maintainer's call -- so `pull_request.merged` is false and the
merged-PR guard from warpfront#712 does not apply. Every later touch of that branch then
re-runs the full gate on work that is already staged: warpfront#692 and warpfront#723 both
re-ran within minutes of their staging merges, taking the runner from live
rungs, and the same pattern accounted for several of the runs cancelled by hand
tonight.

`select` now asks whether the head is an ancestor of the staging branch. If it
is, the evidence exists and the hardware has nothing to add, so `run_hw` is
false: the lanes, Sol's verdict and the decide phase all skip, and the recorded
decision still governs the status. The PR is not touched and no label changes.

Deliberately an ancestor test rather than a SHA equality test: a rung merges as
a staging commit whose parent is the head, so equality would never match, and
an ancestor test also covers a rung whose branch was merged and then pushed
again without new work.

`workflow_dispatch` is unaffected, so a manual re-gate of a staged rung still
runs -- that is the escape hatch for re-measuring after a gate fix, which is
exactly what warpfront#702 needed tonight.

132/132 hw-gate tests pass; the workflow parses and the select job's step list
and `run_hw` expression were checked.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant