Skip to content

fix(vram): free weight sidecars on unload; record the Ep mesh axis on EP loads - #688

Open
Kaden-Schutt wants to merge 53 commits into
masterfrom
fix/vram-leak-relands
Open

fix(vram): free weight sidecars on unload; record the Ep mesh axis on EP loads#688
Kaden-Schutt wants to merge 53 commits into
masterfrom
fix/vram-leak-relands

Conversation

@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Summary

Two verified findings from the 2026-09-02 audit (PR #685, README findings 3 and 5), re-landed individually as the audit recommended:

  1. Weight sidecar frees. LlamaWeights::free_gpu freed .buf per weight and skipped the PARO rotation and AWQ scale sidecars, leaking one sidecar set per weight per layer on every reload of an AWQ/PARO llama/qwen3 (DeviceBuffer has no Drop). Every WeightTensor now goes through WeightTensor::free_all, mirroring Qwen35Weights::free_gpu / DflashWeights::free_gpu. Qwen2Weights::free_gpu had the same .buf-only pattern and is fixed the same way (no sidecar is allocated for Qwen2 today, but its forward already reads awq_scale).
  2. Gpus::init_ep. Every load_model_ep_* called Gpus::init_tp, which records DeviceMesh::rect(Tp, n), so after an EP load mesh.size_of(Ep) == 1. init_ep is layout-identical to init_tp but records rect(Ep, n); the three EP load sites in hipfire-loader use it. Gpus::single keeps DeviceMesh::single() (absent axes read as 1 by design; the existing single-topology test pins that). feat(runtime): add DeviceMesh topology (fixed #673 G1) #681 landed the mesh type with no readers, so this is the fix before the first reader.

Which crate(s) does this touch?

  • crates/hipfire-runtime (llama.rs, multi_gpu.rs), crates/hipfire-loader, crates/hipfire-arch-qwen2

Test plan

  • cargo test -p hipfire-runtime — 598 pass, incl. new device_mesh_ep_group_and_tp_absent (Ep groups all ranks, size_of(Ep)==n, size_of(Tp)==1, and symmetrically)
  • cargo build -p hipfire-runtime -p hipfire-loader -p hipfire-arch-qwen2 clean
  • Hardware fault scenario for (1) — load an AWQ or PARO (krot>0) llama/qwen3 artifact, unload/reload N times, watch vram_free_mb in the diag JSON: it should stay flat instead of dropping by one sidecar set per weight per layer per reload. Not run: no AWQ llama-family artifact on hipx/hiptrx at the time; there is no freed-buffer accounting hook to unit-test against.
  • hw-gate when ci: hw-gate — two-seat autonomous review rung (Sol decides hardware, Fable decides staging merges) #679 lands

EP examples (ep_minimax.rs, ep_decode_parity.rs, ep_dspark_topology_probe.rs, ep_deepseek4.rs) still call init_tp; dev harnesses with no mesh readers, left for a follow-up.

Kaden-Schutt and others added 30 commits September 4, 2026 15:23
The default ~24-token prompt reports prefill_tok_s ~= 363 tok/s while a
4.4k-token prompt on the same binary reports 886: the short-prompt number
is launch overhead, not prefill, and the JSON gave no way to tell.

- Add --prompt-file <PATH>: prompt read verbatim (raw bytes, no trim),
  mutually exclusive with positional PROMPT words.
- Standard-bench JSON gains top-level prompt_tokens (u64, as the daemon
  reports it in done.prompt_tokens), prompt_md5 (hex md5 of the exact
  prompt bytes), prompt_chars, and warnings[]; warn when prompt_tokens
  < 256 that prefill_tok_s measures launch overhead.
- Same three values printed on the stderr banner next to model:/arch:.
- No existing field renamed, nulled, or removed; default prompt bytes
  unchanged (audit 2026-09-02 fix slice: bench-prompt-evidence).
The daemon's done event has no prompt_tokens key; the prompt is
prefill_tokens (rows prefilled) plus cached_tokens (prompt-cache prefix).
Measured on a 7900 XTX the JSON reported prompt_tokens: null and never
warned on the 24-token default prompt.
…ropped

hw-gate Fable seat on #689: the flag table lost its `--reasoning-on` row (base AGENTS.md:359) while the flag still exists in `hipfire bench --help`. Additive row only, as the PR body says.
… JSON, short-prompt prefill warning) to staging
llama::is_batchable_la admitted MQ4G256V2/MQ6/5/3/2G256V2 for WMMA
prefill only on gfx1200/gfx1201 while qwen35::is_batchable_la admitted
them on gfx11+gfx12 behind HIPFIRE_MQV2_GFX11_WMMA, so plain Llama/Qwen3
dense qt=44 models prefetched per-token on gfx1100/1151 while Qwen3.5/3.8
took WMMA — despite both doc-comments claiming an exact match (audit
2026-09-02 Broken 1).

Move the dtype set + arch set + kill-switch helper into
llama::mqv2_wmma_batchable / llama::mqv2_gfx11_wmma_enabled_from_env in
hipfire-runtime and delegate from both callers, so the lockstep is
structural. MQ4CG256 (qt=45) stays gfx12-only in both by intent.

Tests: rename the two gfx12-only llama admit tests to gfx11+gfx12
expectations, repoint qwen35 env-escape test at the shared helper, and
add mqv2_admit_llama_qwen35_lockstep asserting both gates agree over
the MQ-V2 dtypes x {gfx1100, gfx1151, gfx1201, gfx1030, gfx1010}.
…ections

mq4v2_gemm_parity's Gaussian weights give both halves near-identical
headers, so a wrong half-select hides in quantization noise despite the
header comment promising a systematic blow-up (audit 2026-09-02
Missing 1). Add arm 2 using the disjoint-halves construction from
mq4v2_residual_parity (half0 [-1,1], half1 [96,160]) over the same
batch-size sweep: v2 output must match the f32 reference within 5%
rel-RMS, and the swapped-headers negative control (as in
mq4v2_moe_parity) must DISAGREE. Keep the Gaussian v1-vs-v2 arm and fix
the header comment to state what each arm can and cannot detect. Add a
host-side test proving the fixture discriminates with no GPU.

Docs (audit Would-change 1-2): spec section 9 now records MoE as
production-wired for qt=44, the XBATCH single-row path as ported, and
the gfx11 kt+=2 / residual kt++ stepping; residual_mmq.hip loses its
stale Experimental tag; the gfx12 QKV kernel loses its
HYPOTHESIS/scaffold wording for the validated C-map statement.
DflashScratch::new_with_mq, new_windowed, DflashWeights::load, and
build_generic_dflash_speculator could '?' out mid-construction, leaking
earlier alloc_tensor results (GpuTensor/DeviceBuffer have no Drop).

Record each allocation in a slot vec (gt!/wt!/at!) taken once into the
final owner; the error arm frees completed layers plus staged slots.
new_windowed frees the base scratch via alloc_or_free!; the generic
builder frees weights (and scratch) on later failures. Success path is
byte-identical. Mirrors load_dflash_state's or_free! (audit-Dflash
Broken 3).
The make_spec_emitter Err exit ran after a successful spec.prefill
without production_fail_closed_rollback_live, unlike every other
post-prefill error exit. The target KV/DeltaNet/drafter hidden had
advanced and host seq_pos/conversation_tokens were cleared, so the
next turn could LCP against a dirty GPU. Route it through the same
rollback + fail-closed error (audit-Dflash Broken 4).
…length

generate_dflash fell back to AR only when prompt + max_tokens >
ctx_capacity, but generate_spec hard-errors when prompt + max_tokens
+ block_size > ctx — requests in that band got gen_start followed by
an error instead of the promised AR fallback. Both sites now share
spec_ctx_request_fits (prompt + max + block <= cap).

The mid-loop position + block_size >= ctx_capacity break now sets
SpecRun::ctx_exhausted, which the qwen (v2 + legacy) and dense
epilogues OR into the length decision: finish_reason=length with no
cache store instead of a silent early stop (audit-Dflash Broken 5).
qwen_dflash_semantic_terminal_tests.rs carries historical rustfmt debt; CI
enforces rustfmt on changed files, so adding one test there forces a
6k-line reformat. The new contract lives in qwen_dflash_ctx_exhausted_tests.rs
and the debt file is restored to master byte-for-byte.
…efill_chunk has no V2 arms

hw-gate Fable seat on #690 (run 33895641944), source trace verified: the shared MQ-V2 admit rule made llama::is_batchable_la admit plain Llama/Qwen3-dense qt44/47-50 artifacts to WMMA prefill on gfx11 and gfx12, but llama.rs::forward_prefill_chunk's per-layer matchers (qkv_is_mq ~:2570, wo_is_mq ~:3025, ffn_is_mq ~:3117, w_down_is_mq ~:3248) list only MQ4G256|MQ6G256|MQ3G256|MFP4G32 — an admitted V2 model skips the FWHT rotate and runs the V1 hfq4g256 launchers on V2 blobs: silently incoherent prefill. master's pre-existing mq4_v2_gfx12 arm had the same hole on gfx12; no gfx12 Llama-V2 artifact has ever tripped it.

llama::is_batchable_la now refuses every *G256V2 dtype and MQ4CG256 on every arch, with the reason at the site. qwen35::is_batchable_la keeps the shared mqv2_wmma_batchable rule (its chunk path has the V2 arms; gfx11 kill-switch intact). llama_spec::batched_verify_eligible routes all seven weights through is_batchable_la, so it is covered without an edit. Lockstep test now asserts the true contract: agreement on every non-V2 dtype across 5 arches; for V2, qwen35 admits on gfx11/gfx12 and llama refuses everywhere. Spec §9 row and crate maps corrected.

hipfire-runtime is_batchable_la: 7 passed; qwen35 is_batchable + lockstep: 9 passed.
…id-ladder failure frees all of them

hw-gate Fable seat on #691 (run 33900101473): alloc_or_free! freed the base scratch on failure but the four already-allocated tensors were still locals with no Drop — a failure on the 2nd..5th alloc leaked k_full / v_full / k_cat / v_cat. They are now assigned into s as each succeeds, so the error arm's s.free_gpu covers the whole ladder. Same shape as new_with_mq's at!/live list.
…llama and qwen35; discriminating GEMM parity; spec §9) to staging
…te map

hw-gate staging merge of 136e3f0 hit a 409 on crates/hipfire-runtime/map.md — the generated line-count table that #690 (on beta) and this PR both refreshed. Regenerated via scripts/check-crate-maps.py --write on the merged tree; no source conflicts.
…er-error rollback, one ctx-cap predicate) to staging

Evidence: run 33905366422 — hiptrx gfx1201 + hipx gfx1100 both pass (4 fixtures x battery+chain, 0 attractor/empty/runaway); Sol verdict needs-human with its one technical claim (entry vs loop ctx guard off-by-one) DISPROVEN on hardware in run 33902688013, where Fable also ran 20 DFlash turns identical to master and demonstrated the Broken-5 fix head-to-head. new_windowed alloc-ladder leak found in that run is fixed in 136e3f0. Merged by Main (Fable seat unavailable: credits).
load_model_ep_qwen35 admitted arch-6 MoE through a full 4-rank
weight upload before generate_ep failed it at the dense-TP server.
Refuse when num_experts > 0 right after the host-side config parse,
before Gpus::init_tp, naming the combination and the TP/single-GPU
alternative. Dense Qwen3.5 still takes the dense-TP EP path.
The carrier uploaded lowered weights, scratch and KV before ar.rs
refused generate on Gemma4Lowered. Refuse in load_gemma4_bundle right
after the host-side lowered/eager decision, before the first device
allocation, with the same message (now shared as
LOWERED_GENERATE_REFUSAL so load and generate cannot drift). The
generate arm stays as the fail-closed net.
generate_ep's _ => arm served any non-9/10 EP arch with the DeepSeek4
EP server — wrong weights, wrong protocol. Dispatch explicitly on
EpServeTarget (9/10/5|6 keep their servers) and emit a refusal naming
the arch otherwise. Admission side: both load_model_ep_* entries now
call ep_admission right after the host-side HFQ probe, before any
device init, so LFM2/Cohere2/anything without an EpArch fails at load,
never at first decode.
The generate-side eligibility returned false unconditionally for LFM
while caps advertised true, so staging allocated Lfm2DecodeBatchState
VRAM that was never driven. Make the capability truthful (false) and
remove arch 11 from continuous_batch_route, so the staging fallback arm
runs and no batch state is allocated. The dead eligibility checks stay:
their symbols are still referenced by staging/reset paths. Updates the
two scheduler contract tests that pinned the old behavior.
Route every WeightTensor through WeightTensor::free_all instead of
freeing .buf directly, so the PARO rotation (pairs/theta/scales) and
AWQ scale sidecars are released on unload. Previously each reload of
an AWQ/PARO llama/qwen3 model leaked one sidecar set per weight per
layer (DeviceBuffer has no Drop).

The tied-lm_head alias carries no sidecars by construction
(tied_lm_head_alias sets paro/awq_scale to None), so skipping the
whole output weight when lm_head_aliases_embd is set still frees
exactly once. Mirrors Qwen35Weights::free_gpu and DflashWeights.

Audit: audit-2026-09-02 finding 3 (VRAM leaks on reload),
audit-Runtime.md section 1 (llama.rs:683).
Kaden-Schutt and others added 9 commits September 4, 2026 20:19
…e canonical target

find_model_path canonicalizes, so a target that is a symlink into another
directory (every ladder artifact on the bench boxes) has a parent with no
draft in it and the registry sidecar was never found. load_params now takes
the models dir and the resolver searches it first, then beside the target.
Regression test with a symlinked target. Measured: serve_harness
--speculation dflash on qwen3.8-27b.mq5 ran AR (tau=None, 37 tok/s) before;
the direct-tag probe resolved the draft (tau=14.2).
Resolving a path-form model to its registry entry changed /health.model
from the requested path to the tag. serve_harness's warm probe compares
health.model to the launched path by realpath, so it never saw the serve as
warm and killed/respawned it every 180 s (measured: two spawn attempts,
zero turns). Keep the entry lookup for sidecars and policy; name the served
model the way it was requested.
Measured on a 7900 XTX (serve_harness session_coding, greedy, thinking off,
q8 KV): qwen3.8:27b-mq5 + its mq5 draft completes turns 1-2 (tau 3.6/3.5)
then dies at turn 3, ctx ~4.9k, with spec_step hipMemCreate out of memory;
every later turn is an empty response. The same session under AR passes all
8 turns (13.4k ctx, 38.1 -> 34.4 tok/s). 18.7 GB weights + ~5 GB fixed
residency + 1.7 GB draft leaves no room for KV growth. Drop the sidecar
from the 27B mq5/mq6 tiers (and qwen3.5:27b-mq6); mq4-tier and below keep
theirs (measured 202 tok/s on qwen3.8:27b).
…ll declares

hw-gate Fable seat on #686 (hardware probe): `hipfire rm qwen3.8:27b-mq4-pro` deleted qwen38-27b-dflash-mq4.hfq while qwen3.8:27b and qwen3.8:27b-mq4-xt — both declaring that sidecar — were still on disk; those siblings then ran AR under dflash_mode=auto or refused under `on`. Same shape for the mq3 draft (3 targets) and the 9B mq4 draft (3 targets).

rm now removes a declared sidecar only when no OTHER registry entry declaring the same file still has its own target present in the models dir; otherwise it prints `keeping DFlash sidecar <file>: still declared by <tags>`. rm_command is a thin wrapper over rm_with_registry(&RegistryV1) so the rule is unit-testable without env or network.

Tests: rm_keeps_shared_dflash_sidecar_while_sibling_target_present, rm_removes_dflash_sidecar_with_last_declaring_target, rm_without_dflash_declaration_leaves_draft_file_alone.
hw-gate Fable seat on #686: a daemon that went through one refused dflash_mode=on load held ~5.17 GB more VRAM with the next model resident than a clean daemon, compounding under serve's lazy retry. Cause: free_qwen35_bundle returns every buffer to the Gpu pool (free_tensor has no size cap, dispatch.rs:3261) and only unload_model drains it (lib.rs:3871-3872); a load that fails in finish_qwen35_load never reaches unload_model, so the whole target stayed pooled and the next load reused only the same-sized buckets.

rollback_unfinished_qwen35 now mirrors unload_model: invalidate_graph_state + drain_pool after the frees. Covers all three callers: CASK eviction failure, dflash_mode=on draft failure, and the pre-existing mtp=on head failure (same leak, older than this PR).
… artifact

hw-gate Fable seat on #686 (run 33889233321): resolve_tag matched any path by its final component, so `hipfire rm /elsewhere/qwen3.6-27b.mq4` — a different file sharing the basename — resolved to the qwen3.6:27b entry and deleted the installed model's triattn and DFlash sidecars while the model itself stayed; a same-basename foreign file loaded by path inherited the entry's sidecar and kv/max_seq policy.

registry: resolve_tag drops the file_name() arm (a path is not a tag); bare entry.file names still resolve; new entry_for_file for exact matches.

cli: registry_entry_for_path(paths, registry, input) — a path-form input resolves only when canonicalize(input) == canonicalize(models_dir/entry.file) for some entry, so the -xt symlink into ~/qcal still matches by target and a lookalike elsewhere never does. rm, run, bench, and serve (incl. the pre-warm thread) route through it. dflash_mode=on on a path with no entry and no explicit draft now fails closed: "DFlash draft required (dflash_mode=on) but <path> is not a registry-managed artifact; pass developer.dflash_draft or use the registry tag" (auto still serves it as a bare artifact) — closes Fable's earlier note that `on` ran AR silently there. The daemon only consumes CLI-lowered params and needs no change.

cargo test -p hipfire-registry: 21 passed (2 new); -p hipfire-cli: 220 passed (4 new).
# Conflicts:
#	crates/hipfire-cli/map.md
#	crates/hipfire-runtime/map.md
…, pool drain on refused load, path-identity fail-closed) to staging

Policy floor (registry/models.json, registry/v1.json) means the decide seat can never land this; merged by Main on the gate's hardware evidence after four rounds of Fable findings, all fixed and measured:
- pool drain on refused load (d233d2a): 18,950 MB retained -> 281 MB, and the same daemon then serves qwen3.8+draft at -20 KB vs a clean daemon (hipx gfx1100)
- shared-sidecar rm guard (def19e3): rm no longer deletes a sidecar another target still declares
- path-basename identity (876cf28): registry_entry_for_path requires canonicalize equality, so a foreign /elsewhere/qwen3.6-27b.mq4 gets no sidecar and dflash_mode=on fails closed instead of running AR silently

Enablement is unchanged: dflash_mode default stays off and the sidecar is resolved only under auto/on, so a paired draft on disk still never drafts until the user opts in. The only user-visible delta is pull size: +0.55 GB (9B) / +0.92-0.98 GB (27B). 249 tests pass in hipfire-registry + hipfire-cli at 397a366 (includes the beta merge with regenerated cli/runtime crate maps).
@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate evidence — 2 lane(s) — verdict pass

lane hiptrx (gfx1201)

hw-gate evidence

field value
base 3bd2914bac09bb6d48b03cf5b0155473ff1bc922
head a0aa126817b43f5f76accc5b35495d34c07b4e48
buckets kernel,load,serve
host gfx gfx1201
host rocm 7.15.26333-0000000
device 3
runner hiptrx
daemon_md5 e4ac4ad7260938e5c6ba6a2b17efc813
hipfire_md5 29624d0046e39e42824c7e9913863664
build_seconds 40.78373956680298
verdict pass
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 50.5 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 132 62 1.8 14.2 False False False True
battery 1 stop 30 0 82 70 453.2 31.2 False False False True
battery 2 stop 31 0 16 13 486.2 31.4 False False False True
battery 3 stop 47 0 31 8 530.8 31.3 False False False True
battery 4 stop 47 0 17 7 535.1 31.3 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
            
    return s[max_start:max_start + max_length]
```
qwen3.6:27b battery turn 1
During the day, sunlight passes through a thinner layer of the atmosphere, allowing shorter blue wavelengths to scatter more efficiently in all directions. At sunset, however, light travels through a much thicker section of the atmosphere, which scatters away most of the blue and green light before it reaches your eyes. Consequently, only the longer red and orange wavelengths remain to penetrate the atmosphere, giving the sky its warm hues.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the River Seine runs through it.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
17 + 26 = 43

Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 83.9 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 134 62 23.0 7.6 False False False True
battery 1 stop 30 0 79 69 106.6 7.8 False False False True
battery 2 stop 31 0 16 13 110.4 10.6 False False False True
battery 3 stop 47 0 31 8 159.1 11.9 False False False True
battery 4 stop 47 0 17 7 158.9 7.5 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring_without_repeating_chars(s: str) -> str:
    start = 0
    max_start = 0
    max_len = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        
        if end - start + 1 > max_len:
            max_len = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_len]
```
ornith-1.5:35b-a3b-mq4r battery turn 1
The sky appears blue during the day because sunlight interacts with gases in Earth's atmosphere, and shorter blue wavelengths scatter much more easily than longer wavelengths. At sunset, sunlight must travel through a thicker portion of the atmosphere, which scatters away the blue light before it reaches your eyes. This leaves primarily the longer red and orange wavelengths to reach your eyes directly, creating the characteristic warm sunset colors.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the River Seine runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 9.0 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 121 58 None 145.43 False False False True
battery 1 stop 0 0 88 77 None 145.21 False False False True
battery 2 stop 0 0 15 13 None 144.23 False False False True
battery 3 stop 0 0 26 8 None 145.25 False False False True
battery 4 stop 0 0 20 11 None 144.93 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_substring(s):
    seen = {}
    start = 0
    max_len = 0
    max_sub = ""
    
    for i, char in enumerate(s):
        if char in seen and start <= seen[char]:
            start = seen[char] + 1
        else:
            if i - start + 1 > max_len:
                max_len = i - start + 1
                max_sub = s[start:i+1]
        seen[char] = i
    return max_sub
```
lfm2.5:1.2b battery turn 1
The sky appears blue during the day because sunlight, though white, contains various colors that scatter equally when entering the Earth's atmosphere. This scattering process is known as Rayleigh scattering, which makes the blue light from distant objects more visible during the day. At sunset, the sun is low on the horizon, so the sunlight travels through more atmosphere, scattering the shorter blue wavelengths away from the light beam and leaving mostly red and orange hues visible.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the Seine River runs through it.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
Let's calculate the sum: 17 + 26 equals 43.  
Answer: 43

chain — exit 0 seconds 20.5 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 0 0 238 132 None 143.46 False False False True
chain 1 stop 0 0 150 106 None 137.61 False False False True
chain 2 stop 0 0 82 74 None 133.77 False False False True
chain 3 stop 0 0 84 62 None 130.64 False False False True
chain 4 stop 0 0 53 38 None 128.33 False False False True
lfm2.5:1.2b chain turn 0
```python
def merge_sorted(a, b):
    """
    Merge two pre-sorted lists into a single sorted list without using the built-in sorted() function.
    
    Parameters:
    a (list): First sorted list
    b (list): Second sorted list
    
    Returns:
    list: A new sorted list containing all elements from lists a and b.
    """
    merged = []
    i, j = 0, 0
    # Traverse both lists and add the smaller element to merged
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            merged.append(a[i])
            i += 1
        else:
            merged.append(b[j])
            j += 1
    # Append remaining elements from list a
    while i < len(a):
        merged.append(a[i])
        i += 1
    # Append remaining elements from list b
    while j < len(b):
        merged.append(b[j])
        j += 1
    return merged
```

This function merges two sorted lists by comparing elements iteratively while maintaining order.
lfm2.5:1.2b chain turn 1
Here are the steps to solve the problem:

1. Calculate the distance traveled during the first part of the journey (60 mph for 2.5 hours).
2. Calculate the distance traveled during the second part of the journey (40 mph for 1.5 hours).
3. Add both distances to get the total distance.

Let's do the calculations:

- Distance = Speed × Time
- For the first part: Distance = 60 mph × 2.5 hours = 150 miles
- For the second part: Distance = 40 mph × 1.5 hours = 60 miles

Total distance = 150 miles + 60 miles = 210 miles

Final answer: **210 miles**
lfm2.5:1.2b chain turn 2
The seasons on Earth are caused by the tilt of the Earth's axis as it orbits the Sun, resulting in varying amounts of sunlight reaching different parts of the planet at different times of the year. This tilt causes the Northern Hemisphere to be tilted towards or away from the Sun during different seasons, leading to changes in weather and temperature. Consequently, these shifts in sunlight distribution lead to the seasons we experience annually.
lfm2.5:1.2b chain turn 3
One stormy night, the lighthouse keeper stumbled upon a mysterious shell washed ashore near the rocks. Curious and intrigued, he examined the shell, noticing its unusual markings. As he pulled it closer, the keeper heard a faint voice whispering a cryptic message. Suddenly, a bundle of peculiar letters appeared in his hands, leading him on a wild adventure through history and technology.
lfm2.5:1.2b chain turn 4
1. Keep your code modules small and focused.  
2. Use clear and descriptive variable names.  
3. Write comments to explain complex logic.  
4. Maintain consistent formatting and naming conventions.  
5. Test your code thoroughly to catch issues early.

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 50.8 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 133 62 26.7 10.6 False False False True
battery 1 stop 30 0 125 107 36.6 27.9 False False False True
battery 2 stop 31 0 19 16 38.1 9.7 False False False True
battery 3 stop 47 0 17 1 55.9 54.2 False False False True
battery 4 stop 47 0 9 3 56.6 5.1 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    start = 0
    max_length = 0
    max_start = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.8:27b-mq4-xt battery turn 1
Sunlight contains all colors of the visible spectrum, but when it enters the atmosphere, the shorter blue wavelengths scatter off gas molecules much more efficiently than longer wavelengths, a process known as Rayleigh scattering. This scattered blue light is dispersed throughout the sky, making it appear blue to our eyes during midday when sunlight travels through a relatively thin layer of atmosphere. At sunset, the sun is low on the horizon, forcing sunlight to pass through a significantly thicker layer of atmosphere, which scatters away almost all the blue light before it reaches us and allows the longer, less-scattered red and orange wavelengths to dominate the view.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.8:27b-mq4-xt battery turn 3
{"name":"Alice","age":34,"city":"Lisbon"}
qwen3.8:27b-mq4-xt battery turn 4
43

Answer: 43

kernel

status: pass

report pass: True

lane hipx (gfx1100)

hw-gate evidence

field value
base 3bd2914bac09bb6d48b03cf5b0155473ff1bc922
head a0aa126817b43f5f76accc5b35495d34c07b4e48
buckets kernel,load,serve
host gfx gfx1100
host rocm 7.15.26333-0000000
device 0
runner hipx
daemon_md5 e4ac4ad7260938e5c6ba6a2b17efc813
hipfire_md5 29624d0046e39e42824c7e9913863664
build_seconds 45.91004514694214
verdict pass
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 32.4 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 143 67 3.3 18.3 False False False True
battery 1 stop 30 0 84 73 441.4 49.8 False False False True
battery 2 stop 31 0 19 16 454.1 49.9 False False False True
battery 3 stop 47 0 31 8 465.1 49.8 False False False True
battery 4 stop 47 0 6 2 464.4 49.9 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    if not s:
        return ""
    
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.6:27b battery turn 1
During the day, sunlight scatters more readily off air molecules when traveling a shorter path through the atmosphere, allowing shorter blue wavelengths to dominate what we see. At sunset, the light travels through a much thicker layer of air, which filters out most of the blue and green light before it reaches your eyes. Consequently, only the longer red and orange wavelengths survive this longer journey, giving the sunset its characteristic warm hue.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 38.4 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 130 61 458.3 19.9 False False False True
battery 1 stop 30 0 111 97 467.1 29.2 False False False True
battery 2 stop 31 0 16 13 489.7 107.8 False False False True
battery 3 stop 47 0 31 8 720.4 114.2 False False False True
battery 4 stop 47 0 17 7 726.2 116.0 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring_no_repeats(s: str) -> str:
    start = max_length = 0
    best_start = 0
    char_index = {}

    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end

        if end - start + 1 > max_length:
            max_length = end - start + 1
            best_start = start

    return s[best_start:best_start + max_length]
```
ornith-1.5:35b-a3b-mq4r battery turn 1
During the day, the sky appears blue because sunlight interacts with atmospheric gases through a process called Rayleigh scattering, in which shorter blue wavelengths scatter much more efficiently than longer wavelengths, filling the sky with scattered blue light from all directions. At sunset, the sun sits low on the horizon, meaning its light travels through a much thicker layer of atmosphere, so nearly all of the easily-scattered blue light is dispersed away before reaching your eyes. This leaves predominantly the longer red and orange wavelengths to reach you directly, creating the characteristic warm hues of a sunset.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the Seine River runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 8.1 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 114 56 None 217.97 False False False True
battery 1 stop 0 0 74 65 None 222.89 False False False True
battery 2 stop 0 0 15 13 None 227.27 False False False True
battery 3 stop 0 0 26 8 None 226.09 False False False True
battery 4 stop 0 0 25 13 None 227.27 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_substring(s):
    seen = set()
    start = 0
    max_len = 0
    max_sub = ""

    for i in range(len(s)):
        while s[i] in seen:
            start = start ^ 1 if start != i else start
        seen.add(s[i])
        if i - start + 1 > max_len:
            max_len = i - start + 1
            max_sub = s[start:i+1]
    return max_sub
```
lfm2.5:1.2b battery turn 1
The sky appears blue during the day due to Rayleigh scattering, where sunlight interacts with molecules and small particles in the atmosphere, dispersing shorter blue wavelengths more efficiently. At sunset, the angle of the sunlight decreases, so the longer red wavelengths dominate as they travel through more atmosphere. These physical phenomena blend to produce the vibrant colors we see in different times of the day.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the River Seine runs through it.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
Let's calculate the sum: 17 + 26.  
Adding together gives 43.  
Answer: 43

chain — exit 0 seconds 15.3 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 0 0 237 127 None 166.78 False False False True
chain 1 stop 0 0 192 121 None 176.31 False False False True
chain 2 stop 0 0 89 74 None 152.66 False False False True
chain 3 stop 0 0 100 78 None 151.29 False False False True
chain 4 stop 0 0 61 44 None 150.99 False False False True
lfm2.5:1.2b chain turn 0
```python
def merge_sorted(a, b):
    """
    Merges two sorted lists into a single sorted list.
    
    Parameters:
    a (list): First sorted list.
    b (list): Second sorted list.
    
    Returns:
    list: A new sorted list containing all elements from both input lists.
    """
    merged = []
    i, j = 0, 0
    
    # Traverse both lists and add the smallest elements
    while i < len(a) and j < len(b):
        if a[i] < b[j]:
            merged.append(a[i])
            i += 1
        else:
            merged.append(b[j])
            j += 1
    
    # Append remaining elements from list a
    while i < len(a):
        merged.append(a[i])
        i += 1
    
    # Append remaining elements from list b
    while j < len(b):
        merged.append(b[j])
        j += 1
    
    return merged
```

This function efficiently merges two sorted lists into a single sorted list without using the built-in `sorted()`.
lfm2.5:1.2b chain turn 1
Certainly! Let's break this down step by step.

**Step 1: Calculate distance for the first part of the trip**  
Train speed = 60 mph  
Time = 2.5 hours  

Distance = Speed × Time  
Distance�� = 60 mph × 2.5 hours = 150 miles

**Step 2: Calculate distance for the second part of the trip**  
Train speed = 40 mph  
Time = 1.5 hours  

Distance = Speed × Time  
Distance�� = 40 mph × 1.5 hours = 60 miles

**Step 3: Add both distances together**  
Total distance = Distance�� + Distance�� = 150 miles + 60 miles = 210 miles

**Final Answer:** The train traveled a total of **210 miles**.

Let me know if you want a Python version of this calculation!
lfm2.5:1.2b chain turn 2
The seasons on Earth are caused by the tilt of the planet's axis relative to its orbit around the Sun, which changes the amount of sunlight different regions receive throughout the year. As the Earth spins and its axial tilt shifts, different parts of the planet are exposed to varying angles of sunlight, leading to seasonal changes. This cyclical pattern results in the distinct seasons—spring, summer, autumn, and winter—over the course of a year.
lfm2.5:1.2b chain turn 3
A reclusive lighthouse keeper lived on a remote island, keeping watch over the waves with a steady beam of light. One rainy morning, while checking the rocks near the shore, he spotted an odd, weathered object half-buried in the sand. Intrigued, he carefully dug it out, revealing a hidden box filled with mysterious letters. As he read the messages, the keeper realized he had uncovered a secret from decades past, changing his understanding of his isolated life forever.
lfm2.5:1.2b chain turn 4
1. Use clear and consistent naming conventions.  
2. Write modular code with well-defined functions.  
3. Document your code with comments explaining complex logic.  
4. Keep each function small and focused on a single task.  
5. Regularly refactor your code to maintain readability and efficiency.

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 21.8 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 140 63 310.1 29.0 False False False True
battery 1 stop 30 0 102 87 419.8 48.3 False False False True
battery 2 stop 31 0 19 16 432.6 48.5 False False False True
battery 3 stop 47 0 22 6 478.9 48.2 False False False True
battery 4 stop 47 0 17 7 480.3 48.5 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    if not s:
        return ""
    
    char_index = {}
    start = 0
    best_start = 0
    best_end = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start > best_end - best_start:
            best_start = start
            best_end = end
    
    return s[best_start:best_end + 1]
```
qwen3.8:27b-mq4-xt battery turn 1
During the day, sunlight passes through a greater amount of atmosphere, causing shorter-wavelength blue light to scatter in all directions via Rayleigh scattering, which makes the sky appear blue from our perspective. At sunset, the light travels through a significantly thicker layer of the atmosphere, scattering away most of the blue and green light so that only the longer-wavelength red and orange wavelengths remain visible. This differential scattering based on wavelength allows the red light to reach our eyes directly, creating the characteristic hues of a sunset.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.8:27b-mq4-xt battery turn 3
{"name": "Alice", "age": 34, "city": "Lisbon"}
qwen3.8:27b-mq4-xt battery turn 4
17 + 26 = 43

Answer: 43

kernel

status: pass

report pass: True

@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate sol verdict

{
  "claim_verdict": "not-exercised",
  "confidence": 0.97,
  "coverage": {
    "gaps": [
      "No repeated AWQ/PARO Llama-family unload/reload VRAM measurement exercised WeightTensor::free_all sidecar reclamation.",
      "No multi-rank EP load exercised Gpus::init_ep or inspected Ep/Tp mesh dimensions.",
      "No Qwen2 fixture exercised its changed unload path.",
      "No DFlash run exercised sidecar resolution, mandatory-draft failure, context exhaustion, emitter rollback, or speculative terminal behavior.",
      "No Gemma4 lowered/MoE artifact exercised the new admission refusal.",
      "CLI pull/rm behavior and registry sidecar policy were not exercised end to end.",
      "The changed registry/models.json and registry/v1.json policy requires human review."
    ],
    "surfaces_evidenced": [
      "ordinary model loading and coherent serving on gfx1100 and gfx1201",
      "Llama-family battery and chain generation",
      "rotated-weight artifact generation",
      "Qwen3.5 kernel capture stability and HIP/PM4 bit-exact parity"
    ],
    "surfaces_touched": [
      "load and unload VRAM lifecycle",
      "multi-GPU EP topology",
      "serve and continuous-batch admission",
      "DFlash loading and terminal state",
      "kernel and dispatch",
      "CLI pull, remove, and benchmark behavior",
      "registry policy"
    ]
  },
  "decision": "needs-human",
  "eyeball": [
    "All qwen3.6:27b, qwen3.8:27b-mq4-xt, ornith-1.5:35b-a3b-mq4r, and lfm2.5:1.2b battery outputs were nonempty, responsive, and free of attractors.",
    "Both lfm2.5:1.2b chain runs remained responsive across related turns; the gfx1100 arithmetic response contains replacement characters in subscript-like labels but still reaches the correct 210-mile answer.",
    "The gfx1100 LFM longest-substring sample contains suspicious application logic despite coherent syntax; this is model-answer quality rather than evidence tied to the changed lifecycle code."
  ],
  "phase": "verdict",
  "rationale": "Hardware evidence is clean for ordinary load/serve and kernel execution: both lanes passed, Qwen3.5 capture sequences were stable, and HIP/PM4 state was bit-exact. It does not prove either principal PR claim: crates/hipfire-runtime/src/llama.rs:686 was not tested with repeated sidecar-bearing unload/reload VRAM accounting, and crates/hipfire-runtime/src/multi_gpu.rs:319 was not exercised with multiple EP ranks. The much broader diff also changes DFlash state handling and registry policy without corresponding routes, so human review is required.",
  "regressions": []
}

Floor: hard=['policy_paths: registry/models.json,registry/v1.json'] soft=["coverage_gaps: ['No repeated AWQ/PARO Llama-family unload/reload VRAM measurement exercised WeightTensor::free_all sidecar reclamation.', 'No multi-rank EP load exercised Gpus::init_ep or inspected Ep/Tp mesh dimensions.', 'No Qwen2 fixture exercised its changed unload path.', 'No DFlash run exercised sidecar resolution, mandatory-draft failure, context exhaustion, emitter rollback, or speculative terminal behavior.', 'No Gemma4 lowered/MoE artifact exercised the new admission refusal.', 'CLI pull/rm behavior and registry sidecar policy were not exercised end to end.', 'The changed registry/models.json and registry/v1.json policy requires human review.']", 'model needs-human'] model_decision=needs-human final=needs-human

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict needs-human: Ordinary model loading and generation passed on both gfx1100 and gfx1201 for qwen3.6:27b, qwen3.8:27b-mq4-xt, ornith-1.5:35b-a3b-mq4r, and lfm2.5:1.2b, with coherent decoded output and no demonstrated regression. However, the two behaviors changed by the PR were not directly established: crates/hipfire-runtime/src/llama.rs:686 lacked repeated sidecar-bearing reload and VRAM evidence, while crates/hipfire-runtime/src/multi_gpu.rs:319 could not run with multiple EP ranks. Multi-GPU topology changes and these coverage gaps require human disposition.

# Conflicts:
#	crates/hipfire-generate/map.md
#	crates/hipfire-loader/map.md
…ogy combinations before allocation) to staging

Fable's verdict was merge-staging; the only thing that stopped the seat was the recurring generated-map 409 (staging_merge_conflict hard floor on crates/hipfire-{generate,loader}/map.md). Merged by Main after merging beta in and regenerating those blocks with scripts/check-crate-maps.py. Run 33914516085: both lanes pass, and the decide phase ran the #683 repro on the 5x gfx1201 host rather than reading the diff:
- ornith-1.5:35b-a3b --tp 4 is refused at pre-warm with all four cards still at 32548 MB free, where master loads four ranks (~7.86 GB each) and only fails on the first request with 'EP arch mismatch'
- the one path the PR closes that master could reach — the batch-only EP route (batch_staging.rs:231-331) — segfaults inside libamdhip64 on master, 2/2 runs, so refusing it removes a crash rather than a serve path
- dense qwen3.5/3.6 TP through the same entry still loads two ranks and decodes coherently at 20.1 tok/s on the PR build
- LFM2 --continuous-batch-size 2 is byte-identical on both builds while master allocated an Lfm2DecodeBatchState nothing ever drove (batch.rs:180 returns false unconditionally); LFM2 --tp 2 refuses identically on both

Sol was needs-human on coverage, not on a regression: no Gemma4 or DeepSeek-V4/MiniMax fixture exists on this host, and those arms are arch/config-only code before device init. Recorded as unproven in the decision artifact, not as a claim. 70 tests pass in hipfire-loader + hipfire-generate at 5dbe4a9.
@hipfire-fable

hipfire-fable Bot commented Sep 4, 2026

Copy link
Copy Markdown

announcement: Holding on the hard floor, not on your code: the registry/models.json and registry/v1.json edits that tripped the policy rule come in through your 'Merge beta' commit from PR #686, not from any commit you authored, and this head is already on beta from the previous round — the gate re-ran a merged PR. I re-checked the two claims on the 5x gfx1201 host: at this head ornith 4x EP is refused by #687's admission before init_ep ever runs (and master, while it loaded that topology, could not generate on it), so init_ep is now only reachable from DeepSeek V4 / MiniMax loads, which this host cannot run; the reachable 4-rank dense TP path for qwen3.6:27b is byte-identical to master across load/generate/unload/reload. The sidecar free_all change reads correct by ownership (alias guard, per-weight AWQ upload), but there is no AWQ/PARO llama-family or Qwen2 artifact here to measure it, so it stays recorded as unproven rather than proven. Evidence is under fable-evidence/ with a README index.

investigation:

| question | route | result | evidence |

|---|---|---|---|

| Are the policy-path edits that fired the hard floor (registry/models.json, registry/v1.json) authored by this PR, or inherited from staging? | git diff --stat 3bd2914 3f3d0bc -- registry/ ; git diff --stat 3f3d0bc a0aa126 -- registry/ ; git merge-base --is-ancestor HEAD origin/beta | PR-authored commits (e5a016d, 5c4c7ad, 2cf2cf1, 3f3d0bc) touch only llama.rs, qwen2.rs, multi_gpu.rs, loader/lib.rs and two map.md files; the registry diff against base is EMPTY. Both registry files (+108/-4) enter solely via a0aa126 'Merge beta into fix/vram-leak-relands', i.e. PR #686's dflash-sidecar pairing already on staging. HEAD a0aa126 is already an ancestor of origin/beta (7df822e 'hw-gate: merge PR #688 ... to staging', 21:30 UTC) — this run is a re-gate of a merged head. | README.md (commands and result recorded) |

| Is Gpus::init_ep reachable on this host at this head (the previous round's ornith 4x EP evidence predates the beta merge that brought in #687)? | python3 $HW_GATE_EVIDENCE/ep_probe.py $HW_GATE_BIN/daemon $HIPFIRE_MODELS_DIR/ornith-1.5-35b-a3b.mq4r 4 $HW_GATE_EVIDENCE/ep_admission_ornith_tp4_HEAD.json --devices 0,1,2,3 --generate | Refused in 0.282 s: 'load failed: Qwen3.5-MoE (arch_id=6) has no EP serve path; use TP or single-GPU'. vram_free_mb 32548 before and after; no 'EP load:' stderr line, so the refusal (crates/hipfire-loader/src/lib.rs:3416, from #687) fires before Gpus::init_ep at lib.rs:3434. At this head init_ep is reachable only from the DeepSeek V4 (arch 9, lib.rs:3062) and MiniMax (arch 10, lib.rs:3290) loaders; neither has an artifact on this host. | ep_admission_ornith_tp4_HEAD.json |

| A/B: did master actually serve that arch-6 EP path, i.e. is the inherited refusal a lost working path? | python3 $HW_GATE_EVIDENCE/ep_probe.py $HW_GATE_BASE_BIN/daemon $HIPFIRE_MODELS_DIR/ornith-1.5-35b-a3b.mq4r 4 $HW_GATE_EVIDENCE/ep_admission_ornith_tp4_MASTER.json --devices 0,1,2,3 --generate | Master loads it (16.7 s, '[loader] EP load: tp=4 arch=qwen35 experts=256', vram 32548 -> 25512) but generate fails immediately after gen_start with 'EP arch mismatch (expected dense Qwen TP)'; unload returns vram to 32400. So master's init_tp-based arch-6 EP load was never servable; #687 moved a guaranteed post-load failure to admission. Not a regression of a working path, and not this PR's change. | ep_admission_ornith_tp4_MASTER.json |

| Does the multi-GPU path that IS reachable at this head (dense Qwen3.5 TP=4 through the same load_model_ep_qwen35 function, in the file this PR touches) still load, serve, unload and reload identically to master? | for lane in HEAD:$HW_GATE_BIN MASTER:$HW_GATE_BASE_BIN; do python3 $HW_GATE_EVIDENCE/ep_probe.py $bin/daemon $HIPFIRE_MODELS_DIR/qwen3.6-27b.mq4 4 $HW_GATE_EVIDENCE/dense_tp4_qwen36_27b_cycles_$label.json --devices 0,1,2,3 --generate --cycles 2; done | Both binaries: rank-0 vram_free_mb sequence [32548, 27104, 32320, 27026, 32238] across load/unload/reload/unload — identical to the MB; loads 13.7/14.4 s (HEAD) vs 14.5/14.4 s (master); '[loader] dense qwen TP load complete: 4 ranks'. Decoded reasoning identical on both ('Thinking Process: 1. Identify the core question: The user is asking for the capital of France. 2. Retrieve knowledge: Capital of France = Paris...'); both terminate with the fail-closed 'open think span at end of generation' at my 48-token budget, as designed for a thinking model. | dense_tp4_qwen36_27b_cycles_MASTER.json |

| Does the 4-rank dense TP path produce a complete coherent answer on the PR binary across a reload? | python3 $HW_GATE_EVIDENCE/ep_probe.py $HW_GATE_BIN/daemon $HIPFIRE_MODELS_DIR/qwen3.6-27b.mq4 4 $HW_GATE_EVIDENCE/dense_tp4_qwen36_27b_nothink_HEAD.json --devices 0,1,2,3 --generate --cycles 2 (thinking_enabled=false) | Cycle 0 and cycle 1 (after unload+reload) both decode 'The capital of France is Paris.' (8 tokens); vram sequence again [32548, 27104, 32320, 27026, 32238]. The 'commit_ready -> aborted' terminal is the two-phase wire contract (crates/hipfire-generate/src/dense.rs:3711); my raw driver sends no 'commit', so the daemon's 30 s abort is protocol, not a defect. | dense_tp4_qwen36_27b_nothink_HEAD.json |

| Can WeightTensor::free_all in LlamaWeights::free_gpu / Qwen2Weights::free_gpu double-free a shared sidecar (Sol's suspected regression at llama.rs:686)? | source read: crates/hipfire-runtime/src/llama.rs:526-540, paro.rs:284, hfq.rs:1489/1544, weight_backend.rs:233-247 | No double-free by construction: free_all skips PARO sidecars whose ParoRotation.is_alias is true (alias_paro_rotation sets is_alias: true at paro.rs:284; owners at paro.rs:189 / hfq.rs:1838 set false); awq_scale is a fresh per-weight upload via load_awq_scale at hfq.rs:1489 and :1544 (never a shared handle); the tied lm_head alias (weight_backend.rs tied_lm_head_alias) carries paro: None / awq_scale: None, and free_gpu skips the whole output weight when lm_head_aliases_embd. Static only — no artifact on this host reaches these two free paths. | README.md (citations) |

unproven:

  • LlamaWeights::free_gpu sidecar reclamation (llama.rs:685): needs an arch 0/1 artifact carrying AWQ (<w>.awq_scale.weight) or PARO (krot>0) sidecars, cycled load/unload/reload with diag vram_free_mb — master should lose one sidecar set per weight per layer per reload, HEAD should stay flat. The registry's only arch-1 artifacts (qwen3:0.6b, qwen3:8b, HF4, no sidecars) are not on this host, and no registry artifact at all is arch 0/1 with an awq/paro recipe; a locally quantized v3-awq-f1 Qwen3 (arch 1) on any single GPU would prove it.

  • Qwen2Weights::free_gpu (qwen2.rs:283): arch 7 (vibethinker:3b / :3b-mq6) is not on this host; by the diff's own note no sidecar is allocated for Qwen2 today, so this is behaviorally a no-op until one is.

  • Gpus::init_ep on hardware (multi_gpu.rs:329): at this head only the DeepSeek V4 (deepseek-v4-flash*, arch 9) and MiniMax (minimax-m2.7, arch 10) EP loaders reach it, on a >=2-GPU host; neither artifact is present here. The Ep-axis mesh is pinned only by the unit test device_mesh_ep_group_and_tp_absent, and feat(runtime): add DeviceMesh topology (fixed #673 G1) #681's mesh still has no runtime reader, so a wrong axis would be invisible at runtime either way.

rationale: The hard floor fired on registry/models.json and registry/v1.json, so this decision cannot be merge-staging; it is hold, not block, because every mandatory route passed on both lanes (qwen3.6:27b, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b, qwen3.8:27b-mq4-xt batteries and the lfm2.5 chain all coherent; redline capture stable, HIP/PM4 bit-exact on gfx1100 and gfx1201) and nothing I ran regressed. Two facts the maintainer needs: (1) no PR-authored commit touches the registry — git diff 3bd2914b..3f3d0bc6e -- registry/ is empty; the +108/-4 registry delta arrives only through a0aa126 'Merge beta into fix/vram-leak-relands' and is PR #686's already-gated sidecar pairing, so the floor is reacting to staging content, not to this PR; (2) HEAD a0aa126 is already an ancestor of origin/beta (7df822e, 21:30 UTC, previous round's merge-staging), so this is a re-gate of a merged head — the class of run PR #712 exists to stop. What would clear the floor without any code change: compute policy_paths in select.py against the PR's merge-base with the staging branch (or drop files whose blob equals origin/beta's) rather than against master. On the PR's own claims: the previous round's ornith 4x EP evidence for init_ep is no longer valid at this head — ep_admission_ornith_tp4_HEAD.json shows #687's qwen35_ep_moe_refusal (loader/lib.rs:3416) refusing the load in 0.28 s before Gpus::init_ep (lib.rs:3434), and ep_admission_ornith_tp4_MASTER.json shows master loaded that topology but could not serve it ('EP arch mismatch (expected dense Qwen TP)' on generate), so the inherited refusal removed no working path and init_ep is now reachable only from the arch-9/arch-10 loaders, which have no fixture here. The reachable multi-GPU path in the touched function — qwen3.6:27b dense TP=4 — is byte-identical between HEAD and master across two load/generate/unload cycles (vram [32548,27104,32320,27026,32238] on both; identical decoded text; thinking-off run answers 'The capital of France is Paris.' both cycles). The sidecar-free change is sound by ownership (free_all honors ParoRotation.is_alias at paro.rs:284; awq_scale is per-weight at hfq.rs:1489/1544; the tied lm_head alias carries no sidecars) but is unproven on hardware because no arch 0/1/7 artifact exists on this host, and the registry has no AWQ/PARO llama-family artifact at all. I would put my name on the code; I cannot put it on the sidecar claim as measured, and the floor forbids merging from this seat regardless.

hard floor: ['policy_paths: registry/models.json,registry/v1.json']

soft floor: ["coverage_gaps: ['No repeated AWQ/PARO Llama-family unload/reload VRAM measurement exercised WeightTensor::free_all sidecar reclamation.', 'No multi-rank EP load exercised Gpus::init_ep or inspected Ep/Tp mesh dimensions.', 'No Qwen2 fixture exercised its changed unload path.', 'No DFlash run exercised sidecar resolution, mandatory-draft failure, context exhaustion, emitter rollback, or speculative terminal behavior.', 'No Gemma4 lowered/MoE artifact exercised the new admission refusal.', 'CLI pull/rm behavior and registry sidecar policy were not exercised end to end.', 'The changed registry/models.json and registry/v1.json policy requires human review.']", 'model needs-human']

@hipfire-fable hipfire-fable Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging to staging. I took the EP change onto real hardware: a 4x gfx1201 expert-parallel load of ornith-1.5:35b-a3b-mq4r goes through Gpus::init_ep, completes on 4 ranks, and unloads/reloads with VRAM readings byte-identical to the master binary; the engine's 'EP arch mismatch' refusal at generate is the pre-existing #683 behavior and shows identically on master. The sidecar-free change could not run on LlamaWeights here (no llama-arch artifact in the registry on the gate hosts), so I verified at every sidecar construction site that nothing is shared between weights, and exercised the same free_all body four times in the daemon unload path on an AWQ artifact with flat VRAM and identical output after each reload. If you have an AWQ or PARO llama/qwen3-dense artifact on hand, a load/generate/unload loop watching vram_free_mb would close the last gap. The diff is 105 lines across three behaviors. (1) Gpus::init_ep (multi_gpu.rs:329-366) is field-for-field init_tp (multi_gpu.rs:280-317) except mesh: DeviceMesh::rect(Ep, n); grep of crates/ shows the mesh field is read only by constructors and tests, so the runtime delta is zero. I exercised it on real hardware: a 4x gfx1201 EP load of ornith-1.5:35b-a3b-mq4r on the PR daemon logged 'EP load: tp=4 arch=qwen35' after init_ep, completed on 4 ranks, unloaded, reloaded and unloaded again with rank-0 vram_free_mb 32548/25512/32400/25510/32398 -- the identical five numbers the master daemon produced from the same stdin. The harness battery at tp=4 returned 5 empty turns on both binaries because the engine refuses Qwen3.5-MoE EP generation with 'EP arch mismatch (expected dense Qwen TP)' (qwen.rs:385) -- pre-existing on master, the #683 family that #687 refuses at admission; not this PR's regression, and the two serve logs differ only in timing lines. (2) LlamaWeights::free_gpu -> WeightTensor::free_all: no llama-arch fixture exists on this host so the function itself did not run; the risk it introduces is a double free of a shared sidecar, and I checked every sidecar construction site (hfq.rs:1401/1489/1544, paro.rs:183-191, paro.rs:264-286 with the is_alias skip at llama.rs:530, weight_backend.rs:245-246 with the preserved lm_head_aliases_embd guard at llama.rs:688): each sidecar is uniquely owned by one WeightTensor. The free_all body itself ran 4x in the daemon's real unload path on an AWQ-sidecar artifact (qwen3.5-0.8b.mq4, 138 awq_scale tensors) with vram_free_mb flat at 32314 after every unload and byte-identical decoded output after every reload. (3) Qwen2Weights::free_gpu: no sidecar is ever constructed for Qwen2 (qwen2.rs:640, 687 are the only sites, both None), so free_all is exactly the old free_tensor(.buf). The mandatory batteries (qwen3.6:27b, qwen3.8:27b-mq4-xt, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b on gfx1100 and gfx1201) all decoded coherently; none of them touches a changed line, which is why Sol's coverage read is correct and mine adds the multi-GPU and unload-cycle routes above. One observation outside this diff for the maintainer: the gfx1100 lfm2.5:1.2b chain turn 2 rendered subscript identifiers as 'Distance��' while the gfx1201 lane rendered the same turn cleanly; the PR touches neither LFM2 nor tokenizer decode and I have no gfx1100 device in this sandbox to A/B it, so it is noted, not charged. No regression is plausible against master: the new code paths either did not change runtime behavior (mesh, Qwen2) or only free buffers that were previously leaked and are provably uniquely owned. I would put my name on this merge to staging. Override Sol needs-human: I closed the multi-GPU gap myself on this 5x gfx1201 host: Gpus::init_ep was entered through load_model_ep_qwen35 (ornith-1.5:35b-a3b-mq4r, tp=4), the 4-rank load/unload/reload cycle is byte-identical to the master binary in every VRAM reading and log line, and DeviceMesh has no runtime reader (grep: only the three constructors and tests), so the only behavioral delta is the recorded axis pinned by the new unit test. The remaining gaps (LlamaWeights::free_gpu on a real AWQ/PARO llama-arch artifact; Qwen2) cannot be exercised on any hw-gate host with the current registry, and I verified at every sidecar construction site that no sidecar is shared between two WeightTensors, so the only new failure mode (double free) is excluded structurally; the free_all primitive itself ran 4x in the daemon unload path on an AWQ-sidecar artifact with flat VRAM.

@hipfire-fable hipfire-fable Bot added the needs-human hw-gate reviewer requests a human decision label Sep 4, 2026
# Conflicts:
#	crates/hipfire-runtime/map.md
Kaden-Schutt added a commit that referenced this pull request Sep 4, 2026
… axis) to staging

Fable's verdict was merge-staging; blocked only by the recurring generated-map 409 (staging_merge_conflict on crates/hipfire-runtime/map.md). Merged by Main after merging beta in and regenerating that block. Run 33914554146: both lanes pass, and the decide phase measured the claims on the 5x gfx1201 host:
- a 4x gfx1201 EP load of ornith-1.5:35b-a3b-mq4r on the PR daemon logged 'EP load: tp=4 arch=qwen35' after init_ep, completed on 4 ranks, then unloaded/reloaded/unloaded with rank-0 vram_free_mb 32548/25512/32400/25510/32398 -- the identical five numbers the master daemon produced from the same stdin, so the Ep mesh axis is a zero-runtime-delta rename as claimed
- Gpus::init_ep is field-for-field init_tp except mesh: DeviceMesh::rect(Ep, n), and the mesh field is read only by constructors and tests

Sol's needs-human was coverage, not regression: no AWQ/PARO Llama artifact was cycled with free-VRAM diagnostics, and the Qwen2 free_all path has no fixture on either lane. Both recorded as unproven rather than as claims. 21 tests pass in hipfire-runtime + hipfire-loader at a0aa126.
@github-actions github-actions Bot removed the needs-human hw-gate reviewer requests a human decision label Sep 4, 2026
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator Author

Merged to beta as 7df822e9b (branch tip a0aa12681).

Verdict was merge-staging; the seat was stopped only by the recurring generated-map 409 (staging_merge_conflict on crates/hipfire-runtime/map.md). Merged beta in, regenerated the block, merged.

The decide phase measured both claims on the 5×gfx1201 host rather than reasoning about them:

  • 4× EP load of ornith-1.5:35b-a3b-mq4r on the PR daemon logged EP load: tp=4 arch=qwen35 after init_ep, completed on 4 ranks, then unload → reload → unload with rank-0 vram_free_mb 32548 / 25512 / 32400 / 25510 / 32398 — the identical five numbers the master daemon produced from the same stdin. That is the evidence the Ep mesh axis is a zero-runtime-delta rename.
  • Gpus::init_ep is field-for-field init_tp except mesh: DeviceMesh::rect(Ep, n), and the mesh field is read only by constructors and tests.

Sol's needs-human was coverage, not a regression, and it is worth writing down as follow-up rather than losing: no AWQ/PARO Llama artifact was cycled with free-VRAM diagnostics (so the sidecar-leak fix itself is argued, not measured), and the Qwen2 free_all path has no fixture on either lane. Both are recorded as unproven in the decision artifact.

21 tests pass in hipfire-runtime + hipfire-loader at the merged tip.

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict needs-human: Hardware evidence is clean for ordinary load/serve and kernel execution: both lanes passed, Qwen3.5 capture sequences were stable, and HIP/PM4 state was bit-exact. It does not prove either principal PR claim: crates/hipfire-runtime/src/llama.rs:686 was not tested with repeated sidecar-bearing unload/reload VRAM accounting, and crates/hipfire-runtime/src/multi_gpu.rs:319 was not exercised with multiple EP ranks. The much broader diff also changes DFlash state handling and registry policy without corresponding routes, so human review is required.

@hipfire-fable hipfire-fable Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding on the hard floor, not on your code: the registry/models.json and registry/v1.json edits that tripped the policy rule come in through your 'Merge beta' commit from PR #686, not from any commit you authored, and this head is already on beta from the previous round — the gate re-ran a merged PR. I re-checked the two claims on the 5x gfx1201 host: at this head ornith 4x EP is refused by #687's admission before init_ep ever runs (and master, while it loaded that topology, could not generate on it), so init_ep is now only reachable from DeepSeek V4 / MiniMax loads, which this host cannot run; the reachable 4-rank dense TP path for qwen3.6:27b is byte-identical to master across load/generate/unload/reload. The sidecar free_all change reads correct by ownership (alias guard, per-weight AWQ upload), but there is no AWQ/PARO llama-family or Qwen2 artifact here to measure it, so it stays recorded as unproven rather than proven. Evidence is under fable-evidence/ with a README index. The hard floor fired on registry/models.json and registry/v1.json, so this decision cannot be merge-staging; it is hold, not block, because every mandatory route passed on both lanes (qwen3.6:27b, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b, qwen3.8:27b-mq4-xt batteries and the lfm2.5 chain all coherent; redline capture stable, HIP/PM4 bit-exact on gfx1100 and gfx1201) and nothing I ran regressed. Two facts the maintainer needs: (1) no PR-authored commit touches the registry — git diff 3bd2914b..3f3d0bc6e -- registry/ is empty; the +108/-4 registry delta arrives only through a0aa126 'Merge beta into fix/vram-leak-relands' and is PR #686's already-gated sidecar pairing, so the floor is reacting to staging content, not to this PR; (2) HEAD a0aa126 is already an ancestor of origin/beta (7df822e, 21:30 UTC, previous round's merge-staging), so this is a re-gate of a merged head — the class of run PR #712 exists to stop. What would clear the floor without any code change: compute policy_paths in select.py against the PR's merge-base with the staging branch (or drop files whose blob equals origin/beta's) rather than against master. On the PR's own claims: the previous round's ornith 4x EP evidence for init_ep is no longer valid at this head — ep_admission_ornith_tp4_HEAD.json shows #687's qwen35_ep_moe_refusal (loader/lib.rs:3416) refusing the load in 0.28 s before Gpus::init_ep (lib.rs:3434), and ep_admission_ornith_tp4_MASTER.json shows master loaded that topology but could not serve it ('EP arch mismatch (expected dense Qwen TP)' on generate), so the inherited refusal removed no working path and init_ep is now reachable only from the arch-9/arch-10 loaders, which have no fixture here. The reachable multi-GPU path in the touched function — qwen3.6:27b dense TP=4 — is byte-identical between HEAD and master across two load/generate/unload cycles (vram [32548,27104,32320,27026,32238] on both; identical decoded text; thinking-off run answers 'The capital of France is Paris.' both cycles). The sidecar-free change is sound by ownership (free_all honors ParoRotation.is_alias at paro.rs:284; awq_scale is per-weight at hfq.rs:1489/1544; the tied lm_head alias carries no sidecars) but is unproven on hardware because no arch 0/1/7 artifact exists on this host, and the registry has no AWQ/PARO llama-family artifact at all. I would put my name on the code; I cannot put it on the sidecar claim as measured, and the floor forbids merging from this seat regardless.

@hipfire-fable hipfire-fable Bot added the needs-human hw-gate reviewer requests a human decision label Sep 4, 2026
ghazni101 pushed a commit to ghazni101/hipfire that referenced this pull request Sep 5, 2026
…e-gate a merged PR

Two defects observed on 2026-09-04 while driving the ladder, both of which
cost live rungs their hardware lane:

1. The decide phase takes `flock --exclusive` on all five GPUs around the
   whole `review.py --phase decide` process, for its full 45-minute budget.
   warpfront#711's seat had no provider credits and could never produce a verdict, yet
   it held that lock for ~20 minutes while warpfront#687 and warpfront#688 sat queued with
   'hardware (gfx1201)' unable to start. A seat that cannot answer must not
   own the hardware.

   Fixed with a preflight: one 90 s toolless probe of the decide model before
   the lock is taken. If it replies, the locked phase runs exactly as before.
   If it does not, the locked step is skipped entirely and a new unlocked step
   records the hold via `--decider-unavailable REASON`, which short-circuits
   the model call in review.py and lets the floors, comment, and labels run as
   usual. The GPUs are never claimed.

2. `pull_request_target` fires on `labeled` even for a merged PR, so label
   churn re-ran the entire gate on warpfront#711 five minutes after it merged (run
   33915818350) — taking the runner and the exclusive lock from live rungs.
   `select` now refuses any event whose PR is already merged; `workflow_dispatch`
   is unaffected, so a manual re-gate still works.

Test: `test_decide_unavailable_seat_holds_without_calling_the_model` asserts
hold, the reason recorded, no merge, and no model invocation at all (the fake
never opens its log). 105/105 hw-gate tests pass.
ghazni101 pushed a commit to ghazni101/hipfire that referenced this pull request Sep 5, 2026
Every rung of the 2026-09-04 ladder hit the same 409 on the staging merge:
`crates/*/map.md` carries a `<!-- crate-map:generated -->` block that both
branches regenerate, so any two PRs touching the same crate conflict there
while their real code merges cleanly. warpfront#689, warpfront#690, warpfront#691, warpfront#686, warpfront#687, warpfront#688 and
warpfront#682 all needed the same three manual steps -- merge staging in, regenerate the
block with scripts/check-crate-maps.py, merge -- six of them tonight. A gate
that decides merge-staging and then holds on a generated file is asking a human
to run a script, which is not review.

On a 409 the decide phase now retries locally: merge staging into the PR head,
and if the conflicted set is generated maps only, re-run check-crate-maps.py
for those crates, commit, and merge the result.

The retry is deliberately narrow, because auto-resolving conflicts is exactly
where a gate can do damage:
- if ANY conflicted path is not a `map.md`, it declines and the hold stands
  with the offending paths named -- a real code conflict must reach a human
- it regenerates rather than picking a side, so the committed block is what the
  tree actually generates, not whichever branch won
- a failed regeneration, a git error, or a timeout all decline rather than
  force

Test: `test_generated_map_retry_refuses_real_code_conflicts` builds a real repo
with a conflicting `.rs` and asserts the retry returns no merge SHA and names
the file. The guard is the part worth pinning; the happy path is exercised by
the ladder itself.

122/122 hw-gate tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-human hw-gate reviewer requests a human decision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant