Skip to content

feat(loader): source-aware admission — classify and admit before teardown (fixed #674 G2) - #682

Open
fivetide wants to merge 4 commits into
warpfront:masterfrom
fivetide:replan/g2-admission
Open

feat(loader): source-aware admission — classify and admit before teardown (fixed #674 G2)#682
fivetide wants to merge 4 commits into
warpfront:masterfrom
fivetide:replan/g2-admission

Conversation

@fivetide

@fivetide fivetide commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixed version of #674 (G2 source-aware admission). #674 was reverted in a0fca0d6d alongside #668/#673/#676 because it introduced two admission defects that broke production loads:

  • CLS-001: classified Qwen3.5/LFM2 by vision_config/model_type config markers, so every Qwen3.5-family text artifact (whose HF config always embeds vision_config) was refused at admission for "missing vision tower".
  • self-refusing VMM commit: commit_ordinary re-ran the VMM readiness check after the new model's own VMM tensors existed, counting them as pending teardown and refusing every fresh kv_backend=vmm load.

This PR delivers the same G2 intent — classify once, admit one effective topology, refuse before destructive side effects — without either defect.

What changed

  • crates/hipfire-loader/src/admission.rs (new): read-only admit_source — opens the source, classifies arch_id + vision, decides the effective topology (Single/Pipeline/Expert), and refuses no-carrier / ambiguous-carrier / VMM-allowlist / VMM+pp / carrier-pp / EP-arch / lfm2 vision-no-config / DFlash lm-head quant — all before any GPU, VMM, or teardown work. Vision is decided by the tower tensor (classify_vision), preserving the remediation contract 179a20d7f.
  • Carrier::admit_topology (new trait method, default + qwen35/cohere2moe/maple/gemma4 overrides) — mirrors each carrier's load-time pp>1 refusal so a refusal fires at admission.
  • Load-entry split: load_model_with_gemma4_drafter / load_model_ep_with_kv_mode become thin wrappers that classify-then-load via load_admitted_with_gemma4_drafter / load_model_ep_admitted, which consume the retained SourceAdmission (no re-open, no re-classify).
  • Daemon reorder (crates/hipfire-daemon/src/main.rs): daemon topology refusals + admit_source now run before prior-model teardown. On refusal the prior model stays loaded.

How this differs from #674

#674 (reverted) this PR
VL classification config markers (vision_config/model_type) tower tensor only (tensor_data(...).is_some())
VMM commit re-checked after own tensors exist (self-refuses) single pre-allocation ensure_vmm_ready_for_load
refusal ordering after prior-model teardown before teardown (prior model preserved)
source handle re-opened/re-classified at load retained SourceAdmission consumed once

Verification

  • cargo build (workspace): clean. Full workspace test suite: no failures (incl. 5 new admission contract tests).
  • Scoped rustfmt on changed files: clean.
  • Hardware (gfx1151, isolated HOME):
    • hipfire run qwen3.6:27b → 64/64 layers, KV cache: Q8 vmm (fresh-daemon VMM commit), coherent Paris.
    • hipfire run qwen3.6-35b-a3b.mq4 → coherent linked-list-reverse output.
    • Failed-load probe: a bad-path load (No such file or directory) and a contradictory kv_backend=vmm + pp=2 load both refuse at admission, and a subsequent generate on the prior model returns done in both cases (prior model survives refusal).

Dependencies / nonclaims

  • Root of the re-plan: consumes no upstream types (independent of G1's DeviceMesh; G2 admission decides the topology, G5 will bind it to gpus.mesh).
  • DimKind::Ep classification is unchanged from master (EP still routes by arch_id 5|6|9|10); no new axis, no manifest planning, no family cutover.

…down (G2)

Classify a retained source once and decide one effective topology before
any destructive side effect, so a refused load leaves the prior model
usable (issue warpfront#666 G2).

- admission::admit_source (read-only): open the source, classify arch_id +
  vision (tower-tensor decides; contract 179a20d), decide the effective
  topology (single/pp/ep), and refuse no-carrier / ambiguous / VMM
  allowlist / VMM+pp / carrier-pp / EP-arch / lfm2 vision-no-config /
  DFlash lm-head quant — all before any GPU/VMM/teardown work.
- Carrier::admit_topology (default + qwen35/cohere2moe/maple/gemma4
  overrides) mirrors each carrier's load-time pp refusal.
- Split the load entries: path wrappers classify-then-load via
  load_admitted_with_gemma4_drafter / load_model_ep_admitted, which consume
  the retained SourceAdmission (no re-open, no re-classify).
- Reorder the daemon load handler: daemon topology refusals + admission run
  BEFORE prior-model teardown; on refusal the prior model stays loaded.

RATCHET-RAISE: daemon_lines 4155 -> 4176, traded for the ~21-line G2
source-aware admission block inserted before prior-model teardown.

Verification: workspace build clean; full workspace test suite passes;
qwen3.6:27b (fresh-daemon VMM commit) + qwen3.6-35b-a3b load and generate
coherently; a bad-path load and a vmm+pp>1 load both refuse at admission
with the prior model still generating afterward.
@fivetide
fivetide force-pushed the replan/g2-admission branch from b1740e4 to 86f1ab5 Compare September 2, 2026 13:40
Kaden-Schutt added a commit that referenced this pull request Sep 2, 2026
The deciding seat gets a real shell in a sandboxed checkout of the PR head:
every hiptrx GPU reserved (host flock; lanes take it shared, Fable takes it
exclusive), PR and base branch both built for A/B, every registry artifact
read-only, a wall-clock budget, no credentials and no gh (the script posts).
No fixed route vocabulary: Fable runs what proves the change and records it
in an investigation table with evidence files that are uploaded with the
decision. Fable is claude-fable-5-1 at xhigh.

First real session, PR #682 (G2 admission), 11 min, 58 tool calls: wrote a
daemon driver for the refusal-then-generate scenario no harness expresses,
proved the PR's transactional claim byte-for-byte against master with a
resident single-device and a resident 4-rank EP model, ran pp=1/pp=2 and EP
tp=4 A/B, found a pre-existing EP-MoE generation failure on master, and one
real new refusal (DS4 EP + vmm). Decision: hold on the ratchet hard floor,
with everything else proven.
@Kaden-Schutt Kaden-Schutt added the ratchet-raise Maintainer approval for a declared RATCHET-RAISE in this PR label Sep 2, 2026
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

One regression to fix before this lands, found by running the new admission path on hardware against master:

admission.rs:189 refuses kv_backend=vmm for every EP load. Master serves DeepSeek V4 EP with vmm by design.

Master's load_model_ep_with_kv_mode (crates/hipfire-loader/src/lib.rs) only refuses vmm for the arches whose EP loaders don't support it:

10 if kv_backend_kind == KvBackend::Vmm => Err("KV backend 'vmm' requires tp=1")   // MiniMax
5 | 6 if kv_backend_kind == KvBackend::Vmm => Err("KV backend 'vmm' requires tp=1") // Qwen3.5
9 => load_model_ep_ds4(...)                                                         // DS4: vmm allowed

This PR's admit_source:

if kv_backend == KvBackend::Vmm {
    return Err(format!("KV backend '{}' requires tp=1", kv_backend.as_str()));
}

fires for arch 9 too, so a DS4 EP + vmm load that works on 344e17a2e is refused at admission on this branch. The comment above it says "Mirrors load_model_ep_with_kv_mode's arch_id dispatch + VMM refusal" — it doesn't; master's refusal is per-arch. Fix: gate on matches!(arch_id, 5 | 6 | 10), and keep the DS4 arm vmm-capable.

Two smaller things from the same run, non-blocking:

  • load_model_ep_admitted documents no-reopen but still passes path into the per-arch EP loaders, so the retained SourceAdmission.source is dropped on the EP path. Either thread the handle through or don't claim no-reopen for EP yet.
  • Pre-existing on master, not yours: ornith-1.5:35b-a3b-mq4r loads at tp=4 on 4× gfx1201 but every generate fails with EP arch mismatch (expected dense Qwen TP) — on master and this branch alike. I'll open an issue.

Everything else checked out: with a model resident, four different refusal kinds (pp>1 on a non-Qwen carrier, missing path, vmm on an unsupported carrier, EP on an unsupported arch) leave the prior model loaded and generating byte-identical output to a fresh master load, where master tears it down; pp=1/pp=2 on 0.8B and 27B and EP tp=4 loads are byte-identical to master. The ratchet-raise label is applied for the +21 daemon lines.

admit_source refused kv_backend=vmm for every EP load, so a DeepSeek V4
EP + vmm load that master serves (load_model_ep_with_kv_mode arch 9 arm)
was refused at admission. Gate the refusal on matches!(arch_id, 5|6|10),
mirroring master's per-arch dispatch, and keep the DS4 (9) arm vmm-capable.

Also correct the no-reopen claim on the EP path: the per-arch EP loaders
re-open path per rank, so the retained SourceAdmission.source is dropped
there rather than consumed (single/pp route is unchanged).
@fivetide

fivetide commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the blocking regression and the EP-path doc note.

VMM refusal is per-arch now. admit_source gates the EP vmm refusal on matches!(arch_id, 5 | 6 | 10); DeepSeek V4 (9) stays vmm-capable, mirroring load_model_ep_with_kv_mode's per-arch dispatch. Added a unit test locking the contract (ep_vmm_refusal_is_per_arch: 5|6|10 refuse, 9 admitted, non-vmm never refused).

EP no-reopen claim corrected. load_model_ep_admitted and the SourceAdmission.source field doc now state that the per-arch EP loaders re-open path per rank (the retained source is dropped on the EP path; the single/pp route still consumes it). Kept the doc fix rather than threading the handle through the EP loaders, since those are also reached from the non-admission load_model_ep_with_compressor_cache entry.

Smoke-verified on gfx1151 (single device), rebuilt release daemon:

  • DS4 --tp 2 --kv-backend vmm now passes admission and proceeds into load_model_ep_ds4 (fails later only at init_tp: device id 1 out of range (count=1) — no second device on this box).
  • MiniMax --tp 2 --kv-backend vmm still refuses at admission with KV backend 'vmm' requires tp=1.

The ornith-1.5 EP arch-mismatch note was left untouched as pre-existing.

Requesting another hardware validation run on the multi-GPU setup to confirm the DS4 EP + vmm load that previously refused at admission now serves.

Bjoern Agent added 2 commits September 3, 2026 11:36
Regenerate the hipfire-loader generated map block after the per-arch VMM
refusal fix (admission.rs 274 -> 298 lines, 5 -> 6 tests; lib.rs 4879 ->
4881 lines). Keeps scripts/check-crate-maps.py --check green in CI.
@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate sol prelim

summary: Adds read-only source-aware admission before prior-model teardown: opens and classifies the artifact, resolves its carrier and effective single/pipeline/expert topology, rejects unsupported KV/topology/vision/DFlash combinations, then passes the retained admission into the existing load route. It also raises the daemon source-line policy ceiling from 4155 to 4176; that policy edit does not execute or access external resources.

run_hardware: true
run_hardware_reasons: The Rust changes perform ordinary local model-file reads and existing GPU loader operations; they add no network access, credential reads, process spawning, dependency/toolchain changes, unsafe blocks, or opaque payloads.; The only script change is a plain-text policy threshold increase for daemon line count; it is not executable harness logic.; Hardware is necessary because the new admission rules depend on real artifact arch IDs, tensor inventories, quant types, carrier claims, and effective topology, which unit tests using only classify_vision inputs cannot establish.

routes:

mode tag source why
battery qwen3.6:27b bucket bucket load
battery ornith-1.5:35b-a3b-mq4r bucket bucket load
battery lfm2.5:1.2b bucket bucket load
battery qwen3.8:27b-mq4-xt bucket bucket load

unavailable_routes:

(none)

claim_assessment: The author claims all supported artifacts still load, fresh VMM load no longer self-refuses, EP remains coherent, and admission failures preserve the prior model. The available batteries can prove successful real-artifact classification and generation for dense Qwen, Qwen3.8, LFM2, and EP Ornith. A chain can exercise continued usability, but the specific bad-path and contradictory vmm+pp refusal-preservation claims are proven only if the hardware evidence includes those explicit failed-load transitions followed by successful generation from the prior model.

@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate evidence — 2 lane(s) — verdict fail

lane hiptrx (gfx1201)

hw-gate evidence

field value
base 8cd15a62b593e950b6fe0082b88bb79d122d1dc2
head 95e0e654e59a3892777a6fc73894006a576c6e43
buckets load
host gfx gfx1201
host rocm 7.15.26333-0000000
device 3
runner hiptrx
daemon_md5 8afc9f1c778c317ffce80679f22bccdf
hipfire_md5 3ea306ec937a936fabeb526e1729cf6d
build_seconds 6.515400648117065
verdict fail
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1

battery — exit 1 seconds 198.1 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 132 62 1.8 14.1 False False False True
battery 1 stop 30 0 78 68 451.6 31.1 False False False True
battery 2 stop 31 0 19 16 484.6 31.3 False False False True
battery 3 stop 47 0 31 8 528.6 31.2 False False False True
battery 4 stop 47 0 17 7 532.9 31.2 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
            
    return s[max_start:max_start + max_length]
```
qwen3.6:27b battery turn 1
The sky appears blue during the day because sunlight scatters more effectively in shorter blue wavelengths as it interacts with Earth's atmosphere. At sunset, the sun is lower in the sky, meaning its light must travel through a thicker layer of atmosphere to reach your eyes. This extended path scatters away most of the blue light, allowing the longer red and orange wavelengths to dominate the visual field.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
17 + 26 = 43

Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1

battery — exit 1 seconds 20.1 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 134 64 23.1 7.6 False False False True
battery 1 stop 30 0 87 77 105.8 7.4 False False False True
battery 2 stop 31 0 16 13 110.1 10.8 False False False True
battery 3 stop 47 0 22 6 160.2 9.9 False False False True
battery 4 stop 47 0 17 7 158.8 7.4 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring(s):
    if not s:
        return ""
    start = 0
    max_start = 0
    max_len = 0
    char_index = {}
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_len:
            max_len = end - start + 1
            max_start = start
    return s[max_start:max_start + max_len]
```
ornith-1.5:35b-a3b-mq4r battery turn 1
The sky appears blue during the day because sunlight interacts with particles in the atmosphere, and shorter blue wavelengths scatter more easily than longer wavelengths, spreading blue light across the sky. At sunset, the sun sits lower on the horizon, so its light travels through more atmosphere, causing most of the blue light to scatter away before reaching your eyes. This leaves the longer red and orange wavelengths to dominate, painting the sky in warm sunset colors.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the River Seine runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{"name": "Alice", "age": 34, "city": "Lisbon"}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1

battery — exit 1 seconds 20.1 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 120 58 None 145.1 False False False True
battery 1 stop 0 0 68 59 None 145.92 False False False True
battery 2 stop 0 0 15 13 None 145.63 False False False True
battery 3 stop 0 0 26 8 None 146.07 False False False True
battery 4 stop 0 0 25 13 None 146.2 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_sub(str):
    seen = {}
    start = 0
    max_len = 0
    max_sub = ""
    
    for i, char in enumerate(str):
        if char in seen and start <= seen[char]:
            start = seen[char] + 1
        else:
            if i - start + 1 > max_len:
                max_len = i - start + 1
                max_sub = str[start:i+1]
        seen[char] = i
    return max_sub
```
lfm2.5:1.2b battery turn 1
The sky appears blue during the day because blue light from the sun is scattered in all directions by small air molecules—a phenomenon called Rayleigh scattering. At sunset, the sunlight travels through more atmosphere, so the red wavelengths are diffracted less and reach your eyes directly. This shift in color explains why the sky turns reddish hues during twilight.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the Seine River runs through it.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
Let's calculate the sum: 17 + 26.  
Adding together gives 43.  
Answer: 43

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1

battery — exit 1 seconds 20.1 status fail

reason: harness exit 1

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 133 62 26.7 10.6 False False False True
battery 1 stop 30 0 125 107 36.6 27.9 False False False True
battery 2 stop 31 0 19 16 38.1 9.7 False False False True
battery 3 stop 47 0 17 1 55.9 54.2 False False False True
battery 4 stop 47 0 9 3 56.6 5.1 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    start = 0
    max_length = 0
    max_start = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.8:27b-mq4-xt battery turn 1
Sunlight contains all colors of the visible spectrum, but when it enters the atmosphere, the shorter blue wavelengths scatter off gas molecules much more efficiently than longer wavelengths, a process known as Rayleigh scattering. This scattered blue light is dispersed throughout the sky, making it appear blue to our eyes during midday when sunlight travels through a relatively thin layer of atmosphere. At sunset, the sun is low on the horizon, forcing sunlight to pass through a significantly thicker layer of atmosphere, which scatters away almost all the blue light before it reaches us and allows the longer, less-scattered red and orange wavelengths to dominate the view.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.8:27b-mq4-xt battery turn 3
{"name":"Alice","age":34,"city":"Lisbon"}
qwen3.8:27b-mq4-xt battery turn 4
43

Answer: 43

kernel

not run

lane hipx (gfx1100)

hw-gate evidence

field value
base 8cd15a62b593e950b6fe0082b88bb79d122d1dc2
head 95e0e654e59a3892777a6fc73894006a576c6e43
buckets load
host gfx gfx1100
host rocm 7.15.26333-0000000
device 0
runner hipx
daemon_md5 8afc9f1c778c317ffce80679f22bccdf
hipfire_md5 3ea306ec937a936fabeb526e1729cf6d
build_seconds 6.459684610366821
verdict pass
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 32.6 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 133 62 3.2 17.4 False False False True
battery 1 stop 30 0 82 70 441.2 50.3 False False False True
battery 2 stop 31 0 19 16 454.3 50.4 False False False True
battery 3 stop 47 0 31 8 464.3 50.3 False False False True
battery 4 stop 47 0 6 2 469.4 50.5 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    start = 0
    max_len = 0
    max_start = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_len:
            max_len = end - start + 1
            max_start = start
            
    return s[max_start:max_start + max_len]
```
qwen3.6:27b battery turn 1
During the day, sunlight passes through a thinner layer of atmosphere, causing shorter blue wavelengths to scatter more widely and dominate the sky's appearance. At sunset, light travels through a thicker portion of the atmosphere, which scatters away most of the blue and green light before it reaches your eyes. This process leaves primarily the longer red and orange wavelengths to pass through, creating the characteristic reddish hues of sunset.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 38.2 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 142 66 391.2 15.6 False False False True
battery 1 stop 30 0 91 82 473.1 92.7 False False False True
battery 2 stop 31 0 16 13 498.5 107.9 False False False True
battery 3 stop 47 0 31 8 723.2 113.8 False False False True
battery 4 stop 47 0 17 7 725.8 116.1 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring(s):
    if not s:
        return ""
    
    start = 0
    max_start = 0
    max_length = 0
    char_index = {}
    
    for end in range(len(s)):
        char = s[end]
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
ornith-1.5:35b-a3b-mq4r battery turn 1
The sky appears blue during the day because sunlight interacts with gases in the atmosphere through a process called Rayleigh scattering, where shorter blue wavelengths scatter much more readily than longer wavelengths. This scattered blue light then spreads across the sky, reaching our eyes from all directions. At sunset, sunlight travels through more of the atmosphere at a lower angle, allowing the blue light to be scattered away entirely and leaving only the longer red and orange wavelengths to reach your eyes.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the Seine River runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 8.0 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 103 47 None 229.4 False False False True
battery 1 stop 0 0 73 65 None 229.56 False False False True
battery 2 stop 0 0 15 13 None 230.77 False False False True
battery 3 stop 0 0 26 8 None 230.09 False False False True
battery 4 stop 0 0 20 11 None 229.89 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_substring(s):
    char_index = {}
    start = 0
    longest = ""

    for i, char in enumerate(s):
        if char in char_index and char_index[char] > start:
            start = char_index[char] + 1
        char_index[char] = i
        if i - start + 1 > len(longest):
            longest = s[start:i+1]

    return longest
```
lfm2.5:1.2b battery turn 1
During the day, the sky appears blue because the blue light from sunlight is scattered more efficiently by the atmosphere due to the shorter wavelength of blue. At sunset, the sun is lower in the sky, so the red wavelengths of light, which travel longer distances, dominate the light reaching our eyes. This shift in wavelength and scattering patterns gives the sun a reddish hue.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the Seine River runs through it.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
Let's calculate the sum: 17 + 26 equals 43.  
Answer: 43

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 23.8 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 143 67 286.2 28.7 False False False True
battery 1 stop 30 0 87 71 422.5 49.0 False False False True
battery 2 stop 31 0 16 13 432.3 49.3 False False False True
battery 3 stop 47 0 31 8 478.7 49.0 False False False True
battery 4 stop 47 0 17 7 479.3 49.2 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    if not s:
        return ""
    
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.8:27b-mq4-xt battery turn 1
During the day, shorter-wavelength blue light is scattered more effectively by gas molecules in the atmosphere than other colors, a phenomenon known as Rayleigh scattering, causing the sky to appear blue. At sunset, sunlight travels through a much thicker layer of the atmosphere, scattering away most of the shorter blue and green wavelengths before reaching an observer's eyes. This leaves the longer-wavelength red and orange light to dominate the sky's color.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the Seine River runs through it.
qwen3.8:27b-mq4-xt battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.8:27b-mq4-xt battery turn 4
17 + 26 = 43

Answer: 43

kernel

not run

@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate sol verdict

{
  "claim_verdict": "not-exercised",
  "confidence": 0.97,
  "coverage": {
    "gaps": [
      "All four gfx1201 batteries exited 1 and are fixture failures, despite their decoded turns being coherent.",
      "No chain result was produced, so reordered teardown and continued prior-model usability were not exercised.",
      "Neither an invalid-path transition nor the contradictory vmm+pp transition was followed by generation from the prior model.",
      "The Ornith runs used a single configured device and therefore do not establish expert-parallel rank-local reopen or multi-GPU topology behavior.",
      "The daemon line-threshold policy change requires human review."
    ],
    "surfaces_evidenced": [
      "load"
    ],
    "surfaces_touched": [
      "load",
      "policy"
    ]
  },
  "decision": "block",
  "eyeball": [
    "All reported Qwen3.6, Qwen3.8, LFM2.5, and Ornith turns were coherent, answered their prompts, contained every expected substring, and showed no attractor, empty response, runaway generation, or leaked special token.",
    "The gfx1201 Qwen3.6 first turn incurred 18879 ms prefill and all gfx1201 batteries terminated with harness exit 1; inspect the harness logs to determine why otherwise-correct rows were marked failed."
  ],
  "phase": "verdict",
  "rationale": "The gfx1100 batteries show successful coherent loading and generation for all four artifacts, but the gate rules require a block for any fixture failure, and every gfx1201 fixture failed. More importantly, the central transactional claim around crates/hipfire-daemon/src/main.rs:1566 was not tested: there is no failed-load-then-generate transition or chain evidence. The EP behavior introduced at crates/hipfire-loader/src/lib.rs:2930 was also not demonstrated on a multi-device configuration. The evidence therefore neither proves nor disproves the full author claim.",
  "regressions": []
}

Floor: hard=['hw_run_result=failure', "evidence verdict='fail'", 'policy_paths: scripts/leanup-thresholds.txt'] soft=["coverage_gaps: ['All four gfx1201 batteries exited 1 and are fixture failures, despite their decoded turns being coherent.', 'No chain result was produced, so reordered teardown and continued prior-model usability were not exercised.', 'Neither an invalid-path transition nor the contradictory vmm+pp transition was followed by generation from the prior model.', 'The Ornith runs used a single configured device and therefore do not establish expert-parallel rank-local reopen or multi-GPU topology behavior.', 'The daemon line-threshold policy change requires human review.']"] model_decision=block final=block

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict block: Mandatory qwen3.8:27b-mq4-xt load coverage failed with harness exit 1 on both gfx1201 and gfx1100; the gate rules require a block for any fixture failure. The successful batteries support ordinary dense, LFM2, and Ornith MoE compatibility, but the central fresh-VMM and refusal-preserves-prior-model claims were not exercised. The policy ceiling change in scripts/leanup-thresholds.txt also independently requires human review.

Kaden-Schutt added a commit that referenced this pull request Sep 4, 2026
Third defect from tonight's ladder, and the one with real teeth: the hardware
lanes execute `pr/scripts/serve_harness.py` — the branch's own copy of the
instrument that judges it.

#682 is a fork PR based on master from before #703, so its harness has no
`ATTRACTOR_MIN_WINDOW` guard. Run 33921475093 ran that stale copy and flagged
the 3-token answer `Answer: 43` as a token attractor on BOTH lanes, exiting 1
on a fixture whose own rows record finish=stop, gen=9, recall=1/1, attractor
false, empty false, runaway false. Two green lanes turned red and the rung
hard-floored on evidence that was correct.

Two consequences, one worse than the other:
- any PR older than a harness fix inherits that harness's bugs, which is a
  false-failure generator that gets worse as the harness improves
- a branch can weaken its own oracle, which is the part that must never be
  possible in a gate

Fixtures, registry, prompts and review.py already resolve against the gate
checkout for exactly this reason (see the `_gate_root` comment above the
battery-prompt resolution). serve_harness.py and redline_daemon_harness.py now
do too, including their existence preconditions. The PR still supplies the
binaries under test — only the instrument moved.

Test: `test_serve_harness_runs_from_gate_root_not_pr` writes a different
harness into each tree and asserts the gate's is the one invoked. 106/106
hw-gate tests pass.
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Gate run 33921475093 failed both hardware lanes, and it is not this PR's fault — it is a gate defect I have now fixed in #712.

The lanes execute pr/scripts/serve_harness.py, i.e. this branch's copy of the instrument that judges it. This branch is based on master from before #703, so its harness has no ATTRACTOR_MIN_WINDOW guard. It therefore flagged the [reason] turn — which answered Answer: 43, 9 tokens, recall=1/1 — as a token attractor:

[reason]t5  finish=stop  ctx=47  gen=9  (ans 3w)  tau=2.33  recall=1/1 !ATTRACTOR | '43\n\nAnswer: 43'

Every row in both lanes' evidence records attractor:false, empty:false, runaway:false, recall_ok:true with correct content (working Python, correct Rayleigh-scattering answer, Paris/Seine, valid JSON) — the fixture-level exit 1 came purely from the stale heuristic. The other three fixtures (qwen3.6:27b battery+chain, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b) passed on both lanes.

#712 makes serve_harness.py and redline_daemon_harness.py resolve against the gate checkout, the same way fixtures, registry, prompts and review.py already do. Once it lands I will re-dispatch this PR's gate; no change is needed here.

Rebasing onto current master would also dodge it, but that hides a defect that would keep hitting every PR older than a harness fix.

@hipfire-fable

hipfire-fable Bot commented Sep 4, 2026

Copy link
Copy Markdown

announcement: The qwen3.8:27b-mq4-xt failure that blocked this run is not your code: the PR branch's serve_harness.py predates master's ATTRACTOR_MIN_WINDOW short-answer fix, so the coherent 3-token answer '43 / Answer: 43' was scored as a token attractor and the harness exited 1 on both lanes; the same battery passes on your daemon binary with master's harness (exit 0, 5/5 recall, Q8 vmm). Beyond the mandatory routes I exercised the claims directly against the daemon protocol on both your binary and master: six refused loads each followed by a generate on the prior model (yours keeps serving, master says 'no model loaded' for four of them), fresh-daemon VMM plus VMM-to-VMM/contiguous swaps and a DFlash-draft swap, Qwen3.5 pp=2 on two GPUs, and Qwen3.5-MoE EP tp=4 on four GPUs (loads on both, refuses at generate on both — the pre-existing #683 issue). No regression found; static audit confirms the admit_topology default and vision probes mirror every carrier's own refusals. This is a hold rather than a merge because the run carries a leanup ratchet raise (daemon_lines 4155 -> 4176) that needs the maintainer's explicit sign-off — consider hoisting the admission match into a helper so the daemon stops growing — and because the branch needs a rebase onto current master so the gate runs the fixed harness; after that the mandatory routes should pass on their own.

investigation:

| question | route | result | evidence |

|---|---|---|---|

| Is the qwen3.8:27b-mq4-xt harness exit 1 a product regression or a harness artifact? | Read lanes/hw-gate-evidence-{hiptrx,hipx}/hw-gate-logs/qwen3.8-27b-mq4-xt-battery.out; python-imported pr and base serve_harness and evaluated _response_has_attractor('', '43\n\nAnswer: 43'); then cd base && HIP_VISIBLE_DEVICES=0 HIPFIRE_DAEMON_BIN=$HW_GATE_BIN/daemon HIPFIRE_CLI_BIN=$HW_GATE_BIN/hipfire python3 scripts/serve_harness.py --model $HIPFIRE_MODELS_DIR/qwen3.8-27b.mq4-xt --mode battery --max-tokens 256 --thinking off --thinking-effort none --max-think-tokens 0 --home $HIPFIRE_HOME/qwen3.8-battery-masterharness --out ... --prompts-file benchmarks/prompts/hw-gate/serve-battery.json | Both lane .out files end with [reason]t5 ... recall=1/1 !ATTRACTOR \| '43\n\nAnswer: 43' then serve_harness: ... produced a token attractor — a correct, coherent answer. PR harness returns attractor=True for that string, master harness returns False. Rerun on the PR daemon with master's harness: exit=0, attractor=0, recall 5/5, KV cache: Q8 vmm (qwen3.8 tag policy is kv_backend=vmm, so this is also a fresh-daemon VMM commit), t5 = '17 + 26 = 43\n\nAnswer: 43', decode 24-68 tok/s. | DECISION-NOTES.md |

| Does a refused load now leave the prior model usable (the PR's headline claim), and what does master do? | $HW_GATE_EVIDENCE/daemon_probe.py --scenario scenario_refusal.json driving the daemon's native JSON-lines protocol: load qwen3.5:0.8b, generate, then six refused loads (bad path; qwen3.6 vmm+pp=2; lfm2.5 vmm allowlist; lfm2.5 tp=2 EP-arch; qwen3.6 tp=2+pp=2; lfm2.5 pp=2 default admit_topology) each followed by a generate on the prior model and a reload. Run on $HW_GATE_BIN/daemon (device 1) and $HW_GATE_BASE_BIN/daemon (device 2). | PR: all six refusals fire at admission (class=validation) and every subsequent generate returns 'The capital of France is Paris. It is the largest city in France...'; daemon alive, exit 0. Base: after bad-path, vmm+pp2, vmm-allowlist and pp2 refusals the next generate is no model loaded (prior model torn down); only the tp>1 refusals preserve it (deferred EP unload). Refusal strings are identical modulo master's 'load failed: ... GPU: ...' wrapper; error class changes internal -> validation. | refusal-probe-pr.json, refusal-probe-base.json, refusal-probe-AB-summary.txt, *.daemon.stderr |

| Fresh-daemon VMM commit and VMM<->VMM / VMM<->contiguous swaps under the reordered teardown (the #674 self-refusal class), plus DFlash lm-head admission with a real draft. | $HW_GATE_EVIDENCE/daemon_probe.py --scenario scenario_vmmswap.json on $HW_GATE_BIN/daemon device 3: fresh load qwen3.6-27b.mq4 kv_backend=vmm -> gen; swap qwen3.8-27b.mq4-xt vmm -> gen; bad-path and vmm+pp2 refusals -> gen; swap qwen3.5-0.8b contiguous -> gen; swap qwen3.6 vmm with params.draft=qwen36-27b-dflash-mq4.hfq dflash_mode=on -> gen. Also: cd base && HIP_VISIBLE_DEVICES=4 ... serve_harness.py --model qwen3.6-27b.mq4 --mode battery --kv-backend vmm ... (PR binaries). | Every load succeeded (fresh vmm 13.7s, vmm->vmm swap 11.5s, ->contiguous 0.8s, ->vmm+draft 20.8s; stderr: 'KV cache: Asym3 vmm', 'DFlash draft loaded: .../qwen36-27b-dflash-mq4.hfq'), every generate coherent ('The River Seine flows gracefully through the heart of Paris...'), refusals under vmm preserved the loaded model. Battery with --kv-backend vmm: exit 0, 5/5 recall, KV cache: Q8 vmm, decode ~36 tok/s. | vmmswap-probe-pr.json (+ .daemon.stderr), qwen3.6-27b-battery-vmm-prbin.{out,json,serve.log} |

| Does the one carrier with a real pp>1 path (Qwen3.5 HFQ, kept by the admit_topology override) still load and generate on two GPUs, A/B against master? | $HW_GATE_EVIDENCE/daemon_probe.py --scenario scenario_pp2.json: qwen3.5-0.8b pp=2 -> gen, bad-path refusal -> gen, qwen3.6-27b pp=2 -> gen. PR on HIP_VISIBLE_DEVICES=0,1; base on 2,4. | Both binaries: 0.8b pp=2 and 27b pp=2 load and generate coherently; the 0.8b greedy text is byte-identical PR vs base ('The capital of France is Paris. While the capital of the Kingdom of Italy was Rome...'). Under pp=2 the bad-path refusal preserves the model on PR, base answers no model loaded. (Both binaries leak <\|im_end\|> and report finish=None on the pp path — pre-existing, identical on master.) | pp2-probe-pr.json, pp2-probe-base.json (+ .log, .daemon.stderr) |

| Does the EP (tp>1) admission + load_model_ep_admitted dispatch still reach a real multi-GPU load, A/B against master? | cd base && HIP_VISIBLE_DEVICES=0,1,2,4 ... serve_harness.py --model ornith-1.5-35b-a3b.mq4r --mode battery --tp 4 --devices 0,1,2,4 ... with HIPFIRE_DAEMON_BIN=$HW_GATE_BIN (then $HW_GATE_BASE_BIN). tp=2 was tried first. | tp=2: refused 'EP qwen35 MoE requires tp=4' (same string in base lib.rs:3348 / pr lib.rs:3391). tp=4: '[loader] EP load complete: 4 ranks' on BOTH binaries, then every generate fails 'EP arch mismatch (expected dense Qwen TP)' on BOTH (serve.log:197,201). Identical to master — this is the known #683 'loader admits, engine refuses' defect that #687 fixes on staging; not introduced by this PR. | ornith-1.5-35b-a3b-mq4r-battery-ep-tp4-prbin.{out,serve.log}, ...-ep-tp4-basebin.{out,serve.log}, ...-ep-tp2-prbin.{out,serve.log} |

| Static: does the new fail-closed admission refuse anything master loads (pp>1 defaults per carrier, EP arch/VMM gates, vision tower probes, DFlash lm-head check)? | Two read-only scouts over pr/ and base/ crates/hipfire-loader/{lib.rs,carriers.rs,admission.rs}, hipfire-daemon/src/main.rs; spot-checked base carriers.rs:490-507, 780-788, 2253-2261 and pr carriers.rs:539-556, 1538-1577 myself. | Every master carrier load refuses pp>1 for HFQ and Dir (base carriers.rs:173,782,1072,1197,1402,1514,1707,1804,1920,2255) except Qwen35 HFQ (base carriers.rs:503-506 -> load_qwen35_pp), which the PR override keeps (pr carriers.rs:387-400). EP refusal set identical to base lib.rs:2903-2934. probe_vision tensor names and the LFM2 tower-without-config refusal are the same names/condition the carriers use (pr carriers.rs:541-543, 1545-1577). SourceAdmission.has_vision is never consumed. resolve_carrier uses the same REGISTRY/probe as master (base lib.rs:2429-2440). | DECISION-NOTES.md (static section) |

unproven:

  • Qwen3.5-VL and LFM2-VL artifacts (tower-tensor classification path, and the LFM2 tower-without-config admission refusal) — no VL fixture is present under $HIPFIRE_MODELS_DIR; verified only statically against the carriers' own probes. A host with a qwen3.5 VL .hfq and an LFM2-VL .hfq (with and without vision_config metadata) would close this.

  • DeepSeek V4 (arch 9) EP + vmm and MiniMax (arch 10) EP admission — no ds4/minimax fixture on this host; the per-arch ep_vmm_refusal is covered only by the PR's unit test and by static comparison with base lib.rs:2903-2934.

  • The gfx1100 (hipx) lane was not re-run by me; the gate's own hipx run of qwen3.8 failed for the same harness reason (identical '43\n\nAnswer: 43' !ATTRACTOR line) and all other hipx fixtures passed, but the master-harness rerun above is gfx1201 only.

rationale: The hard floor fired on two components. (1) Evidence failure: qwen3.8:27b-mq4-xt battery exit 1 on both lanes. Both lanes' .out files (lanes/hw-gate-evidence-{hiptrx,hipx}/hw-gate-logs/qwen3.8-27b-mq4-xt-battery.out) show all five turns coherent with recall satisfied, and the exit came from the PR checkout's serve_harness _token_attractor (pr:2037) flagging the 3-whitespace-token answer '43\n\nAnswer: 43' (maxfreq 2/3). Master fixed this exact false positive with ATTRACTOR_MIN_WINDOW=8 (base serve_harness.py:2037-2050, citing gate run 33862054891); the PR branch is based on 8cd15a6 and predates it, and the gate runs the harness from the PR tree. The hw-gate.json rows show attractor:false only because the harness raises before writing --out (pr:2675 vs 2738) and stale JSON was picked up. Rerun of the fixture on the PR daemon (md5 8afc9f1c) with master's harness: exit 0, attractor=0, recall 5/5 ($HW_GATE_EVIDENCE/qwen3.8-27b-mq4-xt-battery-prbin-masterharness.out). So Sol's single regression entry is withdrawn. (2) Policy: scripts/leanup-thresholds.txt raises daemon_lines 4155 -> 4176 for a main.rs that grew 4148 -> 4176 (+28) — the admission block is inlined into the load handler rather than hoisted into a helper; that ratchet raise needs the maintainer's label/judgment, which is what keeps this at hold rather than merge-staging. On the behavior the PR actually claims, I closed every one of Sol's coverage gaps on hardware: refused loads (bad path, vmm+pp>1, vmm allowlist, EP arch, tp+pp, default admit_topology) all fire before teardown and the prior model keeps generating on the PR daemon, while the base daemon answers 'no model loaded' after four of the six ($HW_GATE_EVIDENCE/refusal-probe-AB-summary.txt); fresh-daemon VMM commit and VMM<->VMM/contiguous swaps and a swap with an explicit DFlash draft all load and generate coherently (vmmswap-probe-pr.json; qwen3.6-27b-battery-vmm-prbin.out exit 0); Qwen3.5 pp=2 on two GPUs loads and generates byte-identically to master for 0.8b and coherently for 27b (pp2-probe-{pr,base}.json); Qwen3.5-MoE EP tp=4 on four GPUs loads on both binaries and refuses at generate on both ('EP arch mismatch (expected dense Qwen TP)', the pre-existing #683 defect — not this PR's regression). Static audit: no carrier that master loads at pp>1 is refused by the new admit_topology default (only Qwen35 HFQ has a pp path on master, base carriers.rs:503-506, and the override keeps it); probe_vision uses the carriers' own tensor names (pr carriers.rs:541-543, 1545-1577); has_vision is never consumed. One wire change to be aware of: admission refusals now carry class 'validation' with no 'load failed: ... GPU: ...' wrapper where master returned class 'internal'.

hard floor: ['hw_run_result=failure', "evidence verdict='fail'", 'policy_paths: scripts/leanup-thresholds.txt']

soft floor: ["coverage_gaps: ['No explicit fresh-daemon qwen3.6:27b load with kv_backend=vmm exercised the VMM self-refusal fix.', 'The ordinary chain battery did not issue a refused replacement load followed by generation, so prior-model preservation was not exercised.', 'No contradictory vmm+pp=2 refusal or invalid-path replacement probe was recorded.', 'The daemon line-threshold policy change requires human review.']"]

@hipfire-fable hipfire-fable Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The qwen3.8:27b-mq4-xt failure that blocked this run is not your code: the PR branch's serve_harness.py predates master's ATTRACTOR_MIN_WINDOW short-answer fix, so the coherent 3-token answer '43 / Answer: 43' was scored as a token attractor and the harness exited 1 on both lanes; the same battery passes on your daemon binary with master's harness (exit 0, 5/5 recall, Q8 vmm). Beyond the mandatory routes I exercised the claims directly against the daemon protocol on both your binary and master: six refused loads each followed by a generate on the prior model (yours keeps serving, master says 'no model loaded' for four of them), fresh-daemon VMM plus VMM-to-VMM/contiguous swaps and a DFlash-draft swap, Qwen3.5 pp=2 on two GPUs, and Qwen3.5-MoE EP tp=4 on four GPUs (loads on both, refuses at generate on both — the pre-existing #683 issue). No regression found; static audit confirms the admit_topology default and vision probes mirror every carrier's own refusals. This is a hold rather than a merge because the run carries a leanup ratchet raise (daemon_lines 4155 -> 4176) that needs the maintainer's explicit sign-off — consider hoisting the admission match into a helper so the daemon stops growing — and because the branch needs a rebase onto current master so the gate runs the fixed harness; after that the mandatory routes should pass on their own. The hard floor fired on two components. (1) Evidence failure: qwen3.8:27b-mq4-xt battery exit 1 on both lanes. Both lanes' .out files (lanes/hw-gate-evidence-{hiptrx,hipx}/hw-gate-logs/qwen3.8-27b-mq4-xt-battery.out) show all five turns coherent with recall satisfied, and the exit came from the PR checkout's serve_harness _token_attractor (pr:2037) flagging the 3-whitespace-token answer '43\n\nAnswer: 43' (maxfreq 2/3). Master fixed this exact false positive with ATTRACTOR_MIN_WINDOW=8 (base serve_harness.py:2037-2050, citing gate run 33862054891); the PR branch is based on 8cd15a6 and predates it, and the gate runs the harness from the PR tree. The hw-gate.json rows show attractor:false only because the harness raises before writing --out (pr:2675 vs 2738) and stale JSON was picked up. Rerun of the fixture on the PR daemon (md5 8afc9f1c) with master's harness: exit 0, attractor=0, recall 5/5 ($HW_GATE_EVIDENCE/qwen3.8-27b-mq4-xt-battery-prbin-masterharness.out). So Sol's single regression entry is withdrawn. (2) Policy: scripts/leanup-thresholds.txt raises daemon_lines 4155 -> 4176 for a main.rs that grew 4148 -> 4176 (+28) — the admission block is inlined into the load handler rather than hoisted into a helper; that ratchet raise needs the maintainer's label/judgment, which is what keeps this at hold rather than merge-staging. On the behavior the PR actually claims, I closed every one of Sol's coverage gaps on hardware: refused loads (bad path, vmm+pp>1, vmm allowlist, EP arch, tp+pp, default admit_topology) all fire before teardown and the prior model keeps generating on the PR daemon, while the base daemon answers 'no model loaded' after four of the six ($HW_GATE_EVIDENCE/refusal-probe-AB-summary.txt); fresh-daemon VMM commit and VMM<->VMM/contiguous swaps and a swap with an explicit DFlash draft all load and generate coherently (vmmswap-probe-pr.json; qwen3.6-27b-battery-vmm-prbin.out exit 0); Qwen3.5 pp=2 on two GPUs loads and generates byte-identically to master for 0.8b and coherently for 27b (pp2-probe-{pr,base}.json); Qwen3.5-MoE EP tp=4 on four GPUs loads on both binaries and refuses at generate on both ('EP arch mismatch (expected dense Qwen TP)', the pre-existing #683 defect — not this PR's regression). Static audit: no carrier that master loads at pp>1 is refused by the new admit_topology default (only Qwen35 HFQ has a pp path on master, base carriers.rs:503-506, and the override keeps it); probe_vision uses the carriers' own tensor names (pr carriers.rs:541-543, 1545-1577); has_vision is never consumed. One wire change to be aware of: admission refusals now carry class 'validation' with no 'load failed: ... GPU: ...' wrapper where master returned class 'internal'.

@hipfire-fable hipfire-fable Bot added the hw-gate-blocked hw-gate reviewer blocked on evidence label Sep 4, 2026

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict block: The gfx1100 batteries show successful coherent loading and generation for all four artifacts, but the gate rules require a block for any fixture failure, and every gfx1201 fixture failed. More importantly, the central transactional claim around crates/hipfire-daemon/src/main.rs:1566 was not tested: there is no failed-load-then-generate transition or chain evidence. The EP behavior introduced at crates/hipfire-loader/src/lib.rs:2930 was also not demonstrated on a multi-device configuration. The evidence therefore neither proves nor disproves the full author claim.

Kaden-Schutt added a commit that referenced this pull request Sep 4, 2026
…fy and admit before teardown) to staging

Fork PR (fivetide), so it is staged in this repo as staging/pr-682: PR head 95e0e65 merged with beta at a9f4ca8. That merge had a REAL semantic conflict in crates/hipfire-loader/src/lib.rs between #687's ep_admission and #682's admit_source; resolved by keeping #682's classify-once split (load_model_ep_admitted) and #687's shared ep_unsupported_arch_message(id) as the backstop arm so the message cannot drift.

The resolution is proven on hardware (hiptrx gfx1201, daemon md5 99457c71e4e400793d3efede7278ac3f), not just compiled:
- battery on the canonical qwen3.8-27b.mq4-xt (mq4v2) trunk: 5/5 turns, attractor 0, empty 0, runaway 0, recall satisfied on every turn, avg decode 29.5 tok/s
- LFM2 (arch_id=11) --tp 2 refuses with exactly the shared constructor's text, 'EP not supported for arch_id=11 (expected 5|6 for Qwen3.5, 9 for DeepSeek V4 or 10 for MiniMax)', at [validation retryable=false rolled_back=false] with the serve process still alive
- ornith-1.5-35b-a3b (arch_id=6) --tp 2 refuses through #687's rule at 32548 MB free / 32624 MB total, i.e. #682's restructure did NOT bypass the refusal that #687 added - the precise risk of that conflict
- 48 tests pass in hipfire-loader + hipfire-daemon

Fable's investigation (run 33921475093) closed every one of Sol's coverage gaps on hardware: six refusal classes all fire before teardown with the prior model still generating, while the BASE daemon answers 'no model loaded' after four of the six.

Two floors, both human calls, both made: scripts/leanup-thresholds.txt is a policy path, and the ratchet raise daemon_lines <= 4155 -> 4176 is accepted - the logic landed in hipfire_loader::admission and what grew in the daemon is the call site plus emit_uncorrelated_error boilerplate that must live at the protocol boundary. If that ceiling is hit again, collapse the repeated emit blocks into a local helper first.

Unproven and recorded as such: DS4 (arch 9) and MiniMax (arch 10) EP admission, and Qwen3.5-VL / LFM2-VL tower classification - no such artifact exists on either lane.
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Landed on beta as 4d0e0a1b4. This is a fork PR, so it could not be pushed to; it is staged in the main repo as staging/pr-682 — head 95e0e654e merged with beta at a9f4ca860.

That merge had a real semantic conflict, not a generated-map one: #687's ep_admission (now on beta) versus this PR's admit_source, both in crates/hipfire-loader/src/lib.rs. Resolved by keeping your classify-once split (load_model_ep_admitted) and #687's shared ep_unsupported_arch_message(id) for the backstop arm, so the two refusal messages cannot drift apart.

Because that resolution is mine and no gate run covered it, I proved it on hardware (hiptrx gfx1201, daemon md5 99457c71e4e4):

Fable's earlier investigation (run 33921475093) closed every coverage gap Sol raised: six refusal classes all fire before teardown with the prior model still generating, while the base daemon answers no model loaded after four of the six. That is the regression this PR removes, and it is worth more than the diff review.

Two floors, both human calls, both made. scripts/leanup-thresholds.txt is a policy path, and the ratchet raise daemon_lines <= 4155 → 4176 is accepted: the logic did land in hipfire_loader::admission, and what grew in the daemon is the call site plus emit_uncorrelated_error boilerplate that has to live at the protocol boundary. If that ceiling is hit again, collapse the repeated emit blocks into a local helper before raising it further.

Recorded as unproven rather than claimed: DS4 (arch 9) / MiniMax (arch 10) EP admission and the Qwen3.5-VL / LFM2-VL tower classification — no such artifact exists on either lane.

Closing as landed via staging; the commits keep your authorship.

Kaden-Schutt added a commit that referenced this pull request Sep 5, 2026
…judged

#702 was blocked tonight by #682's verdict.

Its own lanes were 8/8 pass (run 33929015500, head 87233ea, evidence
`verdict: pass`), yet the run published a decision carrying
`hw_run_result=failure`, `evidence verdict='fail'`, and an announcement about
"six refused loads ... master says 'no model loaded' for four of them" -- which
is #682's source-aware admission work, not a DFlash kernel PR.

Cause: the runner workspace is reused and `upload-artifact` runs
`if: always()`. #702's decide phase failed before writing its own
decision.json, so the file left behind by the previous run on that runner --
#682's re-gate -- was uploaded as `hw-gate-decision` for #702, and the status
job read it and blocked the PR. #705 fixed the same hazard for
`fable-evidence/` and `fable-home/`; decision.json was missed, and it is worse,
because that file is the gate's verdict rather than an input to it.

Two changes, because cleaning is necessary but not sufficient:

1. The decide step removes a stale `decision.json` alongside the evidence dirs,
   so the common case cannot arise.
2. review.py records `base` and `head` in decision.json, and the status job
   refuses a decision whose `head` is not this run's head:
   "decision artifact is for <sha> but this run is <sha> -- stale decision.json
   from a reused workspace; re-run the gate". An artifact from another commit is
   a gate malfunction, not a verdict, so it fails as one instead of being
   obeyed. Artifacts predating this field warn rather than fail, so an in-flight
   run does not break on merge.

The lane evidence already carried base/head for exactly this reason
(hw-gate.json records both); the decision did not.

Test: `test_decision_records_the_commit_it_judged` asserts both fields match
the commit under review. 107/107 hw-gate tests pass.
Kaden-Schutt added a commit that referenced this pull request Sep 5, 2026
fix(hw-gate): a stale decision.json blocked #702 with #682's verdict
ghazni101 pushed a commit to ghazni101/hipfire that referenced this pull request Sep 5, 2026
Every rung of the 2026-09-04 ladder hit the same 409 on the staging merge:
`crates/*/map.md` carries a `<!-- crate-map:generated -->` block that both
branches regenerate, so any two PRs touching the same crate conflict there
while their real code merges cleanly. warpfront#689, warpfront#690, warpfront#691, warpfront#686, warpfront#687, warpfront#688 and
warpfront#682 all needed the same three manual steps -- merge staging in, regenerate the
block with scripts/check-crate-maps.py, merge -- six of them tonight. A gate
that decides merge-staging and then holds on a generated file is asking a human
to run a script, which is not review.

On a 409 the decide phase now retries locally: merge staging into the PR head,
and if the conflicted set is generated maps only, re-run check-crate-maps.py
for those crates, commit, and merge the result.

The retry is deliberately narrow, because auto-resolving conflicts is exactly
where a gate can do damage:
- if ANY conflicted path is not a `map.md`, it declines and the hold stands
  with the offending paths named -- a real code conflict must reach a human
- it regenerates rather than picking a side, so the committed block is what the
  tree actually generates, not whichever branch won
- a failed regeneration, a git error, or a timeout all decline rather than
  force

Test: `test_generated_map_retry_refuses_real_code_conflicts` builds a real repo
with a conflicting `.rs` and asserts the retry returns no merge SHA and names
the file. The guard is the part worth pinning; the happy path is exercised by
the ladder itself.

122/122 hw-gate tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hw-gate-blocked hw-gate reviewer blocked on evidence ratchet-raise Maintainer approval for a declared RATCHET-RAISE in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants