feat(loader): source-aware admission — classify and admit before teardown (fixed #674 G2) - #682
feat(loader): source-aware admission — classify and admit before teardown (fixed #674 G2)#682fivetide wants to merge 4 commits into
Conversation
…down (G2) Classify a retained source once and decide one effective topology before any destructive side effect, so a refused load leaves the prior model usable (issue warpfront#666 G2). - admission::admit_source (read-only): open the source, classify arch_id + vision (tower-tensor decides; contract 179a20d), decide the effective topology (single/pp/ep), and refuse no-carrier / ambiguous / VMM allowlist / VMM+pp / carrier-pp / EP-arch / lfm2 vision-no-config / DFlash lm-head quant — all before any GPU/VMM/teardown work. - Carrier::admit_topology (default + qwen35/cohere2moe/maple/gemma4 overrides) mirrors each carrier's load-time pp refusal. - Split the load entries: path wrappers classify-then-load via load_admitted_with_gemma4_drafter / load_model_ep_admitted, which consume the retained SourceAdmission (no re-open, no re-classify). - Reorder the daemon load handler: daemon topology refusals + admission run BEFORE prior-model teardown; on refusal the prior model stays loaded. RATCHET-RAISE: daemon_lines 4155 -> 4176, traded for the ~21-line G2 source-aware admission block inserted before prior-model teardown. Verification: workspace build clean; full workspace test suite passes; qwen3.6:27b (fresh-daemon VMM commit) + qwen3.6-35b-a3b load and generate coherently; a bad-path load and a vmm+pp>1 load both refuse at admission with the prior model still generating afterward.
b1740e4 to
86f1ab5
Compare
The deciding seat gets a real shell in a sandboxed checkout of the PR head: every hiptrx GPU reserved (host flock; lanes take it shared, Fable takes it exclusive), PR and base branch both built for A/B, every registry artifact read-only, a wall-clock budget, no credentials and no gh (the script posts). No fixed route vocabulary: Fable runs what proves the change and records it in an investigation table with evidence files that are uploaded with the decision. Fable is claude-fable-5-1 at xhigh. First real session, PR #682 (G2 admission), 11 min, 58 tool calls: wrote a daemon driver for the refusal-then-generate scenario no harness expresses, proved the PR's transactional claim byte-for-byte against master with a resident single-device and a resident 4-rank EP model, ran pp=1/pp=2 and EP tp=4 A/B, found a pre-existing EP-MoE generation failure on master, and one real new refusal (DS4 EP + vmm). Decision: hold on the ratchet hard floor, with everything else proven.
|
One regression to fix before this lands, found by running the new admission path on hardware against master:
Master's 10 if kv_backend_kind == KvBackend::Vmm => Err("KV backend 'vmm' requires tp=1") // MiniMax
5 | 6 if kv_backend_kind == KvBackend::Vmm => Err("KV backend 'vmm' requires tp=1") // Qwen3.5
9 => load_model_ep_ds4(...) // DS4: vmm allowedThis PR's if kv_backend == KvBackend::Vmm {
return Err(format!("KV backend '{}' requires tp=1", kv_backend.as_str()));
}fires for arch 9 too, so a DS4 EP + vmm load that works on Two smaller things from the same run, non-blocking:
Everything else checked out: with a model resident, four different refusal kinds (pp>1 on a non-Qwen carrier, missing path, vmm on an unsupported carrier, EP on an unsupported arch) leave the prior model loaded and generating byte-identical output to a fresh master load, where master tears it down; pp=1/pp=2 on 0.8B and 27B and EP tp=4 loads are byte-identical to master. The |
admit_source refused kv_backend=vmm for every EP load, so a DeepSeek V4 EP + vmm load that master serves (load_model_ep_with_kv_mode arch 9 arm) was refused at admission. Gate the refusal on matches!(arch_id, 5|6|10), mirroring master's per-arch dispatch, and keep the DS4 (9) arm vmm-capable. Also correct the no-reopen claim on the EP path: the per-arch EP loaders re-open path per rank, so the retained SourceAdmission.source is dropped there rather than consumed (single/pp route is unchanged).
|
Addressed the blocking regression and the EP-path doc note. VMM refusal is per-arch now. EP no-reopen claim corrected. Smoke-verified on gfx1151 (single device), rebuilt release daemon:
The Requesting another hardware validation run on the multi-GPU setup to confirm the DS4 EP + vmm load that previously refused at admission now serves. |
Regenerate the hipfire-loader generated map block after the per-arch VMM refusal fix (admission.rs 274 -> 298 lines, 5 -> 6 tests; lib.rs 4879 -> 4881 lines). Keeps scripts/check-crate-maps.py --check green in CI.
hw-gate sol prelimsummary: Adds read-only source-aware admission before prior-model teardown: opens and classifies the artifact, resolves its carrier and effective single/pipeline/expert topology, rejects unsupported KV/topology/vision/DFlash combinations, then passes the retained admission into the existing load route. It also raises the daemon source-line policy ceiling from 4155 to 4176; that policy edit does not execute or access external resources. run_hardware: true routes:
unavailable_routes: (none) claim_assessment: The author claims all supported artifacts still load, fresh VMM load no longer self-refuses, EP remains coherent, and admission failures preserve the prior model. The available batteries can prove successful real-artifact classification and generation for dense Qwen, Qwen3.8, LFM2, and EP Ornith. A chain can exercise continued usability, but the specific bad-path and contradictory vmm+pp refusal-preservation claims are proven only if the hardware evidence includes those explicit failed-load transitions followed by successful generation from the prior model. |
hw-gate evidence — 2 lane(s) — verdict faillane hiptrx (gfx1201)hw-gate evidence
fixturesqwen3.6:27bsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1 battery — exit 1 seconds 198.1 status failreason: harness exit 1
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4ornith-1.5:35b-a3b-mq4rsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1 battery — exit 1 seconds 20.1 status failreason: harness exit 1
ornith-1.5:35b-a3b-mq4r battery turn 0ornith-1.5:35b-a3b-mq4r battery turn 1ornith-1.5:35b-a3b-mq4r battery turn 2ornith-1.5:35b-a3b-mq4r battery turn 3ornith-1.5:35b-a3b-mq4r battery turn 4lfm2.5:1.2bsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1 battery — exit 1 seconds 20.1 status failreason: harness exit 1
lfm2.5:1.2b battery turn 0lfm2.5:1.2b battery turn 1lfm2.5:1.2b battery turn 2lfm2.5:1.2b battery turn 3lfm2.5:1.2b battery turn 4qwen3.8:27b-mq4-xtsource: bucket sha256_ok: ✅ size_ok: ✅ status: fail reason: battery: harness exit 1 battery — exit 1 seconds 20.1 status failreason: harness exit 1
qwen3.8:27b-mq4-xt battery turn 0qwen3.8:27b-mq4-xt battery turn 1qwen3.8:27b-mq4-xt battery turn 2qwen3.8:27b-mq4-xt battery turn 3qwen3.8:27b-mq4-xt battery turn 4kernelnot run lane hipx (gfx1100)hw-gate evidence
fixturesqwen3.6:27bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 32.6 status pass
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4ornith-1.5:35b-a3b-mq4rsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 38.2 status pass
ornith-1.5:35b-a3b-mq4r battery turn 0ornith-1.5:35b-a3b-mq4r battery turn 1ornith-1.5:35b-a3b-mq4r battery turn 2ornith-1.5:35b-a3b-mq4r battery turn 3ornith-1.5:35b-a3b-mq4r battery turn 4lfm2.5:1.2bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 8.0 status pass
lfm2.5:1.2b battery turn 0lfm2.5:1.2b battery turn 1lfm2.5:1.2b battery turn 2lfm2.5:1.2b battery turn 3lfm2.5:1.2b battery turn 4qwen3.8:27b-mq4-xtsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 23.8 status pass
qwen3.8:27b-mq4-xt battery turn 0qwen3.8:27b-mq4-xt battery turn 1qwen3.8:27b-mq4-xt battery turn 2qwen3.8:27b-mq4-xt battery turn 3qwen3.8:27b-mq4-xt battery turn 4kernelnot run |
hw-gate sol verdict{
"claim_verdict": "not-exercised",
"confidence": 0.97,
"coverage": {
"gaps": [
"All four gfx1201 batteries exited 1 and are fixture failures, despite their decoded turns being coherent.",
"No chain result was produced, so reordered teardown and continued prior-model usability were not exercised.",
"Neither an invalid-path transition nor the contradictory vmm+pp transition was followed by generation from the prior model.",
"The Ornith runs used a single configured device and therefore do not establish expert-parallel rank-local reopen or multi-GPU topology behavior.",
"The daemon line-threshold policy change requires human review."
],
"surfaces_evidenced": [
"load"
],
"surfaces_touched": [
"load",
"policy"
]
},
"decision": "block",
"eyeball": [
"All reported Qwen3.6, Qwen3.8, LFM2.5, and Ornith turns were coherent, answered their prompts, contained every expected substring, and showed no attractor, empty response, runaway generation, or leaked special token.",
"The gfx1201 Qwen3.6 first turn incurred 18879 ms prefill and all gfx1201 batteries terminated with harness exit 1; inspect the harness logs to determine why otherwise-correct rows were marked failed."
],
"phase": "verdict",
"rationale": "The gfx1100 batteries show successful coherent loading and generation for all four artifacts, but the gate rules require a block for any fixture failure, and every gfx1201 fixture failed. More importantly, the central transactional claim around crates/hipfire-daemon/src/main.rs:1566 was not tested: there is no failed-load-then-generate transition or chain evidence. The EP behavior introduced at crates/hipfire-loader/src/lib.rs:2930 was also not demonstrated on a multi-device configuration. The evidence therefore neither proves nor disproves the full author claim.",
"regressions": []
}Floor: hard=['hw_run_result=failure', "evidence verdict='fail'", 'policy_paths: scripts/leanup-thresholds.txt'] soft=["coverage_gaps: ['All four gfx1201 batteries exited 1 and are fixture failures, despite their decoded turns being coherent.', 'No chain result was produced, so reordered teardown and continued prior-model usability were not exercised.', 'Neither an invalid-path transition nor the contradictory vmm+pp transition was followed by generation from the prior model.', 'The Ornith runs used a single configured device and therefore do not establish expert-parallel rank-local reopen or multi-GPU topology behavior.', 'The daemon line-threshold policy change requires human review.']"] model_decision=block final=block |
There was a problem hiding this comment.
hw-gate sol verdict block: Mandatory qwen3.8:27b-mq4-xt load coverage failed with harness exit 1 on both gfx1201 and gfx1100; the gate rules require a block for any fixture failure. The successful batteries support ordinary dense, LFM2, and Ornith MoE compatibility, but the central fresh-VMM and refusal-preserves-prior-model claims were not exercised. The policy ceiling change in scripts/leanup-thresholds.txt also independently requires human review.
Third defect from tonight's ladder, and the one with real teeth: the hardware lanes execute `pr/scripts/serve_harness.py` — the branch's own copy of the instrument that judges it. #682 is a fork PR based on master from before #703, so its harness has no `ATTRACTOR_MIN_WINDOW` guard. Run 33921475093 ran that stale copy and flagged the 3-token answer `Answer: 43` as a token attractor on BOTH lanes, exiting 1 on a fixture whose own rows record finish=stop, gen=9, recall=1/1, attractor false, empty false, runaway false. Two green lanes turned red and the rung hard-floored on evidence that was correct. Two consequences, one worse than the other: - any PR older than a harness fix inherits that harness's bugs, which is a false-failure generator that gets worse as the harness improves - a branch can weaken its own oracle, which is the part that must never be possible in a gate Fixtures, registry, prompts and review.py already resolve against the gate checkout for exactly this reason (see the `_gate_root` comment above the battery-prompt resolution). serve_harness.py and redline_daemon_harness.py now do too, including their existence preconditions. The PR still supplies the binaries under test — only the instrument moved. Test: `test_serve_harness_runs_from_gate_root_not_pr` writes a different harness into each tree and asserts the gate's is the one invoked. 106/106 hw-gate tests pass.
|
Gate run 33921475093 failed both hardware lanes, and it is not this PR's fault — it is a gate defect I have now fixed in #712. The lanes execute Every row in both lanes' evidence records #712 makes Rebasing onto current master would also dodge it, but that hides a defect that would keep hitting every PR older than a harness fix. |
|
announcement: The qwen3.8:27b-mq4-xt failure that blocked this run is not your code: the PR branch's serve_harness.py predates master's ATTRACTOR_MIN_WINDOW short-answer fix, so the coherent 3-token answer '43 / Answer: 43' was scored as a token attractor and the harness exited 1 on both lanes; the same battery passes on your daemon binary with master's harness (exit 0, 5/5 recall, Q8 vmm). Beyond the mandatory routes I exercised the claims directly against the daemon protocol on both your binary and master: six refused loads each followed by a generate on the prior model (yours keeps serving, master says 'no model loaded' for four of them), fresh-daemon VMM plus VMM-to-VMM/contiguous swaps and a DFlash-draft swap, Qwen3.5 pp=2 on two GPUs, and Qwen3.5-MoE EP tp=4 on four GPUs (loads on both, refuses at generate on both — the pre-existing #683 issue). No regression found; static audit confirms the admit_topology default and vision probes mirror every carrier's own refusals. This is a hold rather than a merge because the run carries a leanup ratchet raise (daemon_lines 4155 -> 4176) that needs the maintainer's explicit sign-off — consider hoisting the admission match into a helper so the daemon stops growing — and because the branch needs a rebase onto current master so the gate runs the fixed harness; after that the mandatory routes should pass on their own. investigation: | question | route | result | evidence | |---|---|---|---| | Is the qwen3.8:27b-mq4-xt harness exit 1 a product regression or a harness artifact? | Read lanes/hw-gate-evidence-{hiptrx,hipx}/hw-gate-logs/qwen3.8-27b-mq4-xt-battery.out; python-imported pr and base serve_harness and evaluated | Does a refused load now leave the prior model usable (the PR's headline claim), and what does master do? | $HW_GATE_EVIDENCE/daemon_probe.py --scenario scenario_refusal.json driving the daemon's native JSON-lines protocol: load qwen3.5:0.8b, generate, then six refused loads (bad path; qwen3.6 vmm+pp=2; lfm2.5 vmm allowlist; lfm2.5 tp=2 EP-arch; qwen3.6 tp=2+pp=2; lfm2.5 pp=2 default admit_topology) each followed by a generate on the prior model and a reload. Run on $HW_GATE_BIN/daemon (device 1) and $HW_GATE_BASE_BIN/daemon (device 2). | PR: all six refusals fire at admission (class=validation) and every subsequent generate returns 'The capital of France is Paris. It is the largest city in France...'; daemon alive, exit 0. Base: after bad-path, vmm+pp2, vmm-allowlist and pp2 refusals the next generate is | Fresh-daemon VMM commit and VMM<->VMM / VMM<->contiguous swaps under the reordered teardown (the #674 self-refusal class), plus DFlash lm-head admission with a real draft. | $HW_GATE_EVIDENCE/daemon_probe.py --scenario scenario_vmmswap.json on $HW_GATE_BIN/daemon device 3: fresh load qwen3.6-27b.mq4 kv_backend=vmm -> gen; swap qwen3.8-27b.mq4-xt vmm -> gen; bad-path and vmm+pp2 refusals -> gen; swap qwen3.5-0.8b contiguous -> gen; swap qwen3.6 vmm with params.draft=qwen36-27b-dflash-mq4.hfq dflash_mode=on -> gen. Also: cd base && HIP_VISIBLE_DEVICES=4 ... serve_harness.py --model qwen3.6-27b.mq4 --mode battery --kv-backend vmm ... (PR binaries). | Every load succeeded (fresh vmm 13.7s, vmm->vmm swap 11.5s, ->contiguous 0.8s, ->vmm+draft 20.8s; stderr: 'KV cache: Asym3 vmm', 'DFlash draft loaded: .../qwen36-27b-dflash-mq4.hfq'), every generate coherent ('The River Seine flows gracefully through the heart of Paris...'), refusals under vmm preserved the loaded model. Battery with --kv-backend vmm: exit 0, 5/5 recall, | Does the one carrier with a real pp>1 path (Qwen3.5 HFQ, kept by the admit_topology override) still load and generate on two GPUs, A/B against master? | $HW_GATE_EVIDENCE/daemon_probe.py --scenario scenario_pp2.json: qwen3.5-0.8b pp=2 -> gen, bad-path refusal -> gen, qwen3.6-27b pp=2 -> gen. PR on HIP_VISIBLE_DEVICES=0,1; base on 2,4. | Both binaries: 0.8b pp=2 and 27b pp=2 load and generate coherently; the 0.8b greedy text is byte-identical PR vs base ('The capital of France is Paris. While the capital of the Kingdom of Italy was Rome...'). Under pp=2 the bad-path refusal preserves the model on PR, base answers | Does the EP (tp>1) admission + | Static: does the new fail-closed admission refuse anything master loads (pp>1 defaults per carrier, EP arch/VMM gates, vision tower probes, DFlash lm-head check)? | Two read-only scouts over pr/ and base/ crates/hipfire-loader/{lib.rs,carriers.rs,admission.rs}, hipfire-daemon/src/main.rs; spot-checked base carriers.rs:490-507, 780-788, 2253-2261 and pr carriers.rs:539-556, 1538-1577 myself. | Every master carrier unproven:
rationale: The hard floor fired on two components. (1) Evidence failure: qwen3.8:27b-mq4-xt battery exit 1 on both lanes. Both lanes' .out files (lanes/hw-gate-evidence-{hiptrx,hipx}/hw-gate-logs/qwen3.8-27b-mq4-xt-battery.out) show all five turns coherent with recall satisfied, and the exit came from the PR checkout's serve_harness hard floor: ['hw_run_result=failure', "evidence verdict='fail'", 'policy_paths: scripts/leanup-thresholds.txt'] soft floor: ["coverage_gaps: ['No explicit fresh-daemon qwen3.6:27b load with kv_backend=vmm exercised the VMM self-refusal fix.', 'The ordinary chain battery did not issue a refused replacement load followed by generation, so prior-model preservation was not exercised.', 'No contradictory vmm+pp=2 refusal or invalid-path replacement probe was recorded.', 'The daemon line-threshold policy change requires human review.']"] |
There was a problem hiding this comment.
The qwen3.8:27b-mq4-xt failure that blocked this run is not your code: the PR branch's serve_harness.py predates master's ATTRACTOR_MIN_WINDOW short-answer fix, so the coherent 3-token answer '43 / Answer: 43' was scored as a token attractor and the harness exited 1 on both lanes; the same battery passes on your daemon binary with master's harness (exit 0, 5/5 recall, Q8 vmm). Beyond the mandatory routes I exercised the claims directly against the daemon protocol on both your binary and master: six refused loads each followed by a generate on the prior model (yours keeps serving, master says 'no model loaded' for four of them), fresh-daemon VMM plus VMM-to-VMM/contiguous swaps and a DFlash-draft swap, Qwen3.5 pp=2 on two GPUs, and Qwen3.5-MoE EP tp=4 on four GPUs (loads on both, refuses at generate on both — the pre-existing #683 issue). No regression found; static audit confirms the admit_topology default and vision probes mirror every carrier's own refusals. This is a hold rather than a merge because the run carries a leanup ratchet raise (daemon_lines 4155 -> 4176) that needs the maintainer's explicit sign-off — consider hoisting the admission match into a helper so the daemon stops growing — and because the branch needs a rebase onto current master so the gate runs the fixed harness; after that the mandatory routes should pass on their own. The hard floor fired on two components. (1) Evidence failure: qwen3.8:27b-mq4-xt battery exit 1 on both lanes. Both lanes' .out files (lanes/hw-gate-evidence-{hiptrx,hipx}/hw-gate-logs/qwen3.8-27b-mq4-xt-battery.out) show all five turns coherent with recall satisfied, and the exit came from the PR checkout's serve_harness _token_attractor (pr:2037) flagging the 3-whitespace-token answer '43\n\nAnswer: 43' (maxfreq 2/3). Master fixed this exact false positive with ATTRACTOR_MIN_WINDOW=8 (base serve_harness.py:2037-2050, citing gate run 33862054891); the PR branch is based on 8cd15a6 and predates it, and the gate runs the harness from the PR tree. The hw-gate.json rows show attractor:false only because the harness raises before writing --out (pr:2675 vs 2738) and stale JSON was picked up. Rerun of the fixture on the PR daemon (md5 8afc9f1c) with master's harness: exit 0, attractor=0, recall 5/5 ($HW_GATE_EVIDENCE/qwen3.8-27b-mq4-xt-battery-prbin-masterharness.out). So Sol's single regression entry is withdrawn. (2) Policy: scripts/leanup-thresholds.txt raises daemon_lines 4155 -> 4176 for a main.rs that grew 4148 -> 4176 (+28) — the admission block is inlined into the load handler rather than hoisted into a helper; that ratchet raise needs the maintainer's label/judgment, which is what keeps this at hold rather than merge-staging. On the behavior the PR actually claims, I closed every one of Sol's coverage gaps on hardware: refused loads (bad path, vmm+pp>1, vmm allowlist, EP arch, tp+pp, default admit_topology) all fire before teardown and the prior model keeps generating on the PR daemon, while the base daemon answers 'no model loaded' after four of the six ($HW_GATE_EVIDENCE/refusal-probe-AB-summary.txt); fresh-daemon VMM commit and VMM<->VMM/contiguous swaps and a swap with an explicit DFlash draft all load and generate coherently (vmmswap-probe-pr.json; qwen3.6-27b-battery-vmm-prbin.out exit 0); Qwen3.5 pp=2 on two GPUs loads and generates byte-identically to master for 0.8b and coherently for 27b (pp2-probe-{pr,base}.json); Qwen3.5-MoE EP tp=4 on four GPUs loads on both binaries and refuses at generate on both ('EP arch mismatch (expected dense Qwen TP)', the pre-existing #683 defect — not this PR's regression). Static audit: no carrier that master loads at pp>1 is refused by the new admit_topology default (only Qwen35 HFQ has a pp path on master, base carriers.rs:503-506, and the override keeps it); probe_vision uses the carriers' own tensor names (pr carriers.rs:541-543, 1545-1577); has_vision is never consumed. One wire change to be aware of: admission refusals now carry class 'validation' with no 'load failed: ... GPU: ...' wrapper where master returned class 'internal'.
There was a problem hiding this comment.
hw-gate sol verdict block: The gfx1100 batteries show successful coherent loading and generation for all four artifacts, but the gate rules require a block for any fixture failure, and every gfx1201 fixture failed. More importantly, the central transactional claim around crates/hipfire-daemon/src/main.rs:1566 was not tested: there is no failed-load-then-generate transition or chain evidence. The EP behavior introduced at crates/hipfire-loader/src/lib.rs:2930 was also not demonstrated on a multi-device configuration. The evidence therefore neither proves nor disproves the full author claim.
…fy and admit before teardown) to staging Fork PR (fivetide), so it is staged in this repo as staging/pr-682: PR head 95e0e65 merged with beta at a9f4ca8. That merge had a REAL semantic conflict in crates/hipfire-loader/src/lib.rs between #687's ep_admission and #682's admit_source; resolved by keeping #682's classify-once split (load_model_ep_admitted) and #687's shared ep_unsupported_arch_message(id) as the backstop arm so the message cannot drift. The resolution is proven on hardware (hiptrx gfx1201, daemon md5 99457c71e4e400793d3efede7278ac3f), not just compiled: - battery on the canonical qwen3.8-27b.mq4-xt (mq4v2) trunk: 5/5 turns, attractor 0, empty 0, runaway 0, recall satisfied on every turn, avg decode 29.5 tok/s - LFM2 (arch_id=11) --tp 2 refuses with exactly the shared constructor's text, 'EP not supported for arch_id=11 (expected 5|6 for Qwen3.5, 9 for DeepSeek V4 or 10 for MiniMax)', at [validation retryable=false rolled_back=false] with the serve process still alive - ornith-1.5-35b-a3b (arch_id=6) --tp 2 refuses through #687's rule at 32548 MB free / 32624 MB total, i.e. #682's restructure did NOT bypass the refusal that #687 added - the precise risk of that conflict - 48 tests pass in hipfire-loader + hipfire-daemon Fable's investigation (run 33921475093) closed every one of Sol's coverage gaps on hardware: six refusal classes all fire before teardown with the prior model still generating, while the BASE daemon answers 'no model loaded' after four of the six. Two floors, both human calls, both made: scripts/leanup-thresholds.txt is a policy path, and the ratchet raise daemon_lines <= 4155 -> 4176 is accepted - the logic landed in hipfire_loader::admission and what grew in the daemon is the call site plus emit_uncorrelated_error boilerplate that must live at the protocol boundary. If that ceiling is hit again, collapse the repeated emit blocks into a local helper first. Unproven and recorded as such: DS4 (arch 9) and MiniMax (arch 10) EP admission, and Qwen3.5-VL / LFM2-VL tower classification - no such artifact exists on either lane.
|
Landed on That merge had a real semantic conflict, not a generated-map one: #687's Because that resolution is mine and no gate run covered it, I proved it on hardware (hiptrx gfx1201, daemon md5
Fable's earlier investigation (run 33921475093) closed every coverage gap Sol raised: six refusal classes all fire before teardown with the prior model still generating, while the base daemon answers Two floors, both human calls, both made. Recorded as unproven rather than claimed: DS4 (arch 9) / MiniMax (arch 10) EP admission and the Qwen3.5-VL / LFM2-VL tower classification — no such artifact exists on either lane. Closing as landed via staging; the commits keep your authorship. |
…judged #702 was blocked tonight by #682's verdict. Its own lanes were 8/8 pass (run 33929015500, head 87233ea, evidence `verdict: pass`), yet the run published a decision carrying `hw_run_result=failure`, `evidence verdict='fail'`, and an announcement about "six refused loads ... master says 'no model loaded' for four of them" -- which is #682's source-aware admission work, not a DFlash kernel PR. Cause: the runner workspace is reused and `upload-artifact` runs `if: always()`. #702's decide phase failed before writing its own decision.json, so the file left behind by the previous run on that runner -- #682's re-gate -- was uploaded as `hw-gate-decision` for #702, and the status job read it and blocked the PR. #705 fixed the same hazard for `fable-evidence/` and `fable-home/`; decision.json was missed, and it is worse, because that file is the gate's verdict rather than an input to it. Two changes, because cleaning is necessary but not sufficient: 1. The decide step removes a stale `decision.json` alongside the evidence dirs, so the common case cannot arise. 2. review.py records `base` and `head` in decision.json, and the status job refuses a decision whose `head` is not this run's head: "decision artifact is for <sha> but this run is <sha> -- stale decision.json from a reused workspace; re-run the gate". An artifact from another commit is a gate malfunction, not a verdict, so it fails as one instead of being obeyed. Artifacts predating this field warn rather than fail, so an in-flight run does not break on merge. The lane evidence already carried base/head for exactly this reason (hw-gate.json records both); the decision did not. Test: `test_decision_records_the_commit_it_judged` asserts both fields match the commit under review. 107/107 hw-gate tests pass.
Every rung of the 2026-09-04 ladder hit the same 409 on the staging merge: `crates/*/map.md` carries a `<!-- crate-map:generated -->` block that both branches regenerate, so any two PRs touching the same crate conflict there while their real code merges cleanly. warpfront#689, warpfront#690, warpfront#691, warpfront#686, warpfront#687, warpfront#688 and warpfront#682 all needed the same three manual steps -- merge staging in, regenerate the block with scripts/check-crate-maps.py, merge -- six of them tonight. A gate that decides merge-staging and then holds on a generated file is asking a human to run a script, which is not review. On a 409 the decide phase now retries locally: merge staging into the PR head, and if the conflicted set is generated maps only, re-run check-crate-maps.py for those crates, commit, and merge the result. The retry is deliberately narrow, because auto-resolving conflicts is exactly where a gate can do damage: - if ANY conflicted path is not a `map.md`, it declines and the hold stands with the offending paths named -- a real code conflict must reach a human - it regenerates rather than picking a side, so the committed block is what the tree actually generates, not whichever branch won - a failed regeneration, a git error, or a timeout all decline rather than force Test: `test_generated_map_retry_refuses_real_code_conflicts` builds a real repo with a conflicting `.rs` and asserts the retry returns no merge SHA and names the file. The guard is the part worth pinning; the happy path is exercised by the ladder itself. 122/122 hw-gate tests pass.
Summary
Fixed version of #674 (G2 source-aware admission). #674 was reverted in
a0fca0d6dalongside #668/#673/#676 because it introduced two admission defects that broke production loads:vision_config/model_typeconfig markers, so every Qwen3.5-family text artifact (whose HF config always embedsvision_config) was refused at admission for "missing vision tower".commit_ordinaryre-ran the VMM readiness check after the new model's own VMM tensors existed, counting them as pending teardown and refusing every freshkv_backend=vmmload.This PR delivers the same G2 intent — classify once, admit one effective topology, refuse before destructive side effects — without either defect.
What changed
crates/hipfire-loader/src/admission.rs(new): read-onlyadmit_source— opens the source, classifiesarch_id+ vision, decides the effective topology (Single/Pipeline/Expert), and refuses no-carrier / ambiguous-carrier / VMM-allowlist / VMM+pp / carrier-pp / EP-arch / lfm2 vision-no-config / DFlash lm-head quant — all before any GPU, VMM, or teardown work. Vision is decided by the tower tensor (classify_vision), preserving the remediation contract179a20d7f.Carrier::admit_topology(new trait method, default + qwen35/cohere2moe/maple/gemma4 overrides) — mirrors each carrier's load-timepp>1refusal so a refusal fires at admission.load_model_with_gemma4_drafter/load_model_ep_with_kv_modebecome thin wrappers that classify-then-load viaload_admitted_with_gemma4_drafter/load_model_ep_admitted, which consume the retainedSourceAdmission(no re-open, no re-classify).crates/hipfire-daemon/src/main.rs): daemon topology refusals +admit_sourcenow run before prior-model teardown. On refusal the prior model stays loaded.How this differs from #674
vision_config/model_type)tensor_data(...).is_some())ensure_vmm_ready_for_loadSourceAdmissionconsumed onceVerification
cargo build(workspace): clean. Full workspace test suite: no failures (incl. 5 newadmissioncontract tests).HOME):hipfire run qwen3.6:27b→ 64/64 layers,KV cache: Q8 vmm(fresh-daemon VMM commit), coherentParis.hipfire run qwen3.6-35b-a3b.mq4→ coherent linked-list-reverse output.No such file or directory) and a contradictorykv_backend=vmm + pp=2load both refuse at admission, and a subsequent generate on the prior model returnsdonein both cases (prior model survives refusal).Dependencies / nonclaims
DeviceMesh; G2 admission decides the topology, G5 will bind it togpus.mesh).DimKind::Epclassification is unchanged from master (EP still routes byarch_id5|6|9|10); no new axis, no manifest planning, no family cutover.