fix(hw-gate): drive DFlash in the gate; stop refusing user-path CLI diffs - #718
Conversation
hw-gate sol prelimsummary: Changes hardware-gate policy and orchestration: load/serve buckets gain DFlash variants, those variants translate to existing battery/chain harness modes with forced speculation and an explicit locally selected draft, absent draft candidates produce recorded skips, and skipped modes are excluded from fixture aggregation. It also narrows Sol's filesystem-refusal policy for explicit user-supplied CLI paths. run_hardware: true routes:
unavailable_routes: (none) claim_assessment: The author claims the new routes force DFlash, choose a lane-local draft, and preserve a neutral skip state. Unit-test counts do not prove that. Evidence must show the selected draft is model-compatible, the daemon actually enters speculation, decoded battery/chain turns remain coherent, and a lane with no draft is reported as uncovered without allowing required DFlash coverage to disappear from the overall gate verdict. questions_for_author:
|
hw-gate evidence — 2 lane(s) — verdict passlane hiptrx (gfx1201)hw-gate evidence
fixturesqwen3.6:27bsource: sol sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 31.3 status pass
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4qwen3.8:27b-mq4-xtsource: sol sha256_ok: ✅ size_ok: ✅ status: pass reason: chain — exit 0 seconds 35.3 status pass
qwen3.8:27b-mq4-xt chain turn 0qwen3.8:27b-mq4-xt chain turn 1qwen3.8:27b-mq4-xt chain turn 2qwen3.8:27b-mq4-xt chain turn 3qwen3.8:27b-mq4-xt chain turn 4kernelnot run lane hipx (gfx1100)hw-gate evidence
fixturesqwen3.6:27bsource: sol sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 28.5 status pass
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4qwen3.8:27b-mq4-xtsource: sol sha256_ok: ✅ size_ok: ✅ status: pass reason: chain — exit 0 seconds 42.3 status pass
qwen3.8:27b-mq4-xt chain turn 0qwen3.8:27b-mq4-xt chain turn 1qwen3.8:27b-mq4-xt chain turn 2qwen3.8:27b-mq4-xt chain turn 3qwen3.8:27b-mq4-xt chain turn 4kernelnot run |
hw-gate sol verdict{
"claim_verdict": "not-exercised",
"confidence": 0.98,
"coverage": {
"gaps": [
"The evidence contains only plain battery and chain modes; neither battery-dflash nor chain-dflash ran.",
"No explicit draft was selected, loaded, or shown to enter speculation.",
"Cross-version qwen3.6/qwen3.8 draft compatibility was not exercised.",
"The skip aggregation path was not exercised on hardware, and run.py:542 still permits all declared draft candidates to be absent without failing the fixture.",
"The changed Sol filesystem policy is a policy-file change requiring human review."
],
"surfaces_evidenced": [
"load",
"serve"
],
"surfaces_touched": [
"policy",
"load",
"serve",
"speculative-decode",
"harness"
]
},
"decision": "needs-human",
"eyeball": [
"All five qwen3.6 battery responses on gfx1201 and gfx1100 are coherent, answer their prompts, contain required substrings, and show no attractor, empty output, runaway, or special-token leakage.",
"All five qwen3.8 chain responses on gfx1201 and gfx1100 are coherent; the merge function, 210-mile calculation, seasons explanation, lighthouse story, and coding-practices list answer their prompts without degeneration.",
"A human should inspect scripts/hw-gate/run.py:542-548 because absence of every explicitly declared draft produces skip and is excluded from the verdict, allowing a selected DFlash route to contribute no evidence."
],
"phase": "verdict",
"rationale": "The ordinary qwen3.6 battery and qwen3.8 chain routes passed coherently on both gfx1201 and gfx1100, but hw-gate.json contains no *-dflash mode at all. It therefore proves only the existing AR load/serve paths, not the PR's central forced-DFlash routing, lane-local draft selection, cross-version pairing, or skip semantics. Because scripts/hw-gate/fixtures.json, run.py, and sol.md change gate policy and the relevant behavior was not exercised, this requires human review rather than a greenlight.",
"regressions": []
}Floor: hard=['policy_paths: scripts/hw-gate/fixtures.json,scripts/hw-gate/run.py,scripts/hw-gate/sol.md,scripts/hw-gate/tests/test_run.py'] soft=["coverage_gaps: ['The evidence contains only plain battery and chain modes; neither battery-dflash nor chain-dflash ran.', 'No explicit draft was selected, loaded, or shown to enter speculation.', 'Cross-version qwen3.6/qwen3.8 draft compatibility was not exercised.', 'The skip aggregation path was not exercised on hardware, and run.py:542 still permits all declared draft candidates to be absent without failing the fixture.', 'The changed Sol filesystem policy is a policy-file change requiring human review.']", 'model needs-human'] model_decision=needs-human final=needs-human |
There was a problem hiding this comment.
hw-gate sol verdict needs-human: The hardware run succeeded only as a build: buckets and fixtures are empty, so it provides no evidence for the new DFlash routes or skip semantics. The diff is a policy-floor change and therefore requires human review. In scripts/hw-gate/run.py:621-623, removing every skipped result makes all([]) true, allowing an all-skipped fixture to pass; scripts/hw-gate/tests/test_run.py:971 tests a copied expression rather than _run_fixture_harness, so it does not close that gap.
…iffs Two policy gaps this ladder exposed. 1. The gate never ran DFlash. #686 (draft sidecars), #691 (draft ctor rollback), #692 (primer replay) and #702 (dedicated verify kernels) all went through with every lane green while speculation never once executed. #692's DFlash-arm defect -- primer replay systematically missing the most recent assistant body -- was found only because a seat thought to drive twenty turns by hand. That is not a gate. The load bucket now runs `battery-dflash` and the serve bucket `chain-dflash`: the same prompts with `--dflash on` and an explicit `--draft`. `on` rather than `auto` because `auto` silently falls back to AR when the draft is missing, and a route that can pass without speculating proves nothing. The draft is named explicitly because the canonical xt trunk is a symlink out of the models dir, so the daemon's filename auto-match finds nothing and would run AR. `dflash_draft` is a candidate LIST because the lanes hold different drafts: hiptrx has qwen36-27b-dflash-mq4.hfq and no qwen38, hipx has qwen38-27b-dflash-mq4.hfq and no qwen36. A lane speculates with the first candidate it holds; a lane holding none records `skip`. `skip` is neither pass nor fail. The aggregation was `all(status == "pass")`, which would have counted a skip as a fixture failure -- a false negative on evidence the host never had -- while treating it as a pass would claim coverage that did not happen. Skips are recorded and reported, and a genuine failure alongside a skip still fails. Coverage is asymmetric until both hosts hold both drafts. Pulling qwen38-27b-dflash-mq4.hfq to hiptrx and qwen36-27b-dflash-mq4.hfq to hipx (0.92 GB each) makes it symmetric; that is a disk decision, so the evidence says `skip` rather than silently pulling. 2. Sol refused hardware for any diff touching a filesystem path, which caught #689 for adding `--prompt-file` to `hipfire bench` and cost that rung a lane until `hw-run` overrode it. hipfire is a CLI inference engine: users name models, prompts, drafts and sidecars at invocation, and the gate's own harness passes exactly those flags. sol.md now separates whose path it is -- an explicit argument is ordinary product work; credentials, dotfiles, SSH or cloud config, /proc or /sys beyond device enumeration, assembled traversal, or a read whose result leaves the process still warrant refusal. Tests: eight new cases in scripts/hw-gate/tests/test_run.py covering flag translation (battery-dflash -> `--mode battery --dflash on --draft ...`), plain battery never receiving a draft, per-lane draft selection, skip-not-fail with the harness never invoked, chain-dflash keeping its own prompts, the skip-vs-genuine-failure aggregation, and a manifest assertion that the buckets actually carry the routes. 113/113 hw-gate tests pass.
4142d17 to
744aeb3
Compare
There was a problem hiding this comment.
hw-gate sol verdict needs-human: The ordinary qwen3.6 battery and qwen3.8 chain routes passed coherently on both gfx1201 and gfx1100, but hw-gate.json contains no *-dflash mode at all. It therefore proves only the existing AR load/serve paths, not the PR's central forced-DFlash routing, lane-local draft selection, cross-version pairing, or skip semantics. Because scripts/hw-gate/fixtures.json, run.py, and sol.md change gate policy and the relevant behavior was not exercised, this requires human review rather than a greenlight.
…hosts warpfront#718 concluded DFlash coverage was asymmetric -- hiptrx holding a qwen36 draft and hipx a qwen38 one -- and left a lane recording `skip`. That was wrong: it only looked in ~/.hipfire/models. Both hosts carry the whole 3.8 V2 draft ladder under ~/qcal/ladder-v2/drafts: qwen3.8-27b-dflash.mq2v2.hfq 760353792 qwen3.8-27b-dflash.mq3v2.hfq 984978432 qwen3.8-27b-dflash.mq4v2.hfq 1209603072 qwen3.8-27b-dflash.mq5v2.hfq 1434227712 qwen3.8-27b-dflash.mq6v2.hfq 1658852352 byte-identical across hosts, and the mq4v2 draft is the one the canonical fixture identity was measured with: md5 013395583cd0 against target e45d15bfe0c9 (~/qcal/ladder-v2/artifacts/qwen3.8-27b.mq4v2.xt.hfq), giving 157 tokens / 11 cycles / tau 13.1818 / accept 0.8788. Verified on both hosts just now. So there is nothing to pull and no asymmetry -- the manifest simply could not name a draft outside the models dir. - `dflash_draft` candidates are now path-aware: a bare filename still resolves under the models dir, anything path-shaped (or `~`-prefixed) is taken as given. - The xt fixture pins ~/qcal/ladder-v2/drafts/qwen3.8-27b-dflash.mq4v2.hfq with its sha256, so both lanes speculate against the same artifact the identity was measured with, and `chain-dflash`/`battery-dflash` never skip. - The draft is verified like the target: a sha256 mismatch is a hard fail, not a skip. A wrong draft does not fail loudly, it silently changes tau, and tau is exactly the number that gets quoted. - qwen3.6:27b no longer declares a draft. 3.8 mq4v2 supersedes it, and the xt fixture carries the DFlash route on both lanes. Tests: draft outside the models dir resolves and is passed to the harness; a mismatched sha256 fails without ever speculating; the manifest pins the canonical draft and hash. 132/132 hw-gate tests pass.
Two policy gaps this ladder exposed. Both are gate plumbing, no product code.
1 · The gate has never run DFlash
#686 (draft sidecars), #691 (draft ctor rollback), #692 (primer replay) and #702 (dedicated verify kernels) all went through this gate with every lane green while speculation never once executed. #692's DFlash-arm defect — primer replay systematically missing the most recent assistant body — surfaced only because a seat thought to drive twenty turns by hand. That is not a gate.
The load bucket now runs
battery-dflash, the serve bucketchain-dflash: the same prompts with--dflash onand an explicit--draft.on, notauto—autosilently falls back to AR when the draft is missing, and a route that can pass without speculating proves nothing.--draft— the canonical xt trunk is a symlink out of the models dir, so the daemon's filename auto-match finds nothing and would quietly run AR.dflash_draftis a candidate list, because the lanes hold different drafts: hiptrx hasqwen36-27b-dflash-mq4.hfqand no qwen38, hipx hasqwen38-27b-dflash-mq4.hfqand no qwen36.skipis neither pass nor failThis is the part worth reviewing. The aggregation was
all(status == "pass"), so a skipped mode would have counted as a fixture failure — a false negative on evidence the host never had — while treating it as a pass would claim coverage that did not happen. Skips are now recorded, reported in the reason string, and excluded from the verdict; a genuine failure alongside a skip still fails.Coverage is asymmetric until both hosts hold both drafts. Pulling
qwen38-27b-dflash-mq4.hfqto hiptrx andqwen36-27b-dflash-mq4.hfqto hipx (0.92 GB each) makes it symmetric — that is a disk decision, so this reportsskiprather than silently pulling ~2 GB onto your machines.2 · Sol refused hardware for user-path CLI diffs
The rule "filesystem access beyond model/cache/temp paths" caught #689 for adding
--prompt-filetohipfire bench, and cost that rung a hardware lane untilhw-runoverrode it. hipfire is a CLI inference engine: users name models, prompts, drafts and sidecars at invocation, and the gate's own harness passes exactly those flags.sol.mdnow separates whose path it is. An explicit argument is ordinary product work; credentials, dotfiles, SSH or cloud config,/proc//sysbeyond device enumeration, traversal assembled from something other than an argument, or a read whose result leaves the process still warrant refusal.Tests
Eight new cases: flag translation (
battery-dflash→--mode battery --dflash on --draft …), plain battery never receiving a draft, per-lane draft selection, skip-not-fail with the harness never invoked,chain-dflashkeeping its own prompts, the skip-vs-genuine-failure aggregation, and a manifest assertion that the buckets carry the routes. 113/113 hw-gate tests pass.Policy-floor by construction, so it cannot self-merge.