ci: hw-gate — two-seat autonomous review rung (Sol decides hardware, Fable decides staging merges)

[Kaden-Schutt]

Summary

hw-gate: hipfire's autonomous review rung — two model seats, one human owner, hardware evidence on every load/serve/kernel change. Replaces tools.change_gate + the unconfigured agentic-review pipeline as the CI route (deletion of those is a follow-up).

Motivation: #666 (comment) — five PRs merged on static review broke every Qwen3.5-family load. The load fixture set here refuses in 2 s on that tree and passes on master.

Requesting review from @fivetide and @nwoolmer. After reverting the last auto-reviewer I'm not merging a new one on my own read; please push on anything below.

The rung

select ──▶ sol · prelim ──▶ hardware (matrix) ──▶ sol · verdict ──▶ fable · decide ──▶ hw-gate status
            reads diff        hiptrx gfx1201        greenlight /       merge-staging /
            DECIDES whether   hipx   gfx1100        needs-human /      hold / block
            code runs on hw   serve_harness         block              may veto / override Sol
            + which routes    battery / chain       never merges       merges head → beta

• Sol (gpt-5.6-sol, read-only tools in a PR checkout) alone decides whether the PR's code runs on the maintainers' hardware and which routes run: mandatory fixtures for the touched buckets ∪ the author's requested routes ∪ its own. A maintainer's hw-run label can only force a run.
• Hardware drives every route through serve_harness.py (battery; +chain for serve; +Redline parity for kernel) with reasoning off, on both lanes; every decoded turn is posted verbatim. merge_evidence.py combines lanes and treats a missing lane as failure.
• Sol's verdict on diff + evidence: greenlight / needs-human / block, regressions cited by file:line and fixture, and whether the author's claim was proven. Sol never merges or approves.
• Fable (claude-fable-5-1, xhigh) reads everything incl. Sol's verdict, then investigates on the hardware: a real shell in a sandboxed checkout of the PR head, all five hiptrx GPUs reserved (host flock), PR + base built for A/B, every registry artifact read-only, no credentials, no gh, 45-min budget. No fixed route list — it runs what proves the change and returns an investigation table with evidence files. merge-staging / hold / block, may veto or override Sol with a stated reason. Probation: merge-staging merges the head into beta under Fable's identity; master is promoted by a human.
• Hard floor no seat overrides: failed fixture/harness, attractor, policy-file change, unlabelled RATCHET-RAISE, failed staging merge → hold/block. Soft floor (coverage gaps, confidence, Sol's needs-human) is Fable's call.
• Contributor on-ramp: a <!-- hw-gate-request --> JSON block in the PR body names routes and a claim (template ships the skeleton).
• Identities: hipfire-sol[bot] and hipfire-fable[bot] — org-owned GitHub Apps (ids 4803129 / 4803130), installed, secrets set, tokens verified to mint with Contents/PRs/Issues/Checks: write.

What lands

.github/workflows/hw-gate.yml, .github/CODEOWNERS, scripts/hw-gate/{select,run,review,merge_evidence}.py, scripts/hw-gate/{sol,fable}.md, scripts/hw-gate/fixtures.json (qwen3.6:27b, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b, qwen3.8:27b-mq4-xt, sha256-pinned), benchmarks/prompts/hw-gate/, scripts/ratchet-diff.sh + ci.yml (RATCHET-RAISE now also needs the ratchet-raise label), docs/VALIDATION.md, PR template, CONTRIBUTING.md.

Which surface(s) does this touch?

• policy files — the gate itself (hard floor: a human merges this)
• docs / CI / scripts

Evidence

• python3 -m pytest scripts/hw-gate/tests -q → 102 passed (fake omp/gh, no GPU).
• Hardware, gfx1201 (hiptrx device 3), master: 4 fixtures × (battery + chain) = 40 turns, 4/4 pass, sha256 verified, 140 s wall. ornith-1.5:35b-a3b-mq4r benches 205.7 tok/s AR (hipfire bench) vs 139.7 for the bare tag — hence the -mq4r pin.
• The seats reviewed this PR for real (omp, fake gh so nothing posted):
  • Sol prelim: authorized hardware with precise reasons (gate scripts come from base; the request's "docs/CI only" understates the executable changes) and ran the author-requested lfm2.5:1.2b route.
  • Sol verdict: block — three high-severity findings, all correct and all fixed in 702efc0f4: policy-only diffs skipped the seats entirely while the status passed; a failed staging merge kept merge-staging (green with nothing merged); the battery's substring oracle passed a wrong answer ("eight sheep" contained a 9).
  • Fable decide: hold on the hard floor (policy files), and disagreed with Sol in both directions with correct reasoning — noting the 409 path was already fixed in the tree it read and that the select.json it was given was stale. Both true.
• Both lanes on the PR head, merged by merge_evidence.py: hiptrx gfx1201 (daemon f198ed22…) and hipx gfx1100 RX 7900 XTX (daemon a620c1db…), 4 fixtures × (battery + chain) each = 80 turns, 8/8 fixtures pass, every fixture sha256-verified on both hosts. hipx ran from a clean detached git worktree at the PR head (no rsync). The first battery draft tripped on ornith refusing a bare "echo this JSON" prompt on one arch — replaced with a natural extraction task; that is the kind of oracle defect this evidence loop exists to surface.
• Dry run on fivetide's re-lands feat(runtime): add DeviceMesh topology (fixed #673 G1) #681 and feat(loader): source-aware admission — classify and admit before teardown (fixed #674 G2) #682 (fake gh, nothing posted): Sol authorized both, both passed 4 fixtures on both archs; Sol needs-human on each (PP/TP and the refusal path not exercised). Fable: merge-staging on feat(runtime): add DeviceMesh topology (fixed #673 G1) #681 overriding Sol (Gpus.mesh has zero readers; single-device load is the only regressable surface and passed) — and on feat(loader): source-aware admission — classify and admit before teardown (fixed #674 G2) #682, Fable investigated for 11 min / 58 tool calls: wrote a daemon driver for load-A → refused-load-B → generate-from-A, proved the PR's claim byte-for-byte against master with a resident single-device and a resident 4-rank EP model, ran pp=1/pp=2 and EP tp=4 A/B, found a pre-existing EP-MoE generation failure on master, and one real new refusal (DS4 EP + vmm) → hold on the ratchet hard floor with everything else proven. Recovered transcript: the first real PR through the gate also found a run.py prompt-path bug (c66abf3d8).
• The kernel bucket's Redline step is implemented against the harness's real flags/verdict keys but not yet exercised end to end; the first kernel PR is its proof.

After merge (maintainer)

1. Add hw-gate to the required status checks.
2. (Optional) narrow the two apps' installation from "all repositories" to hipfire only.
3. Follow-up PR: delete tools/change_gate, autoresearch/ar/review, .github/agentic-review, .agents/skills/agentic-pr-*.
4. Follow-up: sandbox the hardware lanes in a ROCm-runtime container (Docker is on hiptrx; hipx has none yet) so fork execution stops depending on Sol's risk read.

Architecture-trait change?

No.

[Kaden-Schutt]

@fivetide @nwoolmer — asking for a real look at this one before it lands; I'm not merging a new auto-reviewer on my own read after reverting the last one.

To make it concrete, this exact gate (at 8729d6f1c, seats run under a fake gh so nothing was posted) already reviewed both of fivetide's re-lands today:

• feat(runtime): add DeviceMesh topology (fixed #673 G1) #681 — Sol authorized hardware; 4 fixtures × both archs (gfx1201 + gfx1100) passed; Sol said needs-human (PP/TP not exercised); Fable overrode to merge-staging because Gpus.mesh has no production readers and single-device load — the only regressable surface — passed on both. Merged to master on that basis.
• feat(loader): source-aware admission — classify and admit before teardown (fixed #674 G2) #682 — Sol authorized and added a chain route on its own; Fable then spent 11 min / 58 tool calls on the hardware: wrote a daemon driver for load-A → refused-load-B → generate-from-A, proved the PR's transactional claim byte-for-byte against master with a resident single-device and a resident 4-rank EP model, ran pp=1/pp=2 and EP tp=4 A/B, found EP MoE generate fails after a successful tp=4 load: 'EP arch mismatch (expected dense Qwen TP)' (ornith-1.5:35b-a3b-mq4r, master) #683 (pre-existing on master), and found one real regression (admission.rs:189 refuses vmm for all EP arches; master serves DS4 EP + vmm). Decision: hold — comment is on the PR.

Both dry runs are described in the body with the evidence. What I'd most like eyes on: the trust boundary (pull_request_target + Sol deciding whether PR code runs on my workstation), the hard floor in scripts/hw-gate/review.py, and Fable's sandbox in investigate mode (_build_investigate_env). The rest is plumbing.