fix(ci-e2e): budget queueing apart from running, and stop swallowing failed cancels

[zoroyihan7]

What was wrong

One counter for two different waits. POLL_MAX covered both waiting for a GPU and running the optimizer, so a saturated pool spent the run budget on the queue and then reported not terminal after 13200s — which reads as a hang in the tested commit.

Run 34190618492 spent 166 of its 220 polls in phase=Queued and was failed 54 minutes into a workload that went on to finish successfully 115 minutes later (k8s Job ci-pr-1414-34190618492-a7f21d57-a1: succeeded=1, 171m, crash_count 0). The check never gave a verdict on the code.

The cancel path hid why the pool was saturated. cleanup() discarded the DELETE response, and that response is a 40x. The workload survives, the backend dispatches it hours after its GitHub run ended, and it holds a GPU for a full run nobody is watching.

Of 93 jobs dispatched 2026-09-07 → 09-08, 49 started after their run had already finished, taking 96 of 204 GPU-slot-hours. At 12:19Z all eight GPUs in the pool were held by workloads whose run had ended — none of them belonging to a live check.

What this changes

• Queueing and running get separate budgets and separate verdicts, in both the ci-e2e and forge-e2e dispatchers. A red check now says whether the commit was ever given a GPU.
• A queue timeout cancels unconditionally. There is no on-cluster state to keep for triage when nothing was dispatched, and the workload would otherwise take a GPU for a full run once a slot freed.
• cleanup() checks the status code, retries once, and reports an unreclaimed workload instead of returning success.
• The cancel path is bounded so it can outlive neither the runner's grace period nor a hanging backend: --connect-timeout/--max-time on the DELETE and on post_status, a shorter retry, no retry at all from the signal trap, and the trap disarms itself so a SIGTERM mid-handler cannot re-enter it. Measured against a stubbed hanging backend: 6.4s wall, inside the ~10s SIGINT→SIGKILL window.
• A poll whose curl failed left the phase empty, not Unknown — jq exits 0 printing nothing on empty input, so neither the // default nor the || echo fallback fired. The empty phase fell through the dispatched latch and pinned the run budget on a workload still waiting for a GPU, on the first transient read of the several hundred a run makes.

Verification

No bash test suite covers these scripts, so each behaviour was driven against a stubbed curl:

Case	Result
204	reclaimed, one attempt, rc=0
403	two attempts, NOT reclaimed summary, rc=1
hanging backend	bounded by --max-time; pre-fix the stub was handed --max-time=NONE and slept 600s
SIGINT then SIGTERM 0.3s later	handler runs once, summary printed, 6.4s wall
poll 3 fails mid-queue	stays queue=3/3 phase=Unknown, verdict queue timeout, DELETE sent

Plus bash -n on both scripts and yaml.safe_load on both workflows.

What this does not fix

The leak itself. The DELETE still returns 40x and the workload still survives — this makes that loud and bounded rather than silent. Reclaiming the GPU needs the backend cancel to start succeeding; that is tracked separately.

Note this only takes effect for PRs whose branch carries it: pull_request loads the workflow from the PR head, so every other open PR keeps the old single budget until this is on main.

🤖 Generated with Claude Code

[github-actions]

Hyperloom Forge E2E — ❌ Queue timeout

item	value
result	❌ Queue timeout
example	triton-softmax-forge-loop (max_hours=1.0, max_iters=100)
resources	1× GPU
workspace	control-plan-hyperloom-ci
PR branch	fix/ci-e2e-cancel-and-queue-budget
commit	d057c05be039c5d7e37aab94fa063697f0268f06
session_id	1786bd6c-6108-4cb7-80f8-b223c6755bd6
queue → dispatch	–
run time	–
total	12s
reason	never dispatched: still waiting for a GPU after 2400s

Actions run

[github-actions]

CI E2E report — ❌ Queue timeout

item	value
result	❌ Queue timeout
model	Qwen/Qwen3-0.6B (dense)
resources	1× GPU, TP=1
PR branch	fix/ci-e2e-cancel-and-queue-budget
commit	d057c05be039c5d7e37aab94fa063697f0268f06
session_id	8961bb06-7271-48ae-9864-c957e75660a4
queue → dispatch	-71050s
run time	1196m 25s
total	12m 15s
reason	never dispatched: still waiting for a GPU after 2400s
detail	never dispatched: still waiting for a GPU after 2400s

details