CI: macos-x86 (TCG) full suite hangs silently mid provisioning.test.ts

[mobileskyfi]

First full-suite run on macos-15-intel (TCG) — possible since PR #75 — hung rather than failing: https://github.com/tikoci/quickchr/actions/runs/28749608475/job/85246320846

Observed

The leg ran fine for ~1h: earlier files passed sequentially, then provisioning.test.ts passed 9 of its ~11 tests (each 40-62s, comparable to KVM timings!) and went silent for 30+ min after:

(pass) provisioning corner cases > createUser with an invalid group throws PROCESS_FAILED [41165.41ms]

No further output. The same file passed on windows-x86/TCG in the same run (491s, 11 tests) — so this is not "provisioning can't work under TCG".

Why it's odd

• Tests in this file boot machines sequentially and each prior boot took ~40-60s — a wedged boot should have failed a per-test timeout and printed, not gone silent.
• 30 min of no output at all smells like the bun test process itself starved or blocked — a leftover TCG QEMU pinning both cores (the reap step only runs after the loop), a hung image download without a timeout, or an OS credential-store (Bun.secrets/Keychain) prompt blocking headlessly — the last passing tests exercise the Keychain path.
• Cancellation itself took several minutes to take effect, consistent with a starved runner.
• Matches the historical signature: macos-x86 "runner lost communication ~64 min" (pre-refactor), which was previously masked with continue-on-error instead of investigated.

Evidence

• Run was cancelled at ~T+60min to free the runner; artifact integration-logs-macos-x86-stable (integration-output.txt, timing, machine dirs, boot-log) should carry the post-mortem — analyze which test was next and what its machine's qemu.log shows.
• Windows leg of the same run: 59/60 pass (see Windows support: first-class support research and alignment (umbrella) #40 comment) — the TCG-hostility is macos-x86-specific here.

Done-when

1. Artifact post-mortem identifies the blocked test + blocking syscall/process (or shows the artifact never uploaded — itself evidence of full runner starvation).
2. Local repro attempt: run provisioning.test.ts on the Intel Mac under forced TCG (QUICKCHR_INTEGRATION=1 + accel override) — we have the exact hardware class locally.
3. Fix or bound the hang (e.g. per-file cap — see the run-time caps issue) and re-dispatch integration.yml -f platforms=macos-x86 to confirm the leg completes (pass or honest fail).

🤖 Filed from live-run evidence by Claude Code

[mobileskyfi]

Post-mortem attempt: the runner starvation theory is now grounded — by total evidence loss

After cancellation (submitted ~18:51, job marked cancelled 18:59:03), the step timeline shows the failure mode exactly:

• Run integration tests (sequential per-file) — frozen in_progress, never concluded
• Every subsequent step stayed pending and never ran — including the three if: always() steps (Kill leftover QEMU processes, Assemble metrics, Upload integration logs)
• No artifact was uploaded, and the job's log blob does not exist (BlobNotFound, retried for several minutes after completion) — the runner never even flushed its logs

So the runner didn't just hang the test process: it stopped servicing GitHub's cancellation grace period (SIGINT → always() steps) and log shipping entirely. The only record of the leg's output is the live-streamed view captured in this issue's description.

Consequences for the investigation:

1. CI cannot answer this one — there is nothing more to extract from hosted runs. The local forced-TCG repro on the Intel Mac (done-when fix(net): try normal fetch() first, fail back to public-DNS IPv4 #2) is the only evidence path, and it's a better one anyway (we can watch top, sample the bun process, and check for Keychain prompts directly).
2. This failure mode also invalidates the workflow's own safety net in exactly this scenario: the QEMU-reap step is unreachable when orphans have already starved the runner. CI: per-file runtime caps derived from ci-data timing (hang → fast honest fail) #77's "reap on per-file cap-hit, inside the loop" is the structural fix — the reap must happen while the runner can still execute anything at all.
3. Timeline detail worth keeping: the leg was healthy for ~1h (sequential 40-62s boots, KVM-comparable), then went from fine to fully unresponsive with no intermediate degradation visible in test durations. Whatever starved it did so abruptly after createUser with an invalid group throws PROCESS_FAILED passed — consistent with a specific event (a boot that spun without ever failing, or a blocking prompt) rather than gradual TCG load accumulation.

[mobileskyfi]

Additional evidence from run https://github.com/tikoci/quickchr/actions/runs/28758858863:

• Two more macos/x86_64 full-suite integration legs failed: long-term job 85270515871 and testing job 85270515849.
• Both jobs died while still in Run integration tests (sequential per-file). Step metadata shows no cleanup, metrics, summary, or artifact upload steps ran.
• gh run view --job ... --log, gh run view --log-failed, and the Actions job-log API all report no log for those two failed jobs, so this is the same total-evidence-loss failure mode as the original report.
• The matrix did plan macos-x86 with timeout:300 for this run (platforms=all, routeros-targets=long-term,testing, empty test-filter, tcg-smoke=false), yet the jobs ended around 57-62 minutes. That points at hosted runner failure/lost communication or an external macOS job cap, not our matrix timeout.
• The macos/x86_64 examples smoke jobs for both targets passed in the same run, so the failure is specific to the full integration suite.
• New accelerator caveat: successful issue69-settling.probe.ts artifacts on the same macos-15-intel runner report quickchr detectAccel(x86): hvf, not TCG. The issue title/body likely over-attributes this to TCG; the current grounded bucket is “macos-x86 full-suite runner/job starvation with no logs/artifacts,” not proven TCG.
• No ci-data files exist for these failed macos-x86 integration legs because no metrics artifact was uploaded.

Related but separate: #79 tracks the new linux/aarch64 · long-term clean() post-reset boot timeout from the same run.

[mobileskyfi]

Fourth occurrence: run 29061855653 (2026-07-10)

Same total-evidence-loss signature recurred in job Integration (macos/x86_64 · stable):

• Job conclusion: failure, but every step from Run integration tests (sequential per-file) onward shows conclusion: null — never actually
  concluded, matching the earlier "frozen in_progress, always() steps never
  ran" pattern.
• GET .../actions/jobs/86265259243/logs → BlobNotFound (job never flushed
  logs), same as the prior occurrences.
• The job's check-run annotation this time is explicit and confirms the
  standing theory: "The hosted runner lost communication with the server.
  Anything in your workflow that terminates the runner process, starves it for
  CPU/Memory, or blocks its network access can cause this error." — this is
  the clearest first-party confirmation yet that it's runner-side
  starvation/disconnection, not a quickchr hang per se.
• No artifact uploaded for this leg, so — as with the prior three — there's
  nothing new to extract from this run specifically; the local forced-TCG
  repro (done-when fix(net): try normal fetch() first, fail back to public-DNS IPv4 #2) is still the only path to root-cause which test/step
  triggers the starvation.
• Unrelated, same run: windows/x86_64 · stable also failed in this run, but
  that's a distinct, newly-filed issue (CI: Windows/x86_64 host-OpenSSH batch login (batchVerified) fails 100% in first exercised run #87 — real SSH batch-login failures),
  not this flake.

[mobileskyfi]

Recurred on all three macos-x86 legs — and narrow runs prove it is the full suite, not the platform

Part of the ci:slow-platform-flake series (#76 #79 #80 #91).

Run A — 30279318703, full matrix

All three macos-15-intel Integration legs failed identically:

Job	Started → completed	Elapsed
Integration (macos/x86_64 · long-term)	15:18:47 → 16:22:54	64 min
Integration (macos/x86_64 · testing)	15:18:18 → 16:24:21	66 min
Integration (macos/x86_64 · stable)	15:43:25 → 16:47:32	64 min

Every one carries the same check annotation:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Step conclusions show Run integration tests (sequential per-file) with conclusion: null — it never finished. No logs and no artifacts: GET /actions/jobs/<id>/logs returns BlobNotFound, and no integration-logs-macos-x86-* artifact exists for the run. The job died before the upload step, so there is nothing to post-mortem. This is the same ~64-minute signature the issue already describes as the historical macos-x86 shape, now reproduced three-for-three.

Note the matrix budget was not the limiter: macos-x86 with no filter and tcg-smoke=false gets timeout=300 minutes, so these died at ~64 min for their own reasons, not a workflow timeout.

Run B — 30289164062, narrow

test-filter=start-stop.test.ts,disk.test.ts, run-examples=false:

Job	Result
Integration (macos/x86_64 · stable)	pass
Integration (macos/x86_64 · long-term)	pass

Both green, logs and artifacts intact. So the platform runs quickchr fine under TCG — what kills it is the duration/CPU-burn of the full suite, consistent with this issue's leading theory (leftover TCG QEMU pinning the cores; the reap step only runs after the loop). It supports the starvation reading over any per-test defect.

Interaction with the rest of the series

Run A also produced a Monitor command timed out on macos/x86_64 · stable in the examples-smoke job (via the rollback example) — recorded in #80, since that leg did produce logs and is a different failure mode from the runner death here.

Bearing on #77

This is the strongest argument yet for #77 (per-file runtime caps from ci-data timing): when this leg dies there is zero diagnostic residue — no logs, no artifacts, no annotation beyond the generic runner message. A cap that fails fast and honestly, mid-suite, would convert a 64-minute silent death into an actionable failure with logs attached. Until then, tcg-smoke=true for macos-x86 bounds the cost but also bounds the coverage — worth deciding explicitly rather than by default.