ci: macOS shard runtime has outgrown the 25-minute cap — adding tests can re-pack a shard over the cliff (wall-clock cancel of a passing shard, not a hang)

[Masashi-Ono0611]

tests.yml caps each bats shard at timeout-minutes: 25 with the comment "A shard has generous headroom under this cap; a timeout is a genuine hang." That assumption no longer holds on macOS runners, and the failure it produces is a wall-clock cancellation of a passing shard, not a hang and not a flaky test.

Measured on main (run 31936446984, 2026-08-16): bats (macos-latest 1/4) took 21m17s of the 25m budget; the other quarters took ~10–11m. The shard helper's own header still cites 125s/298s/366s/71s quarters — the suite has grown well past those numbers.

How it bites a PR: sharding is count-balanced and computed from the tree, so a PR that adds tests re-packs the shards. On #507 (+13 @tests), the re-pack made bats (macos-latest 4/4) a 384-test shard that runs all 384 tests to ok 384 and is then cancelled by the timeout — observed twice, at 25m07s and 25m20s, with the final ok landing seconds before the cancellation (second run: last ok 384 at 14:03:39, cancel at 14:04:07). The log shows 1..384 with zero failures; the "genuine hang" the cap was written to catch is exactly what this is not.

With main's largest quarter already at 21m, a test-adding PR that re-packs a shard past ~25m of runtime hits the same cancellation, and the signature (a CANCELLED shard failing the check) reads like the PR broke something.

Possible directions (maintainer's call): raise SHARD_TOTAL for macOS, raise the cap while keeping a genuine-hang detector (e.g. a no-output watchdog instead of wall-clock), or move the balancing weight from @test count to measured runtime. Related but distinct: #595 / #828 are about flaky individual tests; this one is capacity.

Happy to send whichever direction you prefer as a PR.

[fujibee]

Measured where this stands now, 2026-08-21, from the job timestamps of the tests workflow runs (nothing in the issue body was re-measured).

The cap and the shards. timeout-minutes is 30 since #857. On main over 2026-08-17..21 the heaviest macOS shard ran 22–25 min per run (25 once on 08-18, 23–24 after #847's pin); headroom to the cap is 5–8 min. ubuntu shards run 5–12 min, the three Windows jobs ~1 min. Packing is by @test count, greedy longest-first (411/410/410/410 on main today) with the two pinned files from #847 held apart; since the weight is a count and not a runtime, the four macOS shards spread 7–24 min for the same number of tests — a 3.4x spread.

What adding tests does. Three test files landed today (#917, #918, #923). On the macOS shard that carries tests/test_perf_harness.bats, its tests ran from ok 397 to ok 401 in about ten seconds (job log timestamps); the other two are node tests or seconds. The heaviest shard going 22 → 24 min between two runs on the same day is runner-to-runner and repack variance, not those tests. So today, adding a test file moves a shard by seconds against a 5–8 minute margin.

What actually jammed today was not the cap. 35 runs of tests were created today (28 pull_request, 7 push), each with 4 macOS shards = 140 macOS jobs, average ~13 min each, and the maximum number of macOS jobs ever observed running at once was 5 (the runner concurrency ceiling for this account). That is roughly six hours of macOS runner time pushed through five lanes, and it shows up as queue wait, not runtime: macOS shards waited 0–50 min to start (one PR's shard 3/4 waited 50 min), and at 20:40Z three main runs created after 19:18Z had not started a macOS job yet. Several of those 35 runs were superseded — the same PR's head moved two or three times during review, and a rebase of another — and the superseded runs kept their macOS slots until they finished. tests.yml has no concurrency: group, so nothing cancels them.

What each direction fixes, and what it does not.

direction	fixes	does not fix
cancel superseded PR runs (concurrency keyed by PR, cancel-in-progress for pull_request only)	the queue: a head that moved frees its old run's 4 macOS slots immediately; nothing is lost, because a superseded run's result cannot be used for anything	shard runtime or the 3.4x spread; the cap stays where it is
more macOS shards	per-shard runtime	under a 5-job ceiling it adds jobs per run and lengthens the queue
runtime-weighted packing	the 3.4x spread; the heaviest shard comes down toward the mean (~13 min)	the number of runs or of jobs per run
macOS only on main / labelled PRs	the queue, strongly	macOS coverage on PRs — today's most important catch (#928, a bash 3.2 quoting difference) came from a PR's macOS shard failing; hiding that until main loses exactly that

The first is the one being taken now, on its own: it is the only one that costs nothing and whose trigger is exactly what happened today. The others stay here as measured options.

[fujibee]

The remaining half of the measurement: what runtime-weighted packing would buy, computed from a green run rather than implemented.

Method. Run 32515406657 (main, 2026-08-21, all four macOS shards green): each shard's per-ok log timestamps, mapped back to files through the shard-manifest artifacts and each file's @test count at that commit, give a measured per-file duration for all 86 files. Job fixed overhead (checkout/setup) is under a minute — wall time is essentially test time.

Where the current count-balanced packing lands (this run, same runner class):

shard 1: 19.4 min   shard 2: 13.4 min   shard 3: 7.1 min   shard 4: 17.5 min     total 57.5 min
heaviest files: refusal 614 s, test_remote 516 s, status_liveness 257 s, spawn 212 s, team 147 s

Equal test counts (408/407/408/407), a 2.7x spread in time.

Runtime-LPT repack of the same files into the same four shards: all four bins come out at 14.4 min — exactly the 57.5/4 lower bound, so the balance is as good as balance can get. The heaviest shard drops 19.4 → 14.4 min. Scaled to a slow-runner day (this week's worst was 25 min for the same packing, ×1.29): worst ≈ 18.6 min, i.e. headroom to the 30-min cap goes from ~5 to ~11.4 min.

What that headroom is worth in test additions: the three test files added on 2026-08-21 cost the shard that carries them 10–30 s in total. Even assuming every future addition lands on the heaviest shard, ~11 min of headroom is 25–60 additions of that size; with runtime weights an addition goes to the lightest bin, so several times more in practice.

Two limits, and the first is the real one:

1. The floor is a single file. test_remote_engine_start_refusal.bats alone runs 10.2 min (9 tests, ~68 s/test — the cost is waiting, not volume). No number of shards brings the heaviest shard below the heaviest file, so the next cliff is not repack luck; it is the day another file of that class appears — or that file itself growing. Splitting it (or removing its waits) moves the floor; repacking cannot.
2. A checked-in runtime table goes stale. These per-file seconds are one run on one runner (±30% run-to-run this week). Any runtime-weighted scheme would need its weights measured, not maintained by hand — the shard manifests and job logs used here are enough to recompute them automatically, but that is an implementation question, deliberately not taken here.

Together with the earlier comment: #933 addressed the queue (the thing that actually bit on 2026-08-21); balancing would buy ~6 min of bad-day margin and is not urgent at today's growth rate; the floor file is where a real capacity change lives.