bats shards are balanced by test count, so adding tests can push one bucket past the job ceiling

[fujibee]

Adding tests to any file re-assigns every shard, so a PR that changes only the test count can move two slow files into one bucket and push that bucket past the job ceiling. This happened on a PR branch, recovered on a re-run, and main has about five minutes of headroom, so it can happen again to whoever adds tests next.

Recording the measurements first. The conclusion is deliberately left open.

What was measured

1. The buckets are a function of the head, so a shard number is not a stable name for a piece of work.

.github/scripts/shard-tests.sh partitions *.bats by test count (greedy LPT). Derived with that script on each head, for the two slowest files:

head	test_remote_sync.bats	test_sqlite_sync_jq_binary.bats
4c448cb195aafd7991b373847797154403419fae (main)	shard 2/4	shard 1/4 — split
87d91e1a208b3d3c3a7725b6693b857bef104345 (a PR branch)	shard 2/4	shard 2/4 — together

The branch adds fifteen test cases to one file (test_roster_journal.bats, which lands in shard 3/4 on that head). That is the whole input change: nothing about the two files above is touched.

2. The bucket that got both of them exceeded the ceiling, and then did not.

	duration	outcome
branch, bats (macos-latest 2/4), attempt 1	25m22s	cancelled at the job ceiling
branch, bats (macos-latest 2/4), attempt 2	23m37s	success

The cancelled attempt reported its last test ok twelve seconds before the cancellation, so it was still running normally when the ceiling arrived — not stuck.

3. main is already close to the ceiling.

macOS shards on main at 121d27bb9e7f1c2af8fb0813e90359209691b408's predecessor run:

11m05s  1/4
 9m15s  2/4
20m02s  3/4      <- about five minutes of headroom
11m33s  4/4

And on the branch: 11m21s / 25m22s / 10m48s / 14m05s.

What follows, and what does not

• Not established: that the added cases are themselves expensive enough to matter. They are in a different shard, which finished in 10m48s. Their own runtime is not the cost.
• Established: the changed test count moved the partition, and the partition put the two slowest files together.
• Established: balancing by count does not bound duration. Nothing in the current scheme notices that one file takes twenty minutes and another takes twenty seconds.

So the trigger is any PR that changes how many tests exist — which is most of them.

Candidates, not a decision

Listing these separately because picking between them needs measurement this issue does not have:

1. Weight the partition by recorded duration rather than by test count. Needs a source of timings and a decision about where they live.
2. Pin the known slow files apart, as an explicit rule rather than as an accident of counting.
3. Raise the job ceiling. Cheapest, and it does not stop the buckets from drifting further apart.

A useful first step for any of them is publishing per-shard durations alongside the file list each run, so the next occurrence is measurable rather than reconstructed.

Reproducing the derivation

bash .github/scripts/shard-tests.sh <n> 4      # on each head, for n in 1..4

Comparing shard numbers across heads without deriving the membership will mislead: the same number denotes different files on different commits.

[fujibee]

Root cause confirmed with measured data (from #1107's hang-samples, macOS run 34624540956 and tl's 30s-resolution per-file timings).

The property this issue must fix: shard-tests.sh seeds each pinned file at pin_seed = total_tests / total — one shard's average share. That value grows with the whole suite, but a pinned file's real cost does not. As the suite grew ~2.85x (788 → 2246 @tests), pin_seed went 197 → 561, so the two pinned shards now reserve far more than the pinned files actually cost, and the greedy pass concentrates the real load on the non-pinned shards:

4-shard @test weights (measured): 1/4=281  2/4=302  3/4=832  4/4=831

macOS 4/4 (831 tests) reached the 30-min cap. The shard-tests.sh header's own bound ("worst case is the remaining work over two shards ... which the numbers above clear with room") was written when the suite was ~1/3 this size.

And the pin list is protecting the wrong files. Measured on the cancelled macOS 4/4 (22-min window): the real heavies were test_remote 7.5m / test_remote_sync 5.0m / test_watch 3.0m — none pinned. The two currently-pinned files (test_remote_engine_start_refusal, test_remote_status_liveness) did not appear in the window.

Stopgap shipped: #1160 bumps SHARD_TOTAL 4→5 (dilutes the worst-case non-pinned shard 842→629), which opens the merge gate on integration. It does not fix the property.

Real fix (this issue), once #1159's per-file timings land:

1. Re-decide whether the two current pins still earn a pin (per-test seconds), or unpin one and return a shard's worth to the pool.
2. Cap pin_seed at the pin file's measured cost (used as a cap, it fails safe when stale; the current total/total fails dangerous). Or shard by real wall-time rather than @test count.
   Caveat: the measurements above are 1 shard / 1 run, not a distribution — ci: record per-file bats timings #1159 gives all shards × all files.