ci(tests): shard the bats suite 5 ways, not 4, to clear the 30-min macOS cap (#1107) - #1160
Merged
fujibee merged 2 commits intoSep 11, 2026
Merged
Conversation
…cOS cap (#1107) The suite grew ~2.85x (788 -> 2246 @tests) since the partition was tuned, and shard-tests.sh seeds each pinned file at one shard's average share (total_tests/total = 561 now). That over-reserves the two pinned shards, so the whole real load lands on the non-pinned ones: measured 4-shard @test weights were 281/302/832/831, and macOS 4/4 (831 tests) reached the 30-minute cap while still passing tests -- a slow shard, not a hang (the workflow's own #848/#857 note). Five shards instead of four drops the worst-case non-pinned shard from total*(N+2)/N^2 = 842 to 629 (measured 633), ~21 min on macOS with margin. Five, not six, because the account runs at most five macOS jobs at once (the concurrency header): a sixth macOS shard would queue behind the other five of its own run and inflate the wall clock past what a lighter shard buys. test_ci_sharding.bats reads SHARD_TOTAL and the matrix, so the three edits stay in step; it is green at 5. This is a stopgap, not the fix for #1107. The scaling property remains -- pin_seed grows with the suite, and @test count under-values the wait-dominant files (test_remote 7.5m / test_remote_sync 5.0m / test_watch 3.0m on macOS, none the currently pinned pair). #847 carries the real re-fit once #1159's per-file timings land.
…ecks test_ci_workflow.bats greps the workflow for the exact `bats (… /N)` job name to prove the docs-only skip reports the same required contexts as a real run (#798), and again in the mutation control. Both literals said /4 and did not follow the SHARD_TOTAL 4 -> 5 change, so the real check failed on every shard while the mutation control still passed by luck. Point both at /5. They stay literal for now; deriving the name from SHARD_TOTAL so the next N never breaks them is left to #847 with the rest of the partition work, to keep this change to opening the gate.
fujibee
merged commit Sep 11, 2026
63a33f7
into
integration/terminal-driver-v1
77 of 79 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stopgap for #1107. Unblocks the merge gate on
integration/terminal-driver-v1(which #1155 / #1156 / #1157 are all waiting on). Amain-targeted twin follows separately.What #1107 actually is
Not a hang — a slow shard hitting the 30-min wall-clock cap while still passing tests (the workflow's own #848/#857 note warns these look identical). Measured from the hang-samples time series, the cancelled macOS 4/4 was progressing the whole way; it just had too much to do.
Why one shard is overloaded
The suite grew ~2.85x (788 → 2246 @tests) since the partition was tuned.
shard-tests.shseeds each pinned file at one shard's average share (total_tests/total= 561 now), which over-reserves the two pinned shards; the greedy pass then fills the non-pinned shards first. Measured 4-shard @test weights:This change
SHARD_TOTAL: 4 → 5(+ the matrix and the display-name/N, kept in step;test_ci_sharding.batsreads all three and is green at 5). Worst-case non-pinned shardtotal*(N+2)/N^2: 842 → 629 (measured 633), ~21 min on macOS with margin. New weights:Five, not six: the account runs at most five macOS jobs at once (the concurrency header). A sixth macOS shard would queue behind the other five of its own run and inflate the wall clock past what the lighter shard buys — the very "per-shard shorter but overall longer" failure the header cautions about. Five fits the slots exactly.
Not the fix
The scaling property remains and is tracked in #847:
pin_seedgrows with the suite, and @test count under-values the wait-dominant files (measured on the cancelled shard:test_remote7.5m /test_remote_sync5.0m /test_watch3.0m — none of them the currently pinned pair, which never appeared in the 22-min window). The real re-fit (re-pin by measured seconds-per-test / cap the seed) waits for #1159's per-file timings.