ci(tests): shard the bats suite 5 ways, not 4 — the main twin of the #1107 fix - #1162
Merged
Conversation
…1107 fix The same change already landed on the terminal-driver integration branch; this is the main-targeted twin so main PRs stop hitting the macOS 30-min cap that the 4-shard split reaches on this suite too. shard-tests.sh seeds each pinned file at one shard's average share (total_tests/total), which grows with the suite while the pinned files do not, over-reserving the two pin shards and pushing the real load onto the non-pinned ones. main is 1742 @tests, so the worst-case non-pinned shard total*(N+2)/N^2 drops from 653 (N=4) to 488 (N=5), back under the cap with margin. Five, not six, because the account runs at most five macOS jobs at once (the concurrency header). test_ci_workflow.bats reads SHARD_TOTAL and the matrix and follows the /N name; it is green at 5. This does not fix the scaling property (pin_seed still grows with the suite); that re-fit is #847.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The main-targeted twin of the shard change already on
integration/terminal-driver-v1(#1160). Main is still 4-shard, so main PRs — including #1159 — keep hitting the same macOS 30-min cap. This unblocks them.Why 4 shards overload one leg
shard-tests.shseeds each pinned file at one shard's average share (total_tests/total), a value that grows with the suite while the pinned files' real cost does not. That over-reserves the two pinned shards, and the greedy pass fills the non-pinned shards first. Onmainthe suite is 1742 @tests, so the worst-case non-pinned shardtotal*(N+2)/N^2is 653 at N=4; a 4-shard macOS leg reaches the 30-min cap while still passing tests (a slow shard, not a hang — the workflow's own #848/#857 note).This change
SHARD_TOTAL: 4 → 5(+ the matrix and the display-name/N;test_ci_workflow.batsreads all three and is green at 5). Worst-case non-pinned shard 653 → 488, back under the cap with margin.Five, not six: the account runs at most five macOS jobs at once (the concurrency header), so a sixth macOS shard would queue behind the other five of its own run.
Not the fix
The scaling property remains and is tracked in #847:
pin_seedgrows with the suite, and @test count under-values the wait-dominant files. The real re-fit (re-pin by measured seconds-per-test / cap the seed) waits for #1159's per-file timings.