ci(e2e): assign spec directories to shards by measured duration - #3410
Merged
Conversation
Playwright's --shard splits by test count and a file lands where its first test falls. In the 2026-09 audit that left shard 4 (timed, onboarding) at 2m16s of tests as the run's critical path while shard 3 finished in 26s. Hand-assigned directory lists sized from the measured shard timings put each shard near 90s, about 40s off every e2e run. A contract test fails when an e2e/ directory with specs is missing from every list or appears in two, so a new directory cannot silently skip CI. Records the after numbers for #3407 and #3408 in the audit doc. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tyler-dane
added
the
agent-automerge
tyler-dane
enabled auto-merge
September 5, 2026 14:59
First run measured timed at 110s with oauth against booking at 78s. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
actionlint reads directives from the script body, not the YAML above it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: no issue. Fourth fix from the CI audit (#3407, #3408, #3409).
What and why
Playwright's
--shard=n/4splits by test count, and a spec file goes to the shard that contains its first test, so large files spill. The audit measured shard 4 (oauth,onboarding,timed) at 2m16s of tests as the critical path of every e2e run while shard 3 (calendars,life,navigation) finished its tests in 26s. Total test time is about 6 minutes, so an even split is about 1m30s per shard.This replaces
--shardwith a hand-assigned directory list per shard, sized from the measured shard timings and the per-file ratios inside each shard:The four lists select 32 + 48 + 18 + 33 = 131 tests, the full suite, with no directory in two lists (checked with
playwright test --list). The price of a hand list is that a newe2e/directory could silently never run in CI.packages/scripts/src/testing/e2e-shards.test.tsis that price: it fails when a directory with specs is missing from every list or appears in two, and it fails when the run step stops passing the list to Playwright. Verified by removinge2e/oauthfrom the matrix locally: the test fails with the missing directory in the diff.Shard job names change from
e2e-shard (1)toe2e-shard (1, e2e/accessibility e2e/allday). The requirede2egate is unchanged, so no ruleset change.Also records the measured after numbers for #3407 (Install Bun 3s to 7s per shard, no tail) and #3408 in
docs/CI-CD/ci-audit-2026-09.md.Verify
The three failures are the same as in #3407 and #3408:
agent-loop-next.shusesmapfile, which macOS/bin/bash3.2 lacks; not touched by this diff, passes on Linux. The new contract test, all other scripts tests, type-check, lint, and knip pass locally. This PR's own e2e run is the first measurement of the new split; CI on Linux is the arbiter.🤖 Generated with Claude Code