diff --git a/.github/workflows/test-e2e.yml b/.github/workflows/test-e2e.yml index a2ba16a83..669db3df2 100644 --- a/.github/workflows/test-e2e.yml +++ b/.github/workflows/test-e2e.yml @@ -65,6 +65,15 @@ jobs: # PR-to-merge time on 2026-09-03. The required check is the `e2e` gate # below, so the shard legs can change count without touching branch # protection. + # Directories are assigned by hand instead of `--shard=n/4`: Playwright + # splits by test count and a file lands where its first test falls, which + # left shard 4 (timed, onboarding) at 2m16s of tests while shard 3 ran 26s + # in the 2026-09 audit. These lists put each shard near 90s. Every e2e/ + # directory with specs must appear in exactly one list; + # packages/scripts/src/testing/e2e-shards.test.ts fails otherwise, so a new + # directory cannot silently skip CI. Resize by moving directories between + # lists; measured `Run e2e tests` durations are in + # docs/CI-CD/ci-audit-2026-09.md. # No display name: a skipped matrix collapses to one check whose name # would show the unexpanded ${{ matrix.shard }} expression. e2e-shard: @@ -75,7 +84,15 @@ jobs: strategy: fail-fast: false matrix: - shard: [1, 2, 3, 4] + include: + - shard: 1 + specs: e2e/accessibility e2e/allday + - shard: 2 + specs: e2e/booking e2e/oauth + - shard: 3 + specs: e2e/timed + - shard: 4 + specs: e2e/onboarding e2e/calendars e2e/life e2e/navigation e2e/attendees container: # Keep this tag aligned with the Playwright version in bun.lock. image: mcr.microsoft.com/playwright:v1.60.0-noble @@ -125,7 +142,11 @@ jobs: TZ: Etc/UTC GOOGLE_CLIENT_ID: test-client-id API_BASEURL: http://localhost:3000/api - run: bunx playwright test --shard=${{ matrix.shard }}/4 + SPECS: ${{ matrix.specs }} + run: | + # Unquoted on purpose: each directory is a separate file filter. + # shellcheck disable=SC2086 + bunx playwright test $SPECS # The one required status check. Reports success when every shard passed # or when the `changes` gate skipped them all, and fails on a shard diff --git a/docs/CI-CD/ci-audit-2026-09.md b/docs/CI-CD/ci-audit-2026-09.md index 02884cc63..43e6392c4 100644 --- a/docs/CI-CD/ci-audit-2026-09.md +++ b/docs/CI-CD/ci-audit-2026-09.md @@ -228,25 +228,61 @@ that the version in `test-e2e.yml` equals the `bun-version` pin in Before: apt step p50 6s, p95 47s, unbounded tail; all 10 `e2e` gate failures in the 100-run window came from this step (the other 7 gate -non-successes were superseded-push cancellations). After: see PR. +non-successes were superseded-push cancellations). + +After (PR #3407, merged 2026-09-05 14:51 UTC): the replacement `Install +Bun` step took 3s to 7s on every shard of the PR run and the merge-queue +run, versus 8s for apt plus setup-bun at p50 before. Setup per shard is +unchanged at p50 and no longer has a tail. Whether the gate failure rate +drops to zero is a question for the next few hundred runs; the mechanism +that produced every one of the 10 is gone. ### 2. Skip e2e shards for backend-, sync-, and scripts-only pull requests -Planned. `detect-code-changes.sh` gains an `e2e` output; the `e2e-shard` -job gates on it. Docs-only handling, the `e2e` gate reporting Success on a -skip, `merge_group`, and `push` behavior are unchanged. +PR #3408, merged 2026-09-05 14:53 UTC. `detect-code-changes.sh` gains an +`e2e` output; the `e2e-shard` job gates on it. Docs-only handling, the +`e2e` gate reporting Success on a skip, `merge_group`, and `push` behavior +are unchanged. + +Before: 50 of 232 PR e2e runs (22%) in the window were on PRs that could +not affect the suite. After: those PRs report `e2e` as Success in about 10 +seconds (the `changes` job plus the gate) instead of about 3m30s, and use +zero shard runners. ### 3. Make the skip-link keyboard e2e test deterministic -Planned. `preparePublicBookingPage` waits for the slot pane to finish -loading before returning, so tests that start typing immediately are not -racing the slots request. +PR #3409. `preparePublicBookingPage` waits for the "Pick a time" heading, +which renders only once slots have loaded, before returning, unless the +caller is deliberately observing the pending, failed, or unavailable state. +The flaky test's assertions are untouched. + +Before: 5 retry-only passes in 705 shard jobs, 5 branches. After: to be +read from the next flaky-log scan. ### 4. Rebalance the e2e shards by measured duration -Planned. Explicit spec-directory lists per shard, sized from the shard -timings above, with a contract test that fails when an `e2e/` directory is -not assigned to any shard. +Explicit spec-directory lists per shard replace `--shard=n/4`. Sized from +the shard timings above, scaled by the local per-file ratios inside each +shard: + +| Shard | Directories | Tests | Estimated `Run e2e tests` | Measured on PR #3410 | +|---|---|---|---|---| +| 1 | accessibility, allday | 32 | ~92s | 90s | +| 2 | booking, oauth | 51 | ~100s | 78s (booking alone) | +| 3 | timed | 15 | ~85s | 110s (with oauth) | +| 4 | onboarding, calendars, life, navigation, attendees | 33 | ~91s | 94s | + +Before: shard 4 at 2m16s, shard 3 at 0m26s; the run's critical path was +shard 4. First measurement (PR #3410 with oauth on shard 3): slowest shard +110s, run wall clock 3m00s versus 3m32s for the merge-queue run of #3407 +on the old split. `timed/` costs more per test in CI than its local ratio +suggested, so oauth moved to the booking shard before merge; expected +slowest shard about 100s. A contract test (`packages/scripts/src/testing/e2e-shards.test.ts`) +fails when an `e2e/` directory with specs is missing from every list or +appears in two, so a new directory cannot silently skip CI. Shard job names +change from `e2e-shard (1)` to `e2e-shard (1, e2e/accessibility e2e/allday)`; +the required `e2e` gate is unchanged. Measured after: see PR and the next +main runs. ## Looked at, not changed diff --git a/packages/scripts/src/testing/e2e-shards.test.ts b/packages/scripts/src/testing/e2e-shards.test.ts new file mode 100644 index 000000000..f4fc1d94f --- /dev/null +++ b/packages/scripts/src/testing/e2e-shards.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from "bun:test"; +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { join } from "node:path"; + +const WORKFLOW = ".github/workflows/test-e2e.yml"; + +/** `specs:` values from the e2e-shard matrix, one array per shard. */ +function shardSpecLists(): string[][] { + const workflow = readFileSync(WORKFLOW, "utf8"); + return [...workflow.matchAll(/^\s+specs:\s*(.+)$/gm)].map((match) => + (match[1] ?? "").trim().split(/\s+/), + ); +} + +/** Top-level `e2e/` directories that contain at least one spec file. */ +function specDirectories(): string[] { + return readdirSync("e2e") + .filter((name) => statSync(join("e2e", name)).isDirectory()) + .filter((name) => + readdirSync(join("e2e", name)).some((file) => file.endsWith(".spec.ts")), + ) + .map((name) => `e2e/${name}`) + .sort(); +} + +// The e2e workflow assigns spec directories to shards by hand, sized from +// measured durations, instead of letting Playwright split by test count. +// The price of a hand list is that a new directory could silently never run +// in CI. This test is that price. +describe("e2e shard assignment", () => { + it("assigns every e2e directory with specs to exactly one shard", () => { + const lists = shardSpecLists(); + expect(lists.length).toBeGreaterThan(1); + + const assigned = lists.flat().sort(); + expect(assigned).toEqual(specDirectories()); + }); + + it("passes the spec list to playwright as file filters", () => { + const workflow = readFileSync(WORKFLOW, "utf8"); + expect(workflow).toContain("bunx playwright test $SPECS"); + expect(workflow).toContain("SPECS: ${{ matrix.specs }}"); + expect(workflow).not.toMatch(/playwright test [^\n]*--shard/); + }); +});