From 4223f93f1951c75e14e063028ea1025bdaedc73e Mon Sep 17 00:00:00 2001 From: Tyler Dane Date: Sat, 5 Sep 2026 08:58:56 -0600 Subject: [PATCH 1/3] ci(e2e): assign spec directories to shards by measured duration Playwright's --shard splits by test count and a file lands where its first test falls. In the 2026-09 audit that left shard 4 (timed, onboarding) at 2m16s of tests as the run's critical path while shard 3 finished in 26s. Hand-assigned directory lists sized from the measured shard timings put each shard near 90s, about 40s off every e2e run. A contract test fails when an e2e/ directory with specs is missing from every list or appears in two, so a new directory cannot silently skip CI. Records the after numbers for #3407 and #3408 in the audit doc. Co-Authored-By: Claude Fable 5.1 --- .github/workflows/test-e2e.yml | 24 ++++++++- docs/CI-CD/ci-audit-2026-09.md | 53 +++++++++++++++---- .../scripts/src/testing/e2e-shards.test.ts | 45 ++++++++++++++++ 3 files changed, 110 insertions(+), 12 deletions(-) create mode 100644 packages/scripts/src/testing/e2e-shards.test.ts diff --git a/.github/workflows/test-e2e.yml b/.github/workflows/test-e2e.yml index a2ba16a83..29db095d8 100644 --- a/.github/workflows/test-e2e.yml +++ b/.github/workflows/test-e2e.yml @@ -65,6 +65,15 @@ jobs: # PR-to-merge time on 2026-09-03. The required check is the `e2e` gate # below, so the shard legs can change count without touching branch # protection. + # Directories are assigned by hand instead of `--shard=n/4`: Playwright + # splits by test count and a file lands where its first test falls, which + # left shard 4 (timed, onboarding) at 2m16s of tests while shard 3 ran 26s + # in the 2026-09 audit. These lists put each shard near 90s. Every e2e/ + # directory with specs must appear in exactly one list; + # packages/scripts/src/testing/e2e-shards.test.ts fails otherwise, so a new + # directory cannot silently skip CI. Resize by moving directories between + # lists; measured `Run e2e tests` durations are in + # docs/CI-CD/ci-audit-2026-09.md. # No display name: a skipped matrix collapses to one check whose name # would show the unexpanded ${{ matrix.shard }} expression. e2e-shard: @@ -75,7 +84,15 @@ jobs: strategy: fail-fast: false matrix: - shard: [1, 2, 3, 4] + include: + - shard: 1 + specs: e2e/accessibility e2e/allday + - shard: 2 + specs: e2e/booking + - shard: 3 + specs: e2e/timed e2e/oauth + - shard: 4 + specs: e2e/onboarding e2e/calendars e2e/life e2e/navigation e2e/attendees container: # Keep this tag aligned with the Playwright version in bun.lock. image: mcr.microsoft.com/playwright:v1.60.0-noble @@ -125,7 +142,10 @@ jobs: TZ: Etc/UTC GOOGLE_CLIENT_ID: test-client-id API_BASEURL: http://localhost:3000/api - run: bunx playwright test --shard=${{ matrix.shard }}/4 + SPECS: ${{ matrix.specs }} + # Unquoted on purpose: each directory is a separate file filter. + # shellcheck disable=SC2086 + run: bunx playwright test $SPECS # The one required status check. Reports success when every shard passed # or when the `changes` gate skipped them all, and fails on a shard diff --git a/docs/CI-CD/ci-audit-2026-09.md b/docs/CI-CD/ci-audit-2026-09.md index 02884cc63..2dbf03a12 100644 --- a/docs/CI-CD/ci-audit-2026-09.md +++ b/docs/CI-CD/ci-audit-2026-09.md @@ -228,25 +228,58 @@ that the version in `test-e2e.yml` equals the `bun-version` pin in Before: apt step p50 6s, p95 47s, unbounded tail; all 10 `e2e` gate failures in the 100-run window came from this step (the other 7 gate -non-successes were superseded-push cancellations). After: see PR. +non-successes were superseded-push cancellations). + +After (PR #3407, merged 2026-09-05 14:51 UTC): the replacement `Install +Bun` step took 3s to 7s on every shard of the PR run and the merge-queue +run, versus 8s for apt plus setup-bun at p50 before. Setup per shard is +unchanged at p50 and no longer has a tail. Whether the gate failure rate +drops to zero is a question for the next few hundred runs; the mechanism +that produced every one of the 10 is gone. ### 2. Skip e2e shards for backend-, sync-, and scripts-only pull requests -Planned. `detect-code-changes.sh` gains an `e2e` output; the `e2e-shard` -job gates on it. Docs-only handling, the `e2e` gate reporting Success on a -skip, `merge_group`, and `push` behavior are unchanged. +PR #3408, merged 2026-09-05 14:53 UTC. `detect-code-changes.sh` gains an +`e2e` output; the `e2e-shard` job gates on it. Docs-only handling, the +`e2e` gate reporting Success on a skip, `merge_group`, and `push` behavior +are unchanged. + +Before: 50 of 232 PR e2e runs (22%) in the window were on PRs that could +not affect the suite. After: those PRs report `e2e` as Success in about 10 +seconds (the `changes` job plus the gate) instead of about 3m30s, and use +zero shard runners. ### 3. Make the skip-link keyboard e2e test deterministic -Planned. `preparePublicBookingPage` waits for the slot pane to finish -loading before returning, so tests that start typing immediately are not -racing the slots request. +PR #3409. `preparePublicBookingPage` waits for the "Pick a time" heading, +which renders only once slots have loaded, before returning, unless the +caller is deliberately observing the pending, failed, or unavailable state. +The flaky test's assertions are untouched. + +Before: 5 retry-only passes in 705 shard jobs, 5 branches. After: to be +read from the next flaky-log scan. ### 4. Rebalance the e2e shards by measured duration -Planned. Explicit spec-directory lists per shard, sized from the shard -timings above, with a contract test that fails when an `e2e/` directory is -not assigned to any shard. +Explicit spec-directory lists per shard replace `--shard=n/4`. Sized from +the shard timings above, scaled by the local per-file ratios inside each +shard: + +| Shard | Directories | Tests | Estimated `Run e2e tests` | +|---|---|---|---| +| 1 | accessibility, allday | 32 | ~92s | +| 2 | booking | 48 | ~90s | +| 3 | timed, oauth | 18 | ~93s | +| 4 | onboarding, calendars, life, navigation, attendees | 33 | ~91s | + +Before: shard 4 at 2m16s, shard 3 at 0m26s; the run's critical path was +shard 4. Expected after: the slowest shard near 1m35s, about 40s off every +e2e run. A contract test (`packages/scripts/src/testing/e2e-shards.test.ts`) +fails when an `e2e/` directory with specs is missing from every list or +appears in two, so a new directory cannot silently skip CI. Shard job names +change from `e2e-shard (1)` to `e2e-shard (1, e2e/accessibility e2e/allday)`; +the required `e2e` gate is unchanged. Measured after: see PR and the next +main runs. ## Looked at, not changed diff --git a/packages/scripts/src/testing/e2e-shards.test.ts b/packages/scripts/src/testing/e2e-shards.test.ts new file mode 100644 index 000000000..f4fc1d94f --- /dev/null +++ b/packages/scripts/src/testing/e2e-shards.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from "bun:test"; +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { join } from "node:path"; + +const WORKFLOW = ".github/workflows/test-e2e.yml"; + +/** `specs:` values from the e2e-shard matrix, one array per shard. */ +function shardSpecLists(): string[][] { + const workflow = readFileSync(WORKFLOW, "utf8"); + return [...workflow.matchAll(/^\s+specs:\s*(.+)$/gm)].map((match) => + (match[1] ?? "").trim().split(/\s+/), + ); +} + +/** Top-level `e2e/` directories that contain at least one spec file. */ +function specDirectories(): string[] { + return readdirSync("e2e") + .filter((name) => statSync(join("e2e", name)).isDirectory()) + .filter((name) => + readdirSync(join("e2e", name)).some((file) => file.endsWith(".spec.ts")), + ) + .map((name) => `e2e/${name}`) + .sort(); +} + +// The e2e workflow assigns spec directories to shards by hand, sized from +// measured durations, instead of letting Playwright split by test count. +// The price of a hand list is that a new directory could silently never run +// in CI. This test is that price. +describe("e2e shard assignment", () => { + it("assigns every e2e directory with specs to exactly one shard", () => { + const lists = shardSpecLists(); + expect(lists.length).toBeGreaterThan(1); + + const assigned = lists.flat().sort(); + expect(assigned).toEqual(specDirectories()); + }); + + it("passes the spec list to playwright as file filters", () => { + const workflow = readFileSync(WORKFLOW, "utf8"); + expect(workflow).toContain("bunx playwright test $SPECS"); + expect(workflow).toContain("SPECS: ${{ matrix.specs }}"); + expect(workflow).not.toMatch(/playwright test [^\n]*--shard/); + }); +}); From 747c6d61fad6f6a2aab7974826b68b539a956adc Mon Sep 17 00:00:00 2001 From: Tyler Dane Date: Sat, 5 Sep 2026 09:03:12 -0600 Subject: [PATCH 2/3] ci(e2e): move oauth specs to the booking shard First run measured timed at 110s with oauth against booking at 78s. Co-Authored-By: Claude Fable 5.1 --- .github/workflows/test-e2e.yml | 4 ++-- docs/CI-CD/ci-audit-2026-09.md | 19 +++++++++++-------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/.github/workflows/test-e2e.yml b/.github/workflows/test-e2e.yml index 29db095d8..e278f4ba1 100644 --- a/.github/workflows/test-e2e.yml +++ b/.github/workflows/test-e2e.yml @@ -88,9 +88,9 @@ jobs: - shard: 1 specs: e2e/accessibility e2e/allday - shard: 2 - specs: e2e/booking + specs: e2e/booking e2e/oauth - shard: 3 - specs: e2e/timed e2e/oauth + specs: e2e/timed - shard: 4 specs: e2e/onboarding e2e/calendars e2e/life e2e/navigation e2e/attendees container: diff --git a/docs/CI-CD/ci-audit-2026-09.md b/docs/CI-CD/ci-audit-2026-09.md index 2dbf03a12..43e6392c4 100644 --- a/docs/CI-CD/ci-audit-2026-09.md +++ b/docs/CI-CD/ci-audit-2026-09.md @@ -265,16 +265,19 @@ Explicit spec-directory lists per shard replace `--shard=n/4`. Sized from the shard timings above, scaled by the local per-file ratios inside each shard: -| Shard | Directories | Tests | Estimated `Run e2e tests` | -|---|---|---|---| -| 1 | accessibility, allday | 32 | ~92s | -| 2 | booking | 48 | ~90s | -| 3 | timed, oauth | 18 | ~93s | -| 4 | onboarding, calendars, life, navigation, attendees | 33 | ~91s | +| Shard | Directories | Tests | Estimated `Run e2e tests` | Measured on PR #3410 | +|---|---|---|---|---| +| 1 | accessibility, allday | 32 | ~92s | 90s | +| 2 | booking, oauth | 51 | ~100s | 78s (booking alone) | +| 3 | timed | 15 | ~85s | 110s (with oauth) | +| 4 | onboarding, calendars, life, navigation, attendees | 33 | ~91s | 94s | Before: shard 4 at 2m16s, shard 3 at 0m26s; the run's critical path was -shard 4. Expected after: the slowest shard near 1m35s, about 40s off every -e2e run. A contract test (`packages/scripts/src/testing/e2e-shards.test.ts`) +shard 4. First measurement (PR #3410 with oauth on shard 3): slowest shard +110s, run wall clock 3m00s versus 3m32s for the merge-queue run of #3407 +on the old split. `timed/` costs more per test in CI than its local ratio +suggested, so oauth moved to the booking shard before merge; expected +slowest shard about 100s. A contract test (`packages/scripts/src/testing/e2e-shards.test.ts`) fails when an `e2e/` directory with specs is missing from every list or appears in two, so a new directory cannot silently skip CI. Shard job names change from `e2e-shard (1)` to `e2e-shard (1, e2e/accessibility e2e/allday)`; From bce065523d70636b0cb0e8e1a67e4e4d14401304 Mon Sep 17 00:00:00 2001 From: Tyler Dane Date: Sat, 5 Sep 2026 09:18:12 -0600 Subject: [PATCH 3/3] ci(e2e): put the shellcheck directive inside the run script actionlint reads directives from the script body, not the YAML above it. Co-Authored-By: Claude Fable 5.1 --- .github/workflows/test-e2e.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test-e2e.yml b/.github/workflows/test-e2e.yml index e278f4ba1..669db3df2 100644 --- a/.github/workflows/test-e2e.yml +++ b/.github/workflows/test-e2e.yml @@ -143,9 +143,10 @@ jobs: GOOGLE_CLIENT_ID: test-client-id API_BASEURL: http://localhost:3000/api SPECS: ${{ matrix.specs }} - # Unquoted on purpose: each directory is a separate file filter. - # shellcheck disable=SC2086 - run: bunx playwright test $SPECS + run: | + # Unquoted on purpose: each directory is a separate file filter. + # shellcheck disable=SC2086 + bunx playwright test $SPECS # The one required status check. Reports success when every shard passed # or when the `changes` gate skipped them all, and fails on a shard