Unit test launches a real browser to assert on file contents, flaking CI on unrelated PRs

[dinndev2]

What

buildSession records the app address in story.md for a web run asserts only on the contents of a file, but building a web target constructs the real browser adapter — so a text assertion launches Chromium. That launch flakes under CI's parallel load and fails the whole suite on PRs that touch no code.

Evidence

PR #57 adds a single file, .maintainer.yml. It cannot affect any test. Its CI run failed:

1108 pass
52 skip
1 fail
1 error
Ran 1161 tests across 75 files. [15.95s]

The failure:

(fail) buildSession records the app address in story.md for a web run [3.57ms]

error: Failed to connect
  syscall: "connect", errno: -2, code: "ENOENT"
    at connect (node:net:516:16)
    at #createStdioObject (node:child_process:622:42)
    at spawn (node:child_process:701:30)
    at launchProcess (node_modules/playwright-core/lib/coreBundle.js:8764:39)
    at _launchProcess (node_modules/playwright-core/lib/coreBundle.js:39200:81)

gh run rerun --failed on the same job, same SHA, no push: success.

It is not a missing browser

Worth stating, because the ENOENT invites that conclusion. That run was a cache miss, so the full install branch ran and completed:

Cache not found for input keys: Linux-playwright-ade6e55e..., Linux-playwright-
  bunx playwright install --with-deps chromium
Chrome for Testing 151.0.7922.34 (playwright chromium v1234) downloaded to
  /home/runner/.cache/ms-playwright/chromium-1234

The binary was there. The ENOENT is on a connect() inside #createStdioObject during child_process stdio setup, not on resolving the executable. That is a launch-time race, not an install problem.

Locally the suite is green — 1109 pass, 52 skip, 0 fail across the same 1161 tests — which is the expected result on a machine that is not starved.

Root cause

The test does not need a browser. From src/services/session-builder.test.ts:132-141, on the shared timeout constant:

Budget for the two story.md address tests. They assert on file CONTENT, not on speed, but building a web target constructs the real adapter — and CI runs the whole suite in parallel on a 2-vCPU runner alongside a real-browser e2e file. Locally these take ~0.4s; starved, one crossed Bun's 5s default and failed a green branch.

So this was already known to flake, and the mitigation was to raise the timeout to 30s. That addresses a slow launch. It cannot address a failed one, which is what happens here.

The sibling tests in the same file show the intended shape — they are explicitly named for it:

• buildSession wires a desktop target (addendum + adapter) without launching
• buildSession wires an android target (addendum + adapter) without launching

Only the web path drags in a real adapter. The two affected tests are buildSession records the app address in story.md for a web run (line 146) and buildSession honors a per-run url override in story.md (line 157) — the second has the same exposure and simply has not lost the race yet.

Why it matters

The suite already concedes that real-browser work is unreliable on this runner — CI sets SKIP_BROWSER_INTEGRATION: "1" with a comment about the low-level adapter suite flaking "under the runner's parallel 2-vCPU load". These two tests take a real browser launch anyway, without opting into that protection, and they are ordinary unit tests. The result is a required check that can fail on a documentation- or config-only PR, which trains people to re-run rather than read a red build.

Suggested fix

Give the web path the same treatment the desktop and android tests already get: build the session against a fake or injected adapter so the assertion stays on story.md content. That removes the browser from two unit tests, makes the 30s timeout unnecessary, and leaves real-browser coverage where it belongs — in the e2e suite that CI deliberately protects.

If the real adapter must stay for coverage reasons, the alternative is to move both tests behind the same SKIP_BROWSER_INTEGRATION guard as the other browser-dependent suite, so CI stops gating on a launch it has already documented as unreliable.

Secondary observation, not the cause here

Unrelated to this failure but adjacent: on an exact cache hit the install chromium step runs bunx playwright install-deps chromium, which installs system libraries and not the browser. If a restored cache ever lacks the expected chromium build, that branch cannot repair it. It did not fire in this run (the cache missed and the full install ran), so this is a latent edge rather than an observed bug — noting it only so it is not rediscovered from scratch later.

[dinndev2]

Second test in this file, and this time it reproduces on main

Same class, same file, different test: a persona signs the run in before the first step, and leaves no credential in the logs (src/services/session-builder.test.ts:477) times out at 30s under CI's parallel load.

The evidence this issue was missing — a failure on a commit that provably cannot cause it:

Run 31235917066 is the push build for b10171a5 on main. That commit's entire diff:

biome.json  +0-1

One deleted line of lint config (#63). It failed CI with the persona test timing out. This is stronger than the #57 evidence in the original report: not just "a PR that adds one file", but a merge to main whose whole content is a config deletion.

It then hit PR #62 on the next run, 31236193825. Same test, same 30s timeout, 1117 pass / 53 skip / 1 fail. Discriminating facts:

• PR fix(adapter): surface a full-document load the driver did not ask for #62's first CI run on the same branch passed.
• gh run rerun --failed on the failing run — same SHA, no push — came back success on attempt 2. Same recipe that closed the question in the original report.
• The file passes locally in isolation, 3 runs out of 3, on the exact failing commit.

Why it matters beyond a red check

This one cost more than a rerun. An automated merge-on-green tool picked up the failure, treated it as PR #62's, and started editing source to chase it. It was stopped before it committed anything, but that is the real bill for a flake in a gate: not the retry, it is that every automated consumer of CI treats a flaky red as a true red, and "fix it" is the natural next move.

Scope note

The fix as scoped — stop the story.md assertion from launching Chromium — addresses the test named in the original report but not this one. a persona signs the run in genuinely needs a browser: it drives a real login against a Bun.serve fixture and asserts no credential reaches the logs. That coverage is worth keeping.

So the file has two different problems wearing the same symptom:

1. Tests that launch a browser for no reason — the story.md case. Fix by not launching (the original proposal).
2. Tests that legitimately need a browser but sit in a file bun test runs at full 75-file concurrency — this one. Those need the same treatment browser-adapter.integration.test.ts already gets: a skip guard so CI can gate them off, or extraction into a suite that does not race 74 other files for 2 vCPUs. ci.yml already sets SKIP_BROWSER_INTEGRATION=1 for exactly this reason, and there is no equivalent lever here.

Until one of those lands, a red bun (lint + typecheck + test) on this repo cannot be read as "this PR is broken" without opening the log — which is the property a gate exists to provide.