Extension e2e tests have never run in CI (typo'd script name exits 0), and all 4 fail when run

[lmorchard]

Summary

The extension e2e suite has two independent problems: CI has never actually run it, and the suite has rotted to the point where all 4 tests fail when you do run it. The first hid the second.

Verified against main @ 7e7216c1, node 22.23.2, macOS arm64.

1. The CI step is a silent no-op

.github/workflows/build-test.yml (extension job):

- name: Run extension e2e tests
  run: pnpm --filter pilo-extension run test:e2e:headless:chrome

packages/extension/package.json defines test:e2e and test:e2e:headless — there is no test:e2e:headless:chrome. And pnpm --filter <pkg> run <missing-script> exits 0:

$ pnpm --filter pilo-extension run test:e2e:headless:chrome
None of the selected packages has a "test:e2e:headless:chrome" script
$ echo $?
0

So the step has been reporting green while running nothing. Worth noting this failure mode is repo-wide: any pnpm --filter … run step with a typo'd script name passes silently.

2. All 4 tests fail when actually run

pnpm --filter pilo-extension run build:chrome && pnpm --filter pilo-extension run test:e2e:headless → 4 failed. Three distinct causes:

2a. Stale copy in selectors (diagnosed, fix verified)

The sidepanel renders fine — the selectors just describe an older UI. Playwright's page snapshot at the point of failure:

- banner:
  - heading "Tabstack Pilo" [level=1]
  - button "Toggle theme"
  - button "Back to chat"
- heading "Settings" [level=2]

• getByText("Pilo Settings") — the UI now renders Tabstack Pilo (h1) and Settings (h2) as separate nodes, so no single node contains that string.
• getByText("Back to Chat") — the button is actually labelled Back to chat, and getByText is case-sensitive.

This diff makes test 1 pass, and is Playwright-idiomatic (role over raw text):

-    await expect(page.getByText("Pilo Settings")).toBeVisible();
+    await expect(page.getByRole("heading", { name: "Settings" })).toBeVisible();
-    await page.getByText("Back to Chat").click();
+    await page.getByRole("button", { name: "Back to chat" }).click();

The other assertions in test 1 (API Key, Save Settings) are still accurate.

2b. The extensionId fixture doesn't survive sequential launches

With 2a applied, test 1 passes and tests 2–4 all fail in fixture setup, not in the test body:

Test timeout of 30000ms exceeded while setting up "extensionId".

Run test 2 on its own (--grep "should navigate to chat view") and the fixture resolves fine and the test proceeds — so this is specific to launching a second persistent context in the same process, not a broken fixture per se.

e2e/fixtures/extension.ts has two suspects I did not confirm:

• context.waitForEvent("serviceworker", { timeout: 10000 }) never fires if the MV3 service worker registered before the listener was attached (likely on the 2nd+ launch, where the profile may already be warm).
• await serviceWorker.evaluate(() => sw.registration.active) returns a ServiceWorker object, which isn't serializable across the CDP boundary.

The context fixture is test-scoped, so every test pays a full launchPersistentContext — making it suite-scoped would sidestep this and cut runtime (the 4-test run takes ~1.9m).

2c. A further failure in the chat view (not diagnosed)

Even in isolation with 2a applied, test 2 fails later at e2e/sidepanel.spec.ts:49:

await expect(settingsButton).toBeVisible();   // getByTestId("settings-button")

I stopped here rather than keep digging. Unknown whether the settings-button testid was renamed/removed or the chat view genuinely doesn't reach that state.

Suggested order

1. Fix 2a/2b/2c so the suite passes locally.
2. Then correct the workflow script name.

Doing them in the other order turns the extension job red immediately. Given the suite has been dead for some unknown stretch, it's also fair to ask whether these 4 tests are worth repairing versus rewriting against current UI — but the silent-no-op in (1) should be fixed regardless, since it will hide the next breakage too.

[lmorchard]

Correction: cause 2b was wrong

PR #690 fixes this. In the course of it, one of my three diagnoses above doesn't survive scrutiny — flagging it so nobody chases it.

2b ("the extensionId fixture doesn't survive sequential launches") is not a real bug. I probed it directly — a standalone script doing three sequential launchPersistentContext calls with the fixture's exact service-worker logic, logging each step:

=== round 1 ===  launched +2.7s; waitForEvent resolved +2.7s; closed +3.6s
=== round 2 ===  launched +4.8s; waitForEvent resolved +5.3s; closed +6.0s
=== round 3 ===  launched +7.5s; waitForEvent resolved +7.6s; closed +8.3s

No timeout, no hang, every round fine. The Test timeout of 30000ms exceeded while setting up "extensionId" messages were the 30s test budget (which includes fixture setup) being eaten while the machine was loaded by the other tests burning their own 30s timeouts. A symptom of the selector rot, not a separate cause. Once the selectors were fixed, the full suite went from ~1.9m to 14–24s and all 4 passed on three consecutive runs.

So the real story is simpler than this issue says: it was all selector rot — three stale selectors, not two. 2c wasn't a mystery either; settings-button simply doesn't exist as a testid anywhere in src/. The button is aria-label="Open settings".

One thing the probe did turn up, left unfixed in #690: serviceWorker.evaluate(() => sw.registration.active) returns a non-serializable ServiceWorker object, arriving as {} or null. So the comment calling it "critical for MV3 in headless mode" is misleading — it doesn't wait for anything. Harmless today, but it's a decoy for the next person debugging this fixture. Worth either making it a real activation wait or deleting it.