Skip to content

Playwright suite at master: can silently target the wrong server, fixtures symlink outside deploy/dev, and several specs target removed UI #96

Description

@chertsa

Three things we hit while getting the Playwright suite running against a vendored copy of
deploy/dev. Each one reads as an app failure when it isn't, so they cost more time than
their size suggests.

1. The suite can silently run against the wrong server.
deploy/dev/tests/playwright.config.js declares:

webServer: {
  command: `python3 -m http.server 8080 --directory "${DEPLOY_ROOT}"`,
  port: 8080,
  reuseExistingServer: true,
}

Port 8080 is heavily contended on developer machines — Colima and Docker Desktop forward
it, and it's the conventional port for an SSH tunnel to a demo box. With
reuseExistingServer: true, Playwright finds the port bound, skips starting its own
server, and runs the entire suite against whatever is already there.

It doesn't fail as a bind error or a wrong-target error, which would be obvious. Every
spec fails with TimeoutError: page.waitForSelector: Timeout 30000ms exceeded on
#canvas, because the foreign server 404s on /dev/index.html. It reads as a broken
viewer.

Observed here: Colima had 8080 forwarded, and 01-viewer-load.spec.js reported 0/9.
Moving to a dedicated port with reuseExistingServer: false and changing nothing else
took the same spec to 6/9. Six of those nine failures were never real.

Any of these would fix it: a less contended default port; reuseExistingServer: false; or
a beforeAll assertion that GET /dev/index.html returns 200, so a wrong target fails
loudly in milliseconds.

2. Test fixtures are symlinks pointing outside the deployable tree.
deploy/dev/tests/fixtures/test.ifc and test_revised.ifc symlink to
../../../../reference/residential/Ifc4_SampleHouse{,_Revised}.ifc — outside deploy/dev/.
Anyone who deploys, packages or vendors deploy/dev/ as the unit GH_DEPLOY.md designates
as the source of truth gets two dangling symlinks and can't run the specs that use them.
Vendoring the two files into fixtures/, or resolving them at test setup, would make
deploy/dev/ self-contained.

3. Several specs target UI controls that no longer exist, so they can never pass.
Because page.click() waits for an actionable element, each of these burns the full 60 s
test timeout and reports as a timeout rather than "selector not found" — which reads as a
slow or broken app rather than a stale test.

Selector Spec Result at 55900e2d
#theme-btn 01-viewer-load 1.7 No such element anywhere in deploy/dev; the only "theme" in index.html is <meta name="theme-color">
#fly-btn 01-viewer-load 1.9 Referenced from JS, never rendered. tour.js:9 says so: // §S280: may be null (pill removed button)
#disc-body 02-panels 7 of 7 tests fail
#find-axis-disc 40-find-isolate 4 of 4 tests fail
#import-drop-zone 37-s252-revit-colour 1 of 1 test fails

That's at least 14 failures across three fully-red spec files plus two more tests. The
§S280 comment suggests the fly control was deliberately replaced by a pill UI and the
spec wasn't updated; the others look like the same pattern.

Two cheap suggestions: assert selector existence up front, so these fail in milliseconds
with a clear message instead of timing out; and treat a fully-red spec file as a signal
that the spec, not the app, may have drifted.

(For completeness — 41-room-volume-lens also references three missing #find-axis-*
selectors yet passes 3/3, so those must sit in a tolerant path. A missing selector doesn't
by itself imply a failure.)


Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions