Skip to content

e2e: fold dogfooding into the real suite, add functional mask-effect tests - #90

Merged
stolau merged 1 commit into
mainfrom
e2e-dogfood-and-mask-effect
Jul 28, 2026
Merged

e2e: fold dogfooding into the real suite, add functional mask-effect tests#90
stolau merged 1 commit into
mainfrom
e2e-dogfood-and-mask-effect

Conversation

@stolau

@stolau stolau commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • dogfood-viewer.mjs duplicated capture logic in a standalone script; removed it and folded its capture-and-upload flow into e2e.test.ts itself, since the suite already drives the real viewer through the same key screens. dogfoodCapture(page, name) is a no-op unless PPS_DOGFOOD_SERVER_URL is set — when it is, the suite's existing tests also stash a snapshot of each screen they already visit (run list, run detail, snapshot detail dual/single, Settings, Branches & Releases), and afterAll batches them into one upload against a real, persistent instance. Never auto-approves.
  • Adds two tests proving masks have a real effect on pass/fail, not just that their CRUD UI works: a per-image mask drawn live suppresses a real regression on the same (name, viewport), and a category mask created on one snapshot suppresses a regression on a different name sharing that category.
  • Along the way, mutation-testing applicable_masks()'s call site in render.py exposed a real bug in the new tests' setup: the shared static site server's "changed" variant, once flipped by an earlier test, never resets — so a baseline captured this late in the suite via the ordinary helper was already showing the regressed color, making the new tests pass regardless of whether masks did anything. Fixed with captureCleanPage(), which force-sets .box back to its original color client-side, mirroring how captureRegressedPage() already forces it to the regressed color.

Test plan

  • npm run lint --workspaces — clean
  • npm test (client + viewer unit suites) — 91 passed
  • backend/.venv/bin/pytest backend/tests — 142 passed
  • npm run test:e2e -w examples/demo-app — 16/16 passed
  • Mutation test: hard-coding masks = [] in backend/app/render.py correctly breaks both new mask-effect tests (confirmed before and after the captureCleanPage() fix — before the fix it silently didn't, which is what surfaced the bug)
  • Verified the dogfood opt-in path live via Podman: built and ran the real backend image, ran the suite with PPS_DOGFOOD_SERVER_URL set, confirmed a real 6-snapshot run landed on the backend and the candidate image is a genuine viewer screenshot

🤖 Generated with Claude Code

https://claude.ai/code/session_01NrQeh9CXrigN1cvpJuyzy4

…tests

dogfood-viewer.mjs duplicated capture logic in a standalone script; the vitest
e2e suite already drives the real viewer through the same key screens, so it's
now the source of the dogfood images instead. dogfoodCapture(page, name) is a
no-op unless PPS_DOGFOOD_SERVER_URL is set, in which case the suite's existing
tests also stash a snapshot of each screen they already visit and afterAll
batches them into one upload against a real, persistent instance -- never
auto-approved. Verified live via Podman against a real backend.

Also adds two tests proving masks have a real *effect* on pass/fail (not just
that their CRUD UI works): a per-image mask drawn live suppresses a real
regression on the same (name, viewport), and a category mask created on one
snapshot suppresses a regression on a different name sharing that category.
Mutation-testing applicable_masks()'s call site in render.py exposed that the
shared static site server's "changed" variant, once flipped by an earlier
test, never resets -- so a naive baseline capture this late in the suite was
already showing the regressed color, making the tests pass regardless of
whether masks did anything. Fixed with captureCleanPage(), which force-sets
the .box color back to its original value client-side, mirroring how
captureRegressedPage() already forces it to the regressed color.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NrQeh9CXrigN1cvpJuyzy4
@stolau
stolau merged commit 3ba2737 into main Jul 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant