Pest 5, and a browser suite that drives the console instead of looking at it - #6
Merged
Merged
Conversation
pestphp/pest 4.7.5 → 5.1.3, pest-plugin-browser 4.3.1 → 5.0.1, phpunit 12.5.30 → 13.3.1, and the arch/mutate/profanity plugins with them. NO TEST CHANGED. 14 Unit, 1788 Feature and 93 Browser pass unmodified, with pint and phpstan at level max clean — which is worth stating rather than assuming, because a PHPUnit major usually is not free. `composer.json` already required `php ^8.4`, which is Pest 5's floor, and `ext-sockets` is present for the browser plugin's new transport. THE BROWSER PLUGIN IS THE REASON TO DO THIS NOW. v5 replaces the npm `playwright run-server` child process with an amphp websocket client and HTTP server. The 4.x arrangement was fragile in a way that cost most of a day: a killed run leaves `.temp/playwright-server.json` pointing at a dead port and never runs `markAsStopped()`, and `fromPersisted()` dials that pointer without checking anything is listening. That path is only reached under `Parallel::isWorker()`, so it was not what hung this suite — but it is a hang rather than a failure, waiting for whoever turns on parallel testing. The suite that hung for eight hours yesterday runs in 67 seconds here.
Six tests over the two things a request-level suite structurally cannot reach.
DESTRUCTIVE ACTIONS. Every existing test of these posts straight to the route, which
proves the server handles a well-formed request and says nothing about the dialog between
the person and that request. The dialog is where the safety lives: a name typed exactly, a
button dead until it matches, the consequence in words. So the dialog is opened and the
confirm button is asserted DISABLED with the field empty, with a wrong case, and with a
prefix — three near misses, because the mobile-keyboard bug that shipped in this component
was exactly a near miss. Then the same dialog is driven to completion, because a
confirmation nobody can satisfy is its own outage.
Asserted as a STATE, not by clicking: a click on a disabled control times out waiting for
actionability, which is a five-second way of learning nothing and reads as a broken test
rather than a working guard.
CREDENTIALS SHOWN ONCE. "Once" is a claim about the SECOND page load, so it cannot be
tested anywhere but a browser: rotate, read the secret off the screen, ask for the same URL
again, require it to be gone and the page to say where it went. Falsified by removing the
reload — the assertion fails, so it is not passing vacuously.
THE SIGN-IN DOOR, END TO END. Every other test of this starts from a session built in PHP,
so the two-step form, the redirect that lands you where you were going, and the sign-out
that must work from anywhere were exercised by nothing that renders them. Three journeys:
in and out again, a refusal that is visible and keeps the email, and an interrupted visit
that resumes at the page it was asked for.
TWO TRAPS WORTH KNOWING, both cost me a wrong diagnosis first:
- `assertPathIs` does NOT wait for a navigation — it reads the URL as it stands. After a
submit it therefore describes the page being left. `assertSee` waits for the render, so
text goes first and the path second. Written the other way round these fail on a
working sign-in, which is exactly how I first wrote them.
- `press()` matches VISIBLE TEXT, and this page's h1 also says "Sign in". The submit
button is clicked by selector instead. Same family as the mobile drawer's trigger,
whose accessible name and visible text disagree.
Both new files mutation-checked: always-enabled confirm, and a redirect that forgets the
intended URL.
A validation failure is not an edge case — it is what a form does whenever somebody is in
a hurry, and it is the moment the page most needs to be clear. tests/Feature proves the
SERVER refuses: a 422 with a message under the right key. Three separate ways that correct
422 can still be a broken page, none of them visible to a request test:
- the message is drawn nowhere, so the form appears to do nothing when pressed
- the message is drawn, but not on the field it is about
- the form clears itself, punishing the mistake it just reported
All three are asserted, plus the empty state — the one screen every new customer is
guaranteed to see, and one a props test reads as `items: []` while saying nothing about
whether the page then explains itself or draws a table head over nothing.
FINDING A SERVER-ONLY RULE TOOK TWO TRIES, and the reason is worth writing down. The URL
field is `type="url"`, so Chrome refuses a malformed value with its own bubble and never
submits — a first draft used `not-a-url` and was testing the browser, not this
application. `max:500` is a rule only the server knows, which is the shape of every
server-only rule (uniqueness, ownership, entitlement) and therefore the shape worth
proving the page can display at all.
The fieldset error is asserted VISIBLE rather than present: it is a `role="alert"`
paragraph that is `hidden` until there is something to say, and `hidden` is precisely the
attribute that goes wrong — an element that never un-hides reads as "no error" to
everybody and passes any test that only checks the DOM contains the text.
Mutation-checked both ways: unwiring `error={form.errors.url}` from the Field, and pinning
the fieldset's alert `hidden`. Each turns its own test red.
Every other test in tests/Browser asserts something somebody named in advance: this text, that control, no axe violation. A layout regression is the opposite shape — nobody predicted it, and the page stays perfectly accessible and full of the right words while being visibly broken. A card grid that collapses to one column, a rail that doubles in width, a panel whose padding goes, a dark surface that loses its contrast: every one of those passes the entire suite today. Seven baselines, chosen as the console's distinct LAYOUTS rather than seven pages that happen to exist — the public door, the shell with its checklist, a long form, an empty state, each in the dark where it matters, and the shell at a phone width. Small on purpose: a baseline per page per theme per width is a cost paid on every intentional design change, and a suite that cries wolf gets regenerated without being read, which is worse than not having one. WHAT COULD NOT BE MADE STABLE IS ABSENT, and that is the trade rather than an oversight. The audit trail and the activity log are the densest tables in the console and the obvious things to pin; they render dates, so their pictures rot on a calendar. `/dashboard` went the same way — its audit-export card renders a pending count that differed between two consecutive runs. `travelTo()` is not the answer: it moves the clock in the test process, and the page is rendered by a separate server process that never hears about it. EXCLUDED FROM CI, deliberately and with the reason in the workflow. A screenshot baseline is a picture of ONE renderer: these are Chromium on macOS/arm64, and the CI runner is Linux/x64, where text rasterises differently and the pinned font is substituted. Every baseline would fail there for reasons nobody can act on. So this is a local gate today — `vendor/bin/pest --group=visual` before a design change, and READ THE DIFF rather than regenerating it. Making it a CI gate means generating the baselines inside the same container the job runs in, which is worth doing and is not this change. Mutation-checked twice, and the discrimination is the point: a `.cbx-panel` padding change fails the form alone, and a rail width change fails all four console pictures while leaving the two public ones and the phone — where the rail is not drawn — passing. Browser suite: 109 passed.
…on 8.5
TWO LINES FAILED EVERY CI LEG WHILE EVERY TEST PASSED.
Tests: 1904 passed (10177 assertions)
##[error]Process completed with exit code 1
`EveryPageComponentExistsOnceTest` opened with `use RecursiveDirectoryIterator;` and
`use RecursiveIteratorIterator;`. A Pest test file is in the global namespace, so importing
a global class does nothing, and PHP 8.5 raises a warning saying so. `phpunit.xml` sets
`failOnWarning="true"` — so a no-op import is a failing run.
WHY IT TOOK SO LONG TO SEE. The warning is raised while the suite's files are being LOADED,
before any test starts, so it belongs to no test: absent from the summary, absent from
`--log-junit` (0 errors, 0 warnings, 4 tests), and not printed by `--display-all-issues`.
`wasSuccessful()` returns TRUE; the exit code comes from a separate `failOn*` branch that
nothing narrates. Finding it meant instrumenting Pest's kernel to enumerate the result
object's warning events by hand.
It also explains the shape of the reproduction, which had me looking in the wrong place: a
single file path exits 0 and `--testsuite=Feature` exits 1, because a path loads one file
and a suite loads all of them.
`TestFilesRaiseNoWarningsTest` guards it, and names the cause in its failure message so the
next person spends a second rather than an afternoon. It sweeps only the global-namespace
test files — inside `app/` these imports are correct and necessary. Mutation-checked.
AND A REAL BUG PHPSTAN ONLY SEES ON 8.5. `DashboardController::recentProps()` indexed
`$labels[$targetId]` where the id may be null. PHP coerces a null key to `''`, so an audit
entry with no target — a sign-in, a policy change — read whatever sits under the empty
string rather than missing. Nothing puts a value there, which is the only reason it has
been harmless. The null check comes first now.
Worth recording that my "phpstan clean" was PHP 8.4 and production runs 8.5; the 8.5 leg
had been red on the port branch too, so this was not new to Pest 5.
Full suite: 1905 passed, exit 0. Pint and phpstan clean on 8.5.
`composer require --dev pestphp/pest:^5.0 -W` widened the resolve beyond the dev tree: symfony/console, finder and process went v8.1.x → v8.1.5 and polyfill-intl-normalizer v1.38.0 → v1.42.0. No dependency was added or removed — the SET is identical — but the committed SBOM no longer described the versions a build resolves, and the drift check says so on every leg. Same class as the laravel-id SBOM this week: a provenance record is only worth the last time somebody regenerated it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #5 — this branch's base is
feat/inertia-react, so the diff here is the fourcommits below and nothing from the port.
Pest 5, with no test changed
pest 4.7.5 → 5.1.3,pest-plugin-browser 4.3.1 → 5.0.1,phpunit 12.5.30 → 13.3.1, plusthe arch/mutate/profanity plugins. 14 Unit, 1788 Feature and 109 Browser pass unmodified,
with pint and phpstan at level max clean. That is worth stating rather than assuming — a
PHPUnit major usually is not free.
composer.jsonalready requiredphp ^8.4, which isPest 5's floor, and
ext-socketsis present for the browser plugin's new transport.The browser plugin is the reason to do it now: v5 replaces the npm
playwright run-serverchild process with an amphp websocket client. The suite that hung for eight hours on 4.x
runs in 67 seconds here.
Browser coverage: 36 → 109 tests
The old suite was 68
assertSeeand 27 interactions — it looked at pages and barely drovethem. Four areas, every file mutation-checked:
Destructive & credential flows. Every existing test of these posts straight to the
route, which proves the server handles a well-formed request and says nothing about the
dialog carrying the safety. The confirm button is asserted DISABLED with the field empty,
with the wrong case, and with a prefix — three near misses, because the mobile-keyboard bug
that shipped in this component was exactly a near miss. Asserted as a state, not by
clicking: a click on a disabled control times out waiting for actionability, which is a
five-second way of learning nothing.
And credentials shown ONCE — a claim about the second page load, so it cannot be tested
anywhere but a browser. Rotate, read the secret, ask for the same URL again, require it
gone.
Auth journeys. Every other test starts from a session built in PHP, so the two-step
form, the redirect that resumes an interrupted visit, and the sign-out that must work from
anywhere were exercised by nothing that renders them.
Validation & error states. A correct 422 can still be a broken page three separate
ways: drawn nowhere, drawn on the wrong field, or drawn while the form clears what was
typed. All three asserted, plus the empty state — the one screen every new customer is
guaranteed to see.
Visual regression. Seven baselines covering the console's distinct layouts. A rail
width change fails all four console pictures and correctly leaves the two public ones and
the phone passing.
Three traps this work surfaced, written into the tests
assertPathIsdoes not wait for a navigation — it reads the URL as it stands, soafter a submit it describes the page being left.
assertSeewaits. Written path-first,the sign-in tests fail on a working sign-in.
press()matches visible text, and anaria-labeloverrides it for the accessiblename. The sign-in page's h1 also says "Sign in"; the mobile drawer's trigger says "Menu"
and is labelled "Open menu". Both are clicked by selector now.
type="url"inputs are refused by Chrome before submission, so a malformed value teststhe browser rather than the app. Server-only rules (
max:500, uniqueness, ownership) arewhat exercise the server's validation path.
What is deliberately not here
Visual regression is a local gate, not a CI gate. A screenshot baseline is a picture of
one renderer: these are Chromium on macOS/arm64, and CI is Linux/x64, where text rasterises
differently and the pinned font is substituted. Every baseline would fail there for reasons
nobody can act on, and a job that fails that way is a job people learn to ignore. So CI
runs
--exclude-group=visual, with the reasoning in the workflow, and the local command isvendor/bin/pest --group=visualbefore a design change — reading the diff rather thanregenerating it.
Making it a CI gate means generating baselines inside the same container the job runs in.
Worth doing; deliberately not in the change that introduces the tests.
The baselines add 2.5 MB of PNGs to the repository.