What
x verify's e2e step prints a green check while its only test skips. At the gate level a
skipped e2e is indistinguishable from a passing one.
Measured on a clean scaffold from the published @ultimat3/core@20.1.0, bun 1.4.2:
bunx create-ultimate demo --no-git && cd demo && bin/setup && bin/check
- The step reports
✓ e2e 46ms, and --json gives "skipped": false, "findings": [].
- The step contains exactly one test file,
apps/web/app/posts/page.e2e.test.ts, with
exactly one test. Running that file directly gives 0 pass, 1 skip, 0 fail.
- The reason survives only inside the test's name, visible only through a junit reporter:
e2e · /posts renders offline (no browser driver: x build --target static && x e2e)
Whole suite for context: 113 pass, 1 skip, 0 fail, 258 expect() calls across 32 files.
That single skip is this test. Of the 20 verify steps, roadmap is the one honest skip —
it prints - roadmap 0ms and reports "skipped": true.
Why it matters
This is the greenfield story a new user gets: a fresh scaffold on a plain VM with no
docker and no env comes up green in ~5-8s for bin/setup and ~7s for bin/check. That
part is real and was verified with docker, docker-compose, podman, psql,
postgres, pg_ctl, initdb, pg_dump and pg_isready all shadowed by stubs that
exit 127 — none ever fired, so "no docker required" is proven rather than assumed.
But the customer's green gate includes a lane that has never run. They will believe their
end-to-end path is covered when nothing has exercised it, and the first time it matters is
the first time it breaks.
The rule this already has, one layer up
The plan slice that commissioned this work stated the principle for the database case:
a missing embedded binary is X_PG_MISSING with a fix: command, never a silent skip.
That was satisfied for Postgres and left open for the browser driver. The same treatment
belongs here.
The X_BUN_VERSION gate is the model to copy, and it is already right — on a box with bun
1.3.14, bunx create-ultimate refuses before writing any file:
X_BUN_VERSION
cause: Bun 1.3.14 is older than the required 1.4.0
fix: bun upgrade
docs: https://github.com/developerz-ai/ultimate/wiki/Error-Codes
A cause, a fix: line, and a non-zero exit. That is what a missing browser driver should
produce.
Fix shape
Either the e2e step reports honestly as skipped when no browser driver is present — so
--json says "skipped": true the way roadmap does, and the summary line counts it —
or it fails with an error code and a fix: line naming the command that installs the
driver. What it must not do is print a check mark for a lane that did not execute.
Related, same shape, lower severity
live passes without ever opening a database connection. Its 4 tests assert SQL text via
sourceFor(...).toSQL() plus an authz denial, and the test's own comment says "The denial
lands before any SQL executes, which is why this needs no database." That is a fair test of
what it tests, but a green live is not evidence that embedded Postgres works, and the
step's name implies otherwise.
Also worth noting for whoever picks this up: TEST_REPLICATION_URL no longer appears
anywhere in node_modules/@ultimat3/ in 20.1.0, so any doc or plan still describing the
verify-tests.ts replication mechanism is stale.
What
x verify'se2estep prints a green check while its only test skips. At the gate level askipped e2e is indistinguishable from a passing one.
Measured on a clean scaffold from the published
@ultimat3/core@20.1.0, bun 1.4.2:✓ e2e 46ms, and--jsongives"skipped": false, "findings": [].apps/web/app/posts/page.e2e.test.ts, withexactly one test. Running that file directly gives
0 pass, 1 skip, 0 fail.Whole suite for context:
113 pass, 1 skip, 0 fail, 258 expect() calls across 32 files.That single skip is this test. Of the 20 verify steps,
roadmapis the one honest skip —it prints
- roadmap 0msand reports"skipped": true.Why it matters
This is the greenfield story a new user gets: a fresh scaffold on a plain VM with no
docker and no env comes up green in ~5-8s for
bin/setupand ~7s forbin/check. Thatpart is real and was verified with
docker,docker-compose,podman,psql,postgres,pg_ctl,initdb,pg_dumpandpg_isreadyall shadowed by stubs thatexit 127— none ever fired, so "no docker required" is proven rather than assumed.But the customer's green gate includes a lane that has never run. They will believe their
end-to-end path is covered when nothing has exercised it, and the first time it matters is
the first time it breaks.
The rule this already has, one layer up
The plan slice that commissioned this work stated the principle for the database case:
a missing embedded binary is
X_PG_MISSINGwith afix:command, never a silent skip.That was satisfied for Postgres and left open for the browser driver. The same treatment
belongs here.
The
X_BUN_VERSIONgate is the model to copy, and it is already right — on a box with bun1.3.14,
bunx create-ultimaterefuses before writing any file:A cause, a
fix:line, and a non-zero exit. That is what a missing browser driver shouldproduce.
Fix shape
Either the
e2estep reports honestly as skipped when no browser driver is present — so--jsonsays"skipped": truethe wayroadmapdoes, and the summary line counts it —or it fails with an error code and a
fix:line naming the command that installs thedriver. What it must not do is print a check mark for a lane that did not execute.
Related, same shape, lower severity
livepasses without ever opening a database connection. Its 4 tests assert SQL text viasourceFor(...).toSQL()plus an authz denial, and the test's own comment says "The deniallands before any SQL executes, which is why this needs no database." That is a fair test of
what it tests, but a green
liveis not evidence that embedded Postgres works, and thestep's name implies otherwise.
Also worth noting for whoever picks this up:
TEST_REPLICATION_URLno longer appearsanywhere in
node_modules/@ultimat3/in 20.1.0, so any doc or plan still describing theverify-tests.tsreplication mechanism is stale.