Hold the console's layout on every pull request - #7
Merged
Conversation
A screenshot baseline is a picture of ONE renderer. The committed ones were drawn by Chromium on macOS/arm64; CI is Linux/x64, where text rasterises differently and the font the assertion pins is substituted — so every baseline fails there for a reason nobody can act on, which is why the visual group has been excluded from CI since it was introduced. This is the missing half: a manual workflow that draws them on the runner. Run it, download the artifact, commit what is in tests/.pest/snapshots, and the layout can be held on every pull request. The first run reports INCOMPLETE rather than passing — a snapshot that did not exist is not a pass — so the upload is `if: always()`.
The visual group has been excluded from CI since it was introduced, for a reason that was
never about flakiness: a screenshot baseline is a picture of ONE renderer. The committed
pictures were drawn by Chromium on macOS/arm64 and CI is Linux/x64, where text rasterises
differently and the font the assertion pins is substituted — so every baseline failed
there for reasons nobody could act on.
The runner draws them now. `visual-baselines.yml` mints the set on ubuntu-latest and
uploads it; these seven are that artifact. CI runs the whole suite, visual included.
Two things the first attempt at that workflow got wrong, both worth keeping written down:
- `--update-snapshots` is REQUIRED to mint a baseline. Without it a missing snapshot is
an assertion failure, not a quiet creation — which is the right default everywhere
else, because it means no ordinary run can mint the picture it was about to be judged
against. This job is the one place that minting is the point. It looked automatic
locally only because a missing snapshot reports as INCOMPLETE rather than as a pass.
- `upload-artifact` skips dotted paths, and `.pest` begins with a dot. The first run
uploaded nothing and said so only as "No files were found with the provided path".
The temporary push trigger goes with this commit; the workflow is on the default branch
after this merge, so `workflow_dispatch` works from here on.
NOTE FOR ANYONE RUNNING `--group=visual` ON A MAC: it will show diffs on all seven and
none of them mean anything. Run it to LOOK at a page; to change what is committed, run the
workflow and commit what it uploads. Said in the test file too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The visual group runs in CI now, judged against baselines drawn by the runner itself.
A screenshot baseline is a picture of one renderer. The pictures committed earlier were drawn by Chromium on macOS/arm64; CI is Linux/x64, where text rasterises differently and the pinned font is substituted — so they could never have passed there.
visual-baselines.ymlmints the set onubuntu-latestand uploads it; the seven baselines here are that artifact.Two traps the first attempt hit, both now written into the workflow:
--update-snapshotsis required to mint a baseline. Without it a missing snapshot is an assertion failure rather than a quiet creation — the right default everywhere else, since it stops an ordinary run minting the picture it is about to be judged against. This job is the one place minting is the point.upload-artifactskips dotted paths, and.pestbegins with a dot. The first run uploaded nothing and reported only "No files were found".Running
--group=visualon a Mac will now show diffs on all seven and none of them mean anything — run it to look at a page, and use the workflow to change what is committed. That is said in the test file as well as here.