Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ component_visual_test(
name = "editor_vrt",
browser = ":linux_browser", # browser_runtime; see docs/browser-runtime.md
shell = ":editor_shell",
matching = ":matching",
matching = {"threshold": "0.1", "maxDiffPixels": "0"},
baselines = glob(["screenshots/*.png"], allow_empty = True),
baseline_dir = "screenshots",
)
Expand All @@ -50,9 +50,15 @@ component_visual_test(
| `base_url` / `base_url_env` | Existing application endpoint, replacing `server` or `shell` |
| `playwright` | Optional reusable runtime target grouping client packages; minimum 1.63.0 |
| `browser` | Required VRT runtime containing declared Linux Chromium, Node, libraries, and fonts |
| `matching` | Compiled VRT comparison policy; render settings stay in `.visual.tsx` |
| `matching` | Declared comparison options or a compiled policy module; render settings stay in `.visual.tsx` |

For example, compile this `matching.ts` module:
Both component and page VRT accept the same `matching` dictionary. Values are
JSON numeric strings because Starlark has no floating-point values; for example,
`matching = {"maxDiffPixelRatio": "0.01"}` allows a one-percent mismatch budget.
The options are declared action inputs, without an environment adapter or a
TypeScript compilation target. See the [matching reference](docs/api.md#vrt-matching).

Existing compiled `matching.ts` modules remain supported:

```ts
import type {VisualMatching} from '@rules-web-e2e/vrt'
Expand Down
20 changes: 19 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Both `component_visual_test` and `visual_test` accept:

| Attribute | Default | Contract |
| -------------- | ------------------- | ------------------------------------------------------------------ |
| `matching` | Exact pixel budget | Compiled module exporting `VisualMatching` |
| `matching` | Exact pixel budget | Dictionary of JSON numeric strings, or compiled `VisualMatching` module |
| `baselines` | `[]` | Existing PNG input labels |
| `baseline_dir` | `"__screenshots__"` | Package-relative directory exclusively owned by this visual target |

Expand Down Expand Up @@ -110,6 +110,24 @@ only the documented pinned version has been exercised by this repository's CI.

## VRT matching

Both `component_visual_test` and `visual_test` accept declared comparison
options directly:

```starlark
matching = {"threshold": "0.1", "maxDiffPixelRatio": "0.01"}
```

Use JSON numeric strings for all dictionary values (Starlark has no floats).
The rule generates a dependency-free ESM policy included in the action's
runfiles. Each target can supply its own options without reading ambient
environment variables or compiling an adapter. Omitted options retain the
defaults below; `matching = {}` uses those defaults too. String contents are
parsed as JSON, never evaluated as JavaScript, and go through the same numeric
validation as compiled policies. Malformed or invalid values fail before capture.

For policies that need a compiled module, the existing label form remains
supported:

```ts
import type {VisualMatching} from '@rules-web-e2e/vrt'

Expand Down
3 changes: 2 additions & 1 deletion docs/component-vrt.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ The consumer build owns strict typechecking, transpilation, providers, CSS,
fonts, and generated assets. The runner does not compile the application.
See [the complete example](../examples/react/BUILD.bazel).

A compiled `matching` module exports `VisualMatching`: configure per-pixel
`matching` accepts a dictionary of JSON numeric strings or a compiled module
exporting `VisualMatching`, just like page VRT: configure per-pixel
`threshold` and either `maxDiffPixels` or `maxDiffPixelRatio`. Defaults use
Playwright's pixelmatch comparator with threshold 0.1 and zero mismatched pixels.
Viewport, language, theme, density, and capture hooks remain visual options.
Expand Down
3 changes: 2 additions & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,8 @@ see the [React gallery](../examples/react/gallery.tsx).

## VRT matching and updates

Compile a module exporting `VisualMatching` and pass it as `matching`.
Pass declared options such as `matching = {"maxDiffPixelRatio": "0.01"}`,
or compile a module exporting `VisualMatching` and pass its label as `matching`.
Use `threshold` for per-pixel color tolerance, and either `maxDiffPixels` or
`maxDiffPixelRatio` for the allowed mismatch budget. Defaults are threshold
0.1 and zero mismatched pixels. See the [matching reference](api.md#vrt-matching).
Expand Down
3 changes: 2 additions & 1 deletion examples/react/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ visual_test(
allow_empty = True,
),
config = ":native_config",
matching = {"threshold": "0.1", "maxDiffPixels": "0"},
tests = ":native_visual_specs",
)

Expand Down Expand Up @@ -160,7 +161,7 @@ component_visual_test(
["__screenshots__/*.png"],
allow_empty = True,
),
matching = ":matching",
matching = {"threshold": "0.1", "maxDiffPixels": "0"},
shell = ":app_shell",
)

Expand Down
1 change: 1 addition & 0 deletions internal/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ load(":remote.bzl", "linux_platform")

exports_files([
"browser.bzl",
"matching.bzl",
"remote.bzl",
])

Expand Down
10 changes: 9 additions & 1 deletion internal/browser.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

load("@aspect_rules_js//js:defs.bzl", "js_library", "js_test")
load("//playwright:defs.bzl", "BrowserRuntimeInfo", "PlaywrightInfo", "runfile")
load(":matching.bzl", "matching_config")
load(":remote.bzl", "remote_browser_test")

ShellInfo = provider(fields = ["directory", "entry_point"])
Expand Down Expand Up @@ -123,7 +124,7 @@ def browser_test(
fail("Remote browser tests require explicit env values instead of env_inherit")
if browser and base_url_env and base_url_env not in env:
fail("Remote browser base_url_env must have an explicit env value")
if not visual and matching:
if not visual and matching != None:
fail("matching is only supported by visual targets")
if execution_timeout_seconds <= 0:
fail("execution_timeout_seconds must be positive")
Expand All @@ -138,6 +139,13 @@ def browser_test(
fail("base_url must not be empty")
if base_url_env and base_url_env not in env and base_url_env not in env_inherit:
env_inherit = env_inherit + [base_url_env]
if type(matching) == "dict":
matching_config(
name = name + "_matching",
options = matching,
tags = tags,
)
matching = ":" + name + "_matching"
js_library(name = name + "_sources", srcs = baselines, data = data)
_inputs(
name = name + "_inputs",
Expand Down
21 changes: 21 additions & 0 deletions internal/matching.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
"""Generate an ESM comparison policy from declared Bazel options."""

def _matching_config_impl(ctx):
for name in ctx.attr.options:
if name not in ["threshold", "maxDiffPixels", "maxDiffPixelRatio"]:
fail("Unknown VRT matching option: " + name)
module = ctx.actions.declare_file(ctx.label.name + ".mjs")

# Starlark has no floats. Parse JSON numeric strings without evaluating code;
# the shared runtime validator enforces numeric types, ranges, and budgets.
entries = [
"%s: JSON.parse(%s)" % (json.encode(name), json.encode(value))
for name, value in ctx.attr.options.items()
]
ctx.actions.write(module, "export default {" + ", ".join(entries) + "};\n")
return [DefaultInfo(files = depset([module]), runfiles = ctx.runfiles(files = [module]))]

matching_config = rule(
implementation = _matching_config_impl,
attrs = {"options": attr.string_dict()},
)
22 changes: 22 additions & 0 deletions runtime/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
load("@aspect_rules_js//js:defs.bzl", "js_library", "js_test")
load("@aspect_rules_js//npm:defs.bzl", "npm_package")
load("@aspect_rules_ts//ts:defs.bzl", "ts_project")
load("//internal:matching.bzl", "matching_config")
load("//playwright:defs.bzl", "playwright_runtime")

ts_project(
Expand Down Expand Up @@ -213,6 +214,27 @@ js_test(
entry_point = "built-inputs.test.js",
)

MATCHING_FIXTURES = {
"matching_defaults": {},
"matching_ratio": {"threshold": "0.2", "maxDiffPixelRatio": "1e-2"},
"matching_count": {"maxDiffPixels": "5"},
"matching_empty": {"threshold": ""},
"matching_nonfinite": {"threshold": "1e999"},
"matching_null": {"threshold": "null"},
"matching_out_of_range": {"maxDiffPixelRatio": "1.1"},
"matching_fractional_count": {"maxDiffPixels": "0.5"},
"matching_conflicting_budgets": {"maxDiffPixels": "1", "maxDiffPixelRatio": "0.01"},
}

[matching_config(name = name, options = options, testonly = True) for name, options in MATCHING_FIXTURES.items()]

js_test(
name = "matching_config_test",
size = "small",
data = ["package.json", ":typecheck"] + [":" + name for name in MATCHING_FIXTURES],
entry_point = "matching-config.test.js",
)


js_test(
name = "capture_browser_test",
Expand Down
27 changes: 27 additions & 0 deletions runtime/matching-config.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import assert from 'node:assert/strict'
import {test} from 'node:test'
import {screenshotMatching} from './matching.js'

test('declared comparison policies use the same defaults and validation as compiled modules', async () => {
for (const [name, expected] of [
['matching_defaults', {threshold: 0.1, maxDiffPixelRatio: 0}],
['matching_ratio', {threshold: 0.2, maxDiffPixelRatio: 0.01}],
['matching_count', {threshold: 0.1, maxDiffPixels: 5}],
] as const) {
const {default: matching} = await import(`./${name}.mjs`)
assert.deepEqual(screenshotMatching(matching), expected)
}
for (const name of [
'matching_empty',
'matching_nonfinite',
'matching_null',
'matching_out_of_range',
'matching_fractional_count',
'matching_conflicting_budgets',
]) {
await assert.rejects(async () => {
const {default: matching} = await import(`./${name}.mjs`)
screenshotMatching(matching)
}, {message: /JSON|VRT matching option|Choose maxDiffPixels/})
}
})
Loading