Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/actiond-production.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,18 @@ on:
- 'runtime/**'
- 'internal/**'
- 'playwright/**'
- 'examples/react/**'
- 'MODULE.bazel'
- 'package.json'
- 'pnpm-lock.yaml'
- '.github/workflows/actiond-production.yaml'

permissions:
contents: read

jobs:
vrt:
name: VRT tests (Linux amd64)
runs-on: ubuntu-24.04
timeout-minutes: 40
env:
Expand Down
77 changes: 0 additions & 77 deletions .github/workflows/actiond-prototype.yaml

This file was deleted.

35 changes: 0 additions & 35 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,40 +64,5 @@ jobs:
- name: Run host browser suites without Docker
working-directory: examples/react
run: bazelisk test //:e2e_test //:component_test //:remote_integration_test //:native_config_test --test_output=errors

component-vrt:
name: VRT tests (Linux amd64)
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: bazel-contrib/setup-bazel@c5acdfb288317d0b5c0bbd7a396a3dc868bb0f86 # 0.19.0
with:
bazelisk-cache: true
repository-cache: true
- name: Preload declared images and verify credential-free browser startup
run: |
bazelisk build //runtime:images
manifest="$(bazelisk info bazel-bin)/runtime/images.json"
jq -r '.images[] | [.image, (.platform // "")] | @tsv' "$manifest" |
while IFS="$(printf '\t')" read -r image platform; do
if [ -n "$platform" ]; then
docker pull --platform "$platform" "$image"
else
docker pull "$image"
fi
done
bazelisk test //runtime:preload_browser_test --test_output=errors
- name: Verify VRT with only declared tools and Docker
run: bash tests/preloaded-vrt.sh
- name: Verify generated capture bounds and pixels
run: bazelisk test //runtime:capture_browser_test --test_output=errors
- name: Run visual tests and container isolation coverage
working-directory: examples/react
run: bazelisk test //:visual_test //:component_visual_test //:native_visual_test //:network_isolation_test --test_output=errors
- name: Upload visual test artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: browser-tests
path: examples/react/bazel-testlogs/
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,5 @@ rules_web_e2e-*.tar.gz
release_notes.txt

.playwright-browsers/

examples/react/runtime.tar
23 changes: 13 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Bazel rules for Playwright E2E, component browser tests, and visual regression
testing (VRT). E2E and component tests use host Playwright browsers; only VRT
uses Testcontainers and a pinned Linux Chromium image.
runs in isolated actiond Linux actions with caller-owned browser runtimes.
Bring compiled tests, your own server, or a built shell/template. Your build
owns typechecking and bundling; the rules own execution and baseline updates.

Expand Down Expand Up @@ -33,6 +33,7 @@ component_browser_test(

component_visual_test(
name = "editor_vrt",
browser = ":linux_browser", # browser_runtime; see docs/browser-runtime.md
shell = ":editor_shell",
matching = ":matching",
baselines = glob(["screenshots/*.png"], allow_empty = True),
Expand All @@ -47,7 +48,8 @@ component_visual_test(
| `server` | Compiled adapter returning a ready URL and cleanup callback |
| `shell` | Built HTML/JS/CSS directory plus its entry point; served without a bundler |
| `base_url` / `base_url_env` | Existing application endpoint, replacing `server` or `shell` |
| `playwright` | Optional reusable runtime target grouping client packages and a pinned image; minimum 1.63.0 |
| `playwright` | Optional reusable runtime target grouping client packages; minimum 1.63.0 |
| `browser` | Required VRT runtime containing declared Linux Chromium, Node, libraries, and fonts |
| `matching` | Compiled VRT comparison policy; render settings stay in `.visual.tsx` |

For example, compile this `matching.ts` module:
Expand All @@ -74,21 +76,22 @@ attributes; see [migration](docs/getting-started.md#migrating-from-100).

## Try it

Install Bazelisk and provision host Chromium first (with the locked Playwright version).
Start Docker and [preload the pinned images](docs/api.md#vrt-image-manifest-and-ci-preloading)
for VRT. From this checkout:
Provision host Chromium for interaction tests. For VRT, supply the example's
`runtime.tar` from your image build and configure a patched actiond worker using
[the execution guide](docs/actiond.md).

```sh
export PLAYWRIGHT_BROWSERS_PATH="$(pwd)/.playwright-browsers"
pnpm exec playwright install chromium
cd examples/react
bazelisk test //:e2e_test //:component_test //:component_visual_test
bazelisk run //:component_visual_test.update
bazelisk test //:e2e_test //:component_test
bazelisk test --config=vrt //:component_visual_test
bazelisk run --config=vrt //:component_visual_test.update
```

Review PNG changes before committing. Browser targets are manual, local, and
uncached; select them explicitly in CI. Screenshot baselines are validated on
Linux amd64. Each VRT target owns a separate baseline directory.
Review PNG changes before committing. Each VRT target owns its baseline directory.
VRT capture/comparison actions are cacheable; baseline application stays local.
Host browser targets remain manual, local, and uncached.

See [development and releases](docs/development.md) for build checks, hooks,
and BCR publishing.
Expand Down
4 changes: 2 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
| Document | Read it for |
| --------------------------------------------------------- | ------------------------------------------------------------- |
| [Architecture](architecture.md) | Ownership, input staging, and runtime boundaries |
| [Testcontainers and VRT stability](testcontainers-vrt.md) | Container lifecycle, reproducibility, and isolation limits |
| [VRT on actiond](actiond.md) | Worker setup, declared runtimes, and isolation limits |
| [Visual testing design](visual-testing-design.md) | Shared visual modules, capture phases, and baseline ownership |
| [Scope and roadmap](oss-browser-testing-plan.md) | Implemented capabilities and possible extensions |
| [Development and releases](development.md) | Repository checks, hooks, and release automation |

See [host browser provisioning and AGI/FormatJS migration](host-browsers.md) for
the VRT-only Testcontainers execution model.
host provisioning and the VRT actiond migration.
2 changes: 2 additions & 0 deletions docs/actiond-evaluation.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# actiond evaluation for VRT

Historical evaluation; the implemented path is described in [VRT on actiond](actiond.md).

Source review on 2026-09-13 at upstream commit
[`8a42c3d`](https://github.com/hermeticbuild/actiond/tree/8a42c3d481df3a1bf1b80e95a9bb991a207fc035).
The initial review below is followed by a [runnable prototype](../experiments/actiond/README.md).
Expand Down
132 changes: 39 additions & 93 deletions docs/actiond-migration.md
Original file line number Diff line number Diff line change
@@ -1,93 +1,39 @@
# Replace Testcontainers with actiond

Active implementation plan. The existing Chromium prototype in PR #27 passes
through actiond's Linux amd64 VM after applying `actiond-advice.patch` (upstream
actiond PR #48). Production VRT still uses Testcontainers.

## Execution contract

The fixture server, Playwright test runner, Chromium, and screenshot comparison
execute together as a Linux Bazel action. Browser files, libraries, fonts, Node,
test code, assets, and baselines are declared inputs. The executor provides
isolation; no Docker daemon, image pulls, or reaper run inside the action.

Callers may construct their own OCI image. Packaging must convert its pinned
contents into declared runtime files before execution, including an explicit
browser executable and architecture. Building Chromium from source is optional.
Keep the rule compatible with REAPI rather than depending on actiond's CLI inside
the test runner.

Baseline capture produces declared downloadable outputs. A local `.update`
wrapper applies successful captures to the source tree using the existing
destination validation. Failed or empty captures must not replace baselines.

## Implementation sequence

1. Add declared browser runtime metadata and direct browser launch support.
Exercise the existing gallery and native screenshot runners, not just the
standalone HTML prototype.
2. Package a caller-owned pinned runtime through Bazel, removing the prototype's
manual Docker extraction prerequisite from the supported execution path.
3. Add Linux execution actions for comparison and baseline capture, plus the
local baseline application wrapper. Select amd64 explicitly; an ARM64 worker
must not silently produce shared amd64 baselines.
4. Validate on a patched actiond VM with local fallback disabled. Check artifacts,
baseline updates, failures, timeouts, cancellation, and network isolation.
5. Provide concrete AGI and FormatJS callsite migrations, including built assets,
caller fixture servers, custom Playwright configuration, and native specs.
External services must become declared local fixtures or have an explicitly
documented unsupported migration case; the VM has no external network.
6. Remove Testcontainers, Ryuk, their patches, and Docker-specific VRT plumbing
once the replacement passes these checks. Update examples, CI, and docs.

## Completion evidence

- Both `visual_test` and `component_visual_test` compare and update screenshots
through actiond using the production runner.
- Browser runtime acquisition is pinned and separate from offline execution.
- Screenshot outputs and reports survive remote execution and failed tests.
- Baseline application is local, explicit, and refuses failed or empty captures.
- Host E2E/component browser tests continue to work independently.
- AGI and FormatJS migration examples describe the actual supported interface.
- Any necessary actiond changes remain isolated patches with reproductions and
upstream status. The current required patch enables memory-advice syscalls;
no browser-driven relaxation of the action sandbox has been established as
necessary.

Keep dependent PRs stacked with ordinary Git and descriptions concise. Do not
mark the goal complete based solely on the standalone prototype passing.

## Current progress

- Both production VRT modes capture, apply baselines, and compare through the
public rules in actiond's Linux VM (CI run 34779979065).
- Capture jobs generated by the public rules also pass process-isolation tests
with the archive-built runtime, without hand-editing their descriptors.
- Public rules with a declared browser now create remote comparison/capture
actions and local result consumers. Bazel analysis verifies Linux amd64
constraints and declared runtime paths. Explicit remote mnemonic strategies
and disabled fallback keep these actions on the worker.
- Failed and empty captures preserve source baselines; downloaded failure
reports survive through the local test wrapper. Filesystem and subprocess
regression tests pass.
- Public-rule isolation and deliberate post-capture failure jobs pass under the
actiond process runner: Node gets `ENETUNREACH`, the browser cannot access the
external address, and failed captures retain diagnostic PNGs without exposing
eligible baseline outputs. CI also checks empty captures and mismatched
reference PNGs through the real local commands; VM results remain pending.
- The separate actiond `input-rootfs`/`input-rootfs-env` patch passes actiond's
full build, unit tests, and the public-rule production VM workflow.
The hand-staged VM diagnostic hit an undeclared npm file beneath a nested
Bazel package; CI now runs the public rules and their declared runfiles.
- `browser_runtime_archive` unpacks a caller-produced flattened runtime tar
through a declared Python toolchain. It normalizes image-root links, preserves
executables, and rejects dangling links and unflattened OCI whiteouts.
- `browser_runtime_oci` verifies and applies declared OCI layers offline, then
materializes the selected runtime subtree. Layer/whiteout/hardlink tests pass,
and the real OCI-built runtime matches the archive-built browser, Node, and
fixture font. The CI example now constructs and passes an OCI image target.

Next: verify the full public-rule workflow in the VM, migrate concrete
AGI/FormatJS callsites, and remove the legacy backend. Baseline application,
failures, cancellation, and isolation still need end-to-end VM coverage through
the public rules. OCI support itself does not complete the migration.
# Actiond migration

VRT now runs through actiond Linux amd64 actions with caller-owned runtime files
or OCI images. Testcontainers, Ryuk, the control relay, image manifests, and their
preload/patch dependencies are removed. Host E2E/component tests retain host
Chromium and host networking.

The changes are stacked as #27 (initial proof), #28 (declared runtime/actions),
and #29 (backend replacement).

## Validation

[Production VM run 34783532975](https://github.com/perplexityai/rules_web_e2e/actions/runs/34783532975)
passes native/component capture and comparison, local baseline updates, network
isolation, failed/empty captures, screenshot diffs, deadlines, cancellation, and
worker recovery. The native fixture launches a real Bazel `js_binary` server.

Bazel 8.6/9.2 build/tests and host browser suites pass on Linux and macOS. The
actual FormatJS editor gallery builds its Vite/StyleX assets, captures all eight
screenshots, applies references locally, and compares them in actiond process
isolation. See [consumer migration](host-browsers.md) for the actual interface
and the AGI wrapper requirements; this is not a claim that their full CI
migrations have landed.

## Caller changes

- Supply `browser` from a declared `browser_runtime`; move the former runtime
`image` setting into the caller's OCI build and `browser_runtime_oci` target.
- Configure a patched amd64 worker using [the execution guide](actiond.md).
Set `target_platform` when native dependencies need additional ABI constraints.
- Keep compiled specs, built shells, matching, server data, and baseline ownership.
`$(rootpath ...)` expands in explicit environment values, including JSON.
- Replace external VRT services with declared local fixtures. Live deployed
checks stay in host E2E targets; inherited environment and origin exceptions
do not carry over.

The memory-advice patch is upstream PR #48; the declared-rootfs patch remains
isolated locally for upstreaming. Native macOS VM execution and cross-architecture
pixel equivalence remain unvalidated. These baselines require a Linux amd64 worker.
Loading
Loading