FEAT: render-baseline harness — A/B render diff across Remotion bumps (#23 part 2) - #64
Merged
Conversation
…#23 part 2) tests/render-baseline/: fixed 6-block composition (interpolate/spring, Sequence+text in a bundled font, OffthreadVideo on a committed testsrc clip, Audio, TransitionSeries with lib/transitions glitch+zoomBlur, lib/components) with duration from calculateMetadata. No system fonts, no network, no Math.random. scripts/render-baseline.mjs: render <label> (bundle once, one still per block at --scale=0.25 --gl=swangle), diff <a> <b> (pixelmatch perceptual + exact counts, diff PNGs, report.md/json, exit 1 over --threshold, default 0), install <version>, and ab <version> for local one-shot use. .github/workflows/render-baseline.yml: on PRs touching package.json under templates/examples/showcase, tests/render-baseline, lib/ or the script — renders base-branch pin and PR pin on the same runner, diffs, comments the table, uploads stills, fails on any difference so a human looks. workflow_dispatch takes a candidate version. Verified locally: same-version render is byte-identical (0 px on all six); 4.0.425 → 4.0.518 (93 patches) is also 0 px on all six. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…acks the harness Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of #23 — the unlock for routine Remotion bumps.
Design
Render a fixed composition twice on the same machine — at the current pin and at the candidate — and diff the two. No committed "golden" PNGs: those would be noisy across OS/GPU/Chrome builds and rot with every bump. This only ever asks "did this bump change output?"
What's in it
tests/render-baseline/— small pinned Remotion project; one composition, six 30-frame blocks:interpolate/spring/Easing,Sequence+ text in a bundled font,OffthreadVideoon a committed 1 stestsrcclip,Audio,TransitionSeriesusinglib/transitions(glitch,zoomBlur), andlib/components(AnimatedBackground,Label,Vignette,FilmGrain). Duration viacalculateMetadata. Bundled DejaVu-subset font (14 KB), nosystem-ui, no network. Assets total ~50 KB.scripts/render-baseline.mjs—render <label>(bundle once, one still per block at--scale=0.25 --gl=swangle),diff <a> <b>(pixelmatch perceptual count gates; exact count reported alongside for later calibration; diff PNGs +report.md/report.json; exit 1 above--threshold, default 0),install <version>, andab <version>for a local one-shot that restores the pin afterwards..github/workflows/render-baseline.yml— on PRs touchingpackage.jsonundertemplates/,examples/,showcase/, plustests/render-baseline/,lib/, or the script: renders base-branch pin and PR pin on the same runner, diffs, posts the table as a PR comment, uploads stills + diffs as artifacts, fails on any difference so a human reviews.workflow_dispatchtakes a candidate version.Verified locally (macOS)
ab 4.0.518(4.0.425 → 4.0.518, 93 patches) → 0 px on all six, perceptual and exact. Pin restored.Gate stays at "any diff → fail" until a few real bumps have shown what noise looks like.