Stabilize Mobile Visual Snapshots#2936
Conversation
- Standardize mobile projects in playwright.config.ts (Pixel 5, iPhone 12) - Refactor mobile tests to use toHaveScreenshot assertions - Remove manual device emulations in favor of project configuration - Expand disableAnimations to cover View Transitions and scroll behavior - Ensure fonts are fully loaded globally before taking snapshots
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
🤖 AI Technical AuditANTI-AI-SLOPThe refactoring of visual regression tests is generally high-quality and aligns with Playwright best practices. However, the FINAL RECOMMENDATIONapproved with suggestions Review automatically published via RepoAuditor. |
|
🚀 Deployment Details (Last updated: Jun 26, 2026, 3:06 PM PST) 🚀 Pushed to gh-pages; publish in progress
|
- Add 'Mobile Chrome' and 'Mobile Safari' projects to playwright.config.ts - Migrate tests to use expect(page).toHaveScreenshot() for deterministic comparison - Remove manual viewport and device emulations in favor of project settings - Add view-transition-name: none and scroll-behavior: auto to disableAnimations - Await document.fonts.ready in page fixture to prevent typography drift - Relocate and rename snapshots to match new project naming conventions
- Standardize mobile projects (Mobile Chrome, Mobile Safari) - Migrate all visual tests to use the custom visual fixture (fixtures/visual.ts) - Implement global font-loading and animation suppression in the fixture - Replace manual screenshot paths with toHaveScreenshot assertions - Relocate snapshots to spec-specific directories with project-specific naming - Skip mobile tests in desktop projects to prevent dimension mismatches
- Use lowercase project names (mobile-chrome, mobile-safari) - Use hyphenated snapshot names to match Playwright discovery - Migrate all visual tests to custom fixture with font-ready guards - Relocate and rename snapshots for project-based comparison - Skip mobile tests in desktop project runs
arii
left a comment
There was a problem hiding this comment.
Issue audit result
Recommendation: Completed, close
Reason:
The PR implements standard visual test approaches, replacing manual page.screenshot() with expect(page).toHaveScreenshot() across mobile tests. It updates playwright.config.ts to standard 'chromium', 'mobile-chrome', and 'mobile-safari' projects. It introduces test.skip(({ isMobile }) => !isMobile, ...) where appropriate and disables CSS animations globally to reduce flakiness. Snapshot file names now match the required standard. All CI steps including visual tests succeeded.
Implementation evidence:
- Files checked:
playwright.config.ts,tests/affiliate-mobile.spec.ts,tests/blog-post-mobile.spec.ts,tests/fixtures/visual.ts,tests/research-mobile.spec.ts,tests/search_mobile.spec.ts,tests/utils/playwright-helpers.ts,tests/verify_ux_consistency.spec.ts,tests/visual-affiliate.spec.ts - PRs checked: #2936
- Routes checked: N/A
- Tests or validation: CI tests successfully passed on this PR, including Build & E2E checking snapshots.
Remaining work:
None.
|
@jules-fix-ci |
|
🤖 Jules is on it! Initialized autonomous repair session ( |
🤖 AI Technical AuditANTI-AI-SLOPThe refactoring is generally sound; however, FINAL RECOMMENDATIONApproved with Minor Changes DEFINITION OF DONE
Review automatically published via RepoAuditor. |
…t configs (#2952) Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: Ariel Anders <anders.ariel@gmail.com>
🤖 AI Technical AuditANTI-AI-SLOP
FINAL RECOMMENDATIONApproved with Minor Changes DEFINITION OF DONE
Review automatically published via RepoAuditor. |
arii
left a comment
There was a problem hiding this comment.
PR Review
Summary: This PR aims to stabilize mobile visual snapshots by adopting native Playwright snapshot features (toHaveScreenshot) instead of manual file path string manipulations. It removes hardcoded viewports in favor of leveraging Playwright project configurations and adds some fixes for view-transition animations.
Findings:
- CI Failure: The
Build & E2Echeck failed. Looking at the diff, this PR introduces-linux.pngsnapshots (e.g.,tests/verify_ux_consistency.spec.ts-snapshots/detail-page-v2-chromium-linux.png). This directly violates the repository context ("Playwright visual regression snapshots are structured using a custom snapshotPathTemplate... Default Playwright OS suffixes (like -linux.png) are obsolete and should be purged"). - Flawed Snapshot Naming: The PR is actively adding OS-suffixed snapshots which likely broke the CI environment that doesn't expect them, and conflicts with the stated goal of using
snapshotPathTemplate. - Refactoring: The shift to
expect(page).toHaveScreenshot(name)is correct and an improvement overpage.screenshot({ path: ... }).
Recommendation: Not Approved. The PR introduces obsolete -linux.png suffixed snapshots which violates the repository's Playwright configuration constraints and fails the CI. Please remove the OS suffixes from the snapshot names to rely on the project-level directory grouping.
arii
left a comment
There was a problem hiding this comment.
ANTI-AI-SLOP
The Build & E2E check failed. This PR introduces -linux.png snapshots which directly violates the repository context ("Default Playwright OS suffixes (like -linux.png) are obsolete and should be purged").
FINDINGS
This PR aims to stabilize mobile visual snapshots by adopting native Playwright snapshot features (toHaveScreenshot). However, the PR introduces obsolete -linux.png suffixed snapshots which violates the repository's Playwright configuration constraints and fails the CI. Please remove the OS suffixes from the snapshot names to rely on the project-level directory grouping.
FINAL RECOMMENDATION
Not Approved
arii
left a comment
There was a problem hiding this comment.
ANTI-AI-SLOP
The Build & E2E check failed. This PR introduces -linux.png snapshots which directly violates the repository context ("Default Playwright OS suffixes (like -linux.png) are obsolete and should be purged").
FINDINGS
This PR aims to stabilize mobile visual snapshots by adopting native Playwright snapshot features (toHaveScreenshot). However, the PR introduces obsolete -linux.png suffixed snapshots which violates the repository's Playwright configuration constraints and fails the CI. Please remove the OS suffixes from the snapshot names to rely on the project-level directory grouping.
FINAL RECOMMENDATION
Not Approved
arii
left a comment
There was a problem hiding this comment.
ANTI-AI-SLOP
The Build & E2E check failed. This PR introduces -linux.png snapshots which directly violates the repository context ("Default Playwright OS suffixes (like -linux.png) are obsolete and should be purged").
FINDINGS
This PR aims to stabilize mobile visual snapshots by adopting native Playwright snapshot features (toHaveScreenshot). However, the PR introduces obsolete -linux.png suffixed snapshots which violates the repository's Playwright configuration constraints and fails the CI. Please remove the OS suffixes from the snapshot names to rely on the project-level directory grouping.
FINAL RECOMMENDATION
Not Approved
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
This PR stabilizes the Playwright visual regression suite to prevent false-positive failures across different environments (local vs. CI).
Key improvements:
Mobile ChromeandMobile Safariprojects toplaywright.config.ts, ensuring consistent emulation parameters.page.screenshot()to Playwright's nativeexpect(page).toHaveScreenshot(). This allows Playwright to handle environment-specific snapshot naming (e.g.,-linux.png) automatically.disableAnimationsto suppress View Transitions and force immediate scrolling.document.fonts.readycheck to thepagefixture to ensure snapshots are captured only after web fonts have settled.Fixes #2900
PR created automatically by Jules for task 5203909596431698698 started by @arii