Skip to content

[AGENT] Make visual evidence opt-in while retaining browser assertions - #369

Merged
BASIC-BIT merged 4 commits into
mainfrom
codex/optional-visual-tests
Oct 15, 2026
Merged

BASIC-BIT merged 4 commits into
mainfrom
codex/optional-visual-tests

Conversation

@BASIC-BIT

@BASIC-BIT BASIC-BIT commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

[AGENT]

Keep browser assertions automatic while making review screenshots and both Linux image comparisons opt-in with ci:visuals. Selection applies on normal PR events, regardless of changed paths. Label edits do not launch the app pipeline; local capture behavior stays unchanged unless explicitly disabled.

The consolidated report fails on missing required evidence, invalid selection, or failed/cancelled/unexpectedly skipped comparisons. It rejects stale heads and invalidates prior visual claims when the label is removed. Baselines are never updated automatically.

The newly automatic public smoke suite exposed two stale expectations. Tests now verify the existing community-scoped event image route and accessible keyboard selection, preserving image-content, focus and Enter-navigation assertions.

The branch includes main 104dad4e from merged PR #368, preserving its HMR correction and the optional capture guard. The event-editor snapshot spec and baselines match main.

Validation on d65048d2:

  • Default Linux run and selected Linux run both passed. Each ran 112 public smoke cases, 72 public preview assertions, 344 Storybook assertions and 80 mutation-flow cases including HMR isolation, plus hosted flows.
  • Both previews produced zero review PNGs by default. Selected mode produced 86 public and 74 Storybook PNGs; both comparison suites passed (76 public cases and 36 Storybook cases) without baseline changes.
  • Adding ci:visuals during the default run launched no app pipeline. Its report preserved the original selection and noted the later label change. The ready-for-review event selected both comparisons on the same commit.
  • Full local scripts: 259 passed, one skipped, zero failures. Combined HMR-isolation and capture-mode browser checks passed all eight desktop/mobile cases; all 17 helper/report/removal tests passed.
  • Independent merge-result review found no actionable findings and passed all 28 focused workflow, capture-boundary and reporting tests. Fresh Claude and Greptile reviews found no actionable issues. Codex and Copilot were quota-unavailable.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

PR verification report

All configured preview and verification checks passed.

Visuals selected for this run: yes.

Check Result Evidence
Public route smoke PASSED Open artifact
Mutation data flow PASSED Open artifact
Hosted data flow PASSED Open artifact
Public route screenshots PASSED Open artifact
Public route image diff PASSED Open artifact
Storybook screenshots PASSED Open artifact
Storybook image diff PASSED Open artifact

Changed visual baselines: none.

Updated from Baseline Checks run 38680043293, attempt 1 for d65048d. This comment is updated in place.

@BASIC-BIT BASIC-BIT changed the title Make visual evidence opt-in while retaining browser assertions [AGENT] Make visual evidence opt-in while retaining browser assertions Oct 15, 2026
@BASIC-BIT
BASIC-BIT marked this pull request as ready for review October 15, 2026 03:21
Copilot AI balanced review requested due to automatic review settings October 15, 2026 03:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-15T04:10:11.840995Z d65048d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@greptile-apps

greptile-apps Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; no actionable defects were found.

Summary

Makes review screenshots and both image-comparison suites opt-in through ci:visuals, while keeping browser assertions automatic.

  • The ci:visuals label selects review screenshots and image comparisons.
  • Playwright review screenshots follow the selected capture mode.
  • Public-route smoke checks run automatically on desktop and mobile.

The previous and current heads have different merge bases with the target commit, so this review covered the full PR diff. No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  PR[Normal PR event] --> Selection{ci:visuals present?}
  PR --> Smoke[Public smoke and data-flow assertions]
  Selection -->|No| Assertions[Preview assertions without review screenshots]
  Selection -->|Yes| Captures[Preview assertions with review screenshots]
  Selection -->|Yes| Comparisons[Both image comparisons]
  Smoke --> Report[Check results and required artifacts]
  Assertions --> Report
  Captures --> Report
  Comparisons --> Report
  Report --> Head{Current PR head?}
  Head -->|Yes| Publish[Update report with run selection]
  Head -->|No| Ignore[Do not publish]
  Removal[Label removed] --> Historical[Preserve results and mark visuals historical]
Loading

Reviews (2) · Last reviewed commit: "Merge main HMR correction into optional ..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

[AGENT]

Claude Review

Claude review for commit d65048d2cbfc829106be303b8f4f38dfdbf9182b is unavailable because the pull request was merged.

This is not a current review or approval.

Copy link
Copy Markdown
Owner Author

[AGENT] Re the non-blocking coverage observation in the Claude review: the label-change publication path was exercised in the real default run. I added ci:visuals while that assertions-only run was pending. No new app pipeline launched, and the published report retained Visuals selected for this run: no (assertions only), both comparisons NOT SELECTED, and the changed-selection note. That acceptance evidence covers this PR; the additional mocked publisher test remains a non-blocking coverage suggestion.

@BASIC-BIT
BASIC-BIT marked this pull request as draft October 15, 2026 03:53
@BASIC-BIT
BASIC-BIT marked this pull request as ready for review October 15, 2026 04:10
Copilot AI balanced review requested due to automatic review settings October 15, 2026 04:10
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@BASIC-BIT
BASIC-BIT merged commit 7f06484 into main Oct 15, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants