Agent visual verification: reporting extension, semantic snapshot, and skill (#114) - #115
Merged
Conversation
…114) Adds an agent-oriented visual verification loop so AI coding agents can verify Vaadin UIs via one cheap batch test run instead of many Playwright MCP round-trips. - AgentReporting: JUnit 5 AfterTestExecutionCallback that, on failure, writes target/agent-report/{failure.txt, failure.png, component-snapshot.txt} before @AfterEach closes the page. - VisualVerificationTest: base class (connects to an already-running app, 1920x1080 viewport, numbered shot() screenshots, configurable base URL) plus AgentReportProvider for @ExtendWith users, with a reflection fallback. - ComponentSnapshot: compact semantic summary of vaadin-* components, orders of magnitude smaller than an accessibility-tree dump. - skills/visual-verification/SKILL.md + marketplace/plugin registration and README docs. - AgentReportingIT covers the snapshot, screenshot numbering, failure bundle, and the extension wiring (fires on failure, silent on success). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n-playwright-test - Drop the Java skeleton example; point to the vaadin-playwright-test skill for how to write the DramaFinder test. - Prerequisite no longer prescribes run commands/profiles — the project knows how to run its app (defer to its README/CLAUDE.md/AGENTS.md). - Replace the seeded/ui-flow verification-preferences machinery with a short 'Reaching the state' note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #114.
Adds an agent-oriented visual verification loop so AI coding agents can verify Vaadin UIs via one cheap DramaFinder batch run instead of dozens of Playwright MCP round-trips and large accessibility-tree dumps.
What's included
1.
AgentReportingJUnit 5 extensionAfterTestExecutionCallbackthat fires before@AfterEachcloses the page, so it captures the real failing state. On failure it writes totarget/agent-report/(Maven) orbuild/agent-report/(Gradle):failure.txt— assertion message + trimmed stack trace (drops JDK/JUnit/Spring frames, capped) + URLfailure.png— full-page screenshot at failure timecomponent-snapshot.txt— semantic snapshot (below)2.
VisualVerificationTestbase classConnects to an already-running app (no Spring boot-up — keeps iterations cheap), 1920×1080 viewport,
shot(name)for deterministic numbered screenshots (01-…png,02-…png), andopen(path)relative to a configurable base URL (dramafinder.agent.baseUrl/DRAMAFINDER_BASE_URL, defaultlocalhost:8080). Carries@ExtendWith(AgentReporting.class).AgentReportProviderlets standalone@ExtendWithusers expose their page; a reflection fallback finds aPage/AgentReportfield otherwise.3.
ComponentSnapshotutilityWalks the light DOM for
vaadin-*elements in onepage.evaluate()and emits terse lines, e.g.vaadin-grid — 100 rows, columns [First Name, Last Name, Email], 1 selected,vaadin-combo-box "Category" = "Travel",notification: "Saved". Skips internal/structural tags and selection-overlay items.4. Bundled skill
skills/visual-verification/SKILL.md(from the ticket's draft, adapted to the real API), plus.claude-plugin/plugin.json, amarketplace.jsonentry, and README docs.Tests
AgentReportingIT(6 tests, all passing): grid snapshot content, exclusion of internal cell elements, numberedshot()output, the failure bundle, and — via a dynamic-proxyExtensionContext— that the extension fires on failure and stays silent on success.🤖 Generated with Claude Code