skills: invoke-not-read guard + split visual-verification into vaadin-playwright-screenshot - #116
Merged
Merged
Conversation
Both skills point agents at bundled reference files — vaadin-playwright-test at api-reference.md / element-mapping.md / TESTING.md, visual-verification at vaadin-playwright-test — but those load only when a skill is *invoked* via the Skill tool, not when its SKILL.md is opened by hand. Instructions that say "use / follow / load the skill" read as "go read the markdown", so an agent can end up with only SKILL.md and none of the references. Observed failure: an agent read vaadin-playwright-test's SKILL.md directly, never got api-reference.md, and unzipped the DramaFinder jar to discover the API — exactly what the skill's own "never unzip the jar" rule forbids. - vaadin-playwright-test: header guard tying the invoke-don't-read rule to why "never unzip the jar" works; description verb "load" -> "invoke (via the Skill tool)". - visual-verification: header guard; its internal pointer to vaadin-playwright-test now says "invoke ... via the Skill tool". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the visual-verification skill with vaadin-playwright-screenshot, focused solely on capturing screenshots via a temporary DramaFinder test (screenshot validation was generic, not DramaFinder-specific, so it's dropped rather than kept as a separate skill). - Take the acceptance criteria to screenshot as the skill argument (inline or a reference to a spec/ticket); one screenshot proves each criterion. - Portability: state the DramaFinder version that ships the agent helpers and fail clearly if absent; let the criteria argument be a reference. - Add a generated agent-api-reference.md (VisualVerificationTest, AgentReporting, AgentReport, AgentReportProvider, ComponentSnapshot) so agents look the API up without unzipping the sources jar, mirroring the element api-reference: generator tool, CI staleness guard, and SKILL/AGENTS pointers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jcgueriaud1
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch carries two related skill changes.
1. Invoke-don't-read guard (commit 2c14322)
Problem
Both skills direct the agent to bundled reference files —
vaadin-playwright-testpoints atapi-reference.md,element-mapping.md, andTESTING.md. Those files only load into context when a skill is invoked via the Skill tool, not when itsSKILL.mdis opened by hand.But the instructions that route agents here (and the skills' own cross-references) say "use / follow / load the skill" — which reads naturally as "go read its markdown." An agent that does that ends up with
SKILL.mdand none of the references.Observed failure: an agent read
vaadin-playwright-test/SKILL.mddirectly, never receivedapi-reference.md, and sounzip+javap'd the DramaFinder jar to discover the API — the exact thing this skill's own "Never download or unzip the DramaFinder jar/sources to discover its API" rule forbids.Change
vaadin-playwright-test/SKILL.md— a header guard tying the invoke-don't-read rule to why "never unzip the jar" holds; description verbload this skill→invoke this skill (via the Skill tool).2. Split visual-verification → vaadin-playwright-screenshot (commit 33b4126)
Replace
visual-verificationwithvaadin-playwright-screenshot, focused solely on capturing screenshots via a temporary DramaFinder test. Screenshot validation was generic (not DramaFinder-specific), so it's dropped rather than kept as a separate skill.agenthelpers and fails clearly if absent; lets the criteria argument be a reference rather than only inline text.agent-api-reference.md— so agents look upVisualVerificationTest,AgentReporting,AgentReport,AgentReportProvider, andComponentSnapshotwithout unzipping the sources jar, mirroring the elementapi-reference.md: a JBang generator (tools/generate-agent-api-reference.java), a CI staleness guard (.github/workflows/agent-api-reference.yml), and SKILL/AGENTS pointers with a raw-GitHub-URL fallback.🤖 Generated with Claude Code