Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/tessl-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ jobs:
- uses: tesslio/setup-tessl@v2
with:
token: ${{ secrets.TESSL_TOKEN }}
- run: tessl tile publish
- run: tessl tile publish ./skills/vaadin-playwright-test
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,6 @@ package.json

.vscode/.copilot-plugin
.vscode/launch.json

dist
.env
126 changes: 91 additions & 35 deletions skills/vaadin-playwright-test/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,100 +7,153 @@ description: Generate Playwright integration tests for Vaadin 25 views using the

## Best practices

Always follow the guidelines in [@TESTING.md](TESTING.md) when generating tests. Key rules:

- **One test, one assert** — each test method covers a single piece of functionality
- **User-facing locators** — prefer label, `aria-label`, `aria-role`, or `data-testid` over CSS classes or generated IDs
- **DramaFinder elements for all interactions** — never interact with raw locators when a wrapper exists
- **No `Thread.sleep()`** — use Playwright auto-waiting or `waitFor` methods instead
- **Assert on user-visible state** — check visibility, text, or enabled/disabled, not internal CSS or component state
Always follow [@TESTING.md](TESTING.md) when generating tests. Key rules:

- **One test, one assert** — each test method covers a single piece of
functionality
- **User-facing locators** — prefer label, `aria-label`, `aria-role`, or
`data-testid` over CSS classes or generated IDs
- **DramaFinder elements for all interactions** — never interact with raw
locators when a wrapper exists
- **No `Thread.sleep()`** — use Playwright auto-waiting or `waitFor` methods
instead
- **Assert on user-visible state** — check visibility, text, or
enabled/disabled, not internal CSS or component state

## Step 1 — Assess project state

Run these checks in parallel before doing anything else:

1. **DramaFinder on classpath?** — grep `pom.xml` for `dramafinder` or `org.vaadin.addons`
2. **Spring Boot app?** — grep `pom.xml` for `spring-boot-starter` or `vaadin-spring-boot-starter`
3. **Existing IT tests?** — look for `*IT.java` files under `src/test/java`
1. **DramaFinder on classpath?** — grep `pom.xml` for
`<artifactId>dramafinder</artifactId>`.
2. **Spring Boot app?** — grep `pom.xml` for `spring-boot-starter`.
3. **Existing IT tests?** — look for `*IT.java` files under `src/test/java`.
4. **`SpringPlaywrightIT` already in project?** —
`find src/test/java -name SpringPlaywrightIT.java`.

### DramaFinder not found — propose setup, then run it

### DramaFinder not found
Resolve the latest version (Step 1 of [setup.md](setup.md)) and propose the
following in a single confirmation:

Show the user this message and stop:
- Add `org.vaadin.addons:dramafinder:<VERSION>` and
`com.microsoft.playwright:playwright` (test scope) to `pom.xml` with
`<dramafinder.version>` in `<properties>`.
- **Spring Boot only:** also create
`src/test/java/<basePackage>/it/support/SpringPlaywrightIT.java`.

> DramaFinder is not on the classpath. Follow the [setup guide](setup.md) to add the required dependencies, then come back to generate tests.
On confirmation, execute [setup.md](setup.md) end-to-end, then continue with
Step 2.

### DramaFinder found — follow existing patterns

If existing `*IT.java` files are found, read one or two of them to understand the project's conventions (base class, package structure, assertion style, helper methods). Use those as the template for generated tests.
If existing `*IT.java` files are found, read one or two to understand the
project's conventions (base class, package structure, assertion style, helper
methods) and use them as the template.

If no existing IT tests exist, use the default structure in Step 3.

### `SpringPlaywrightIT` location

- 1 hit → use that fully-qualified class name.
- 0 hits + Spring Boot detected → run setup (it will create the file).
- More than 1 hit → ask the user which one to use.

## Step 2 — Map view components to DramaFinder elements

Read the target view source provided by the user. Extract:

- `@Route("value")` → URL path (default: class name lowercased, stripped of "View" suffix)
- `@Route("value")` → URL path (default: class name lowercased, stripped of
`View` suffix, e.g. `PersonView` → `/person`).
- `@PageTitle("...")` → expected page title
- All Vaadin component field declarations and `add(...)` calls → map to DramaFinder elements
- Every interactive component → its DramaFinder wrapper (see table below).
- Form fields → label text used as locator.
- Grids → column headers and row content to assert against.
- Navigation triggers → button labels or menu items that cause route changes.

See [element-mapping.md](element-mapping.md) for the full component → element class table. Each element also has detailed documentation with examples in the [specifications folder](https://github.com/parttio/dramafinder/tree/master/docs/specifications).
See [element-mapping.md](element-mapping.md) for the full component → element
class table. Each element also has detailed documentation with examples in
the [specifications folder](https://github.com/parttio/dramafinder/tree/master/docs/specifications).

For components with **no DramaFinder wrapper**, use a plain Playwright locator. For more complex needs, you can create your own element class extending `VaadinElement`, or [open an issue](https://github.com/vaadin/dramafinder/issues) in the DramaFinder repository to request one.
For components with **no DramaFinder wrapper**, use a plain Playwright locator.
For more complex needs, you can create your own element class extending
`VaadinElement`,
or [open an issue](https://github.com/vaadin/dramafinder/issues) in the
DramaFinder repository to request one.

## Step 3 — Generate the test class

### Default structure (no existing tests to mirror)

```java
package <same.package.as.view>; // mirror src/test/java structure
package

<same.package.as.view>; // mirror src/test/java structure

import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
import org.vaadin.addons.dramafinder.element.TextFieldElement; // import only used elements
import org.vaadin.addons.dramafinder.tests.it.SpringPlaywrightIT; // or AbstractBasePlaywrightIT

import <basePackage>.it.support.SpringPlaywrightIT; // Spring projects: actual location from Step 1
// import org.vaadin.addons.dramafinder.AbstractBasePlaywrightIT; // non-Spring projects

import static com.microsoft.playwright.assertions.PlaywrightAssertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertEquals;

@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT) // omit if not Spring Boot
public class <ViewName>IT extends SpringPlaywrightIT { // or AbstractBasePlaywrightIT
@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
// omit if not Spring Boot
public class <ViewName>IT extends

SpringPlaywrightIT { // or AbstractBasePlaywrightIT

@Override
public String getView() {
public String getView () {
return "/<route-path>";
}

@Test
public void testTitle() {
public void testTitle () {
assertThat(page).hasTitle("<PageTitle value>");
}

// ... component tests below
}
```

Use `SpringPlaywrightIT` if Spring Boot is detected, `AbstractBasePlaywrightIT` otherwise.
Use `SpringPlaywrightIT` if Spring Boot is detected, `AbstractBasePlaywrightIT`
otherwise.

### Component test patterns

**Smoke test (one per component):**

```java
@Test
public void test<ComponentLabel>() {
TextFieldElement field = TextFieldElement.getByLabel(page, "My Label");
field.assertVisible();
field.assertLabel("My Label");
field.assertValue("");
field.setValue("test value");
field.assertValue("test value");
@Test
public void test<ComponentLabel>(){
TextFieldElement field = TextFieldElement.getByLabel(page, "My Label");
field.

assertVisible();
field.

assertLabel("My Label");
field.

assertValue("");
field.

setValue("test value");
field.

assertValue("test value");
}
```

**Form with validation:**

```java

@Test
public void testFormSubmitWithInvalidInput() {
TextFieldElement nameField = TextFieldElement.getByLabel(page, "Name");
Expand All @@ -124,6 +177,7 @@ public void testFormSubmitWithValidInput() {
**Grid data loading:**

```java

@Test
public void testGridLoadsData() {
GridElement grid = GridElement.get(page);
Expand All @@ -138,12 +192,14 @@ Display the full generated test class in a code block. Then ask:

> Shall I write this to `src/test/java/<package>/<ViewName>IT.java`?

Only write the file after explicit confirmation. Place it in `src/test/java` mirroring the view's package under `src/main/java`.
Only write the file after explicit confirmation. Place it in `src/test/java`
mirroring the view's package under `src/main/java`.

## Step 5 — Offer to run the test

After writing, ask:

> Do you want me to run this test now with `mvn verify -Dit.test=<ViewName>IT`?

**Warn the user**: the first Vaadin frontend build takes 3–5 minutes. Subsequent runs are ~25 seconds.
**Warn the user**: the first Vaadin frontend build takes 3–5 minutes. Subsequent
runs are ~25 seconds.
16 changes: 16 additions & 0 deletions skills/vaadin-playwright-test/evals/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Provider selection: "local" (LM Studio) or "gemini"
EVAL_PROVIDER=local

# Local model (LM Studio, OpenAI-compatible)
LOCAL_BASE_URL=http://127.0.0.1:1234/v1
LOCAL_MODEL=openai/gpt-oss-20b

# Gemini (uses Google's OpenAI-compatible endpoint)
GEMINI_API_KEY=
GEMINI_MODEL=gemini-2.0-flash
GEMINI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/

# Langfuse (cloud or self-hosted)
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=https://cloud.langfuse.com
45 changes: 45 additions & 0 deletions skills/vaadin-playwright-test/evals/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# vaadin-playwright-test — eval harness

Phase 0 bootstrap. See `../../../PLAN.md` for the full plan.

The harness uses the OpenAI SDK against any OpenAI-compatible endpoint.
Two providers are supported out of the box:

- `local` — LM Studio (default), e.g. `openai/gpt-oss-20b` at `http://127.0.0.1:1234/v1`
- `gemini` — Google's OpenAI-compatible endpoint

## Setup

```bash
cd skills/vaadin-playwright-test/evals
npm install
cp .env.example .env
# fill in keys / pick a provider
```

For self-hosted Langfuse:

```bash
docker compose up -d
# then set LANGFUSE_HOST=http://localhost:3000 in .env
```

## Run

```bash
# default: local LM Studio
npm run smoke

# Gemini
EVAL_PROVIDER=gemini npm run smoke
```

Runs one hardcoded prompt twice (with/without the skill in the system prompt)
and logs both as Langfuse traces tagged `phase=bootstrap`.

## Note on prompt caching

The PLAN.md calls for prompt caching on the system prompt. Neither LM Studio
nor Gemini's OpenAI-compatible endpoint supports the Anthropic-style
`cache_control` parameter, so caching is a no-op here. Re-introduce it when
the harness is pointed at a provider that supports it.
29 changes: 29 additions & 0 deletions skills/vaadin-playwright-test/evals/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
services:
langfuse-db:
image: postgres:16
restart: always
environment:
POSTGRES_USER: langfuse
POSTGRES_PASSWORD: langfuse
POSTGRES_DB: langfuse
volumes:
- langfuse_db_data:/var/lib/postgresql/data
ports:
- "5432:5432"

langfuse:
image: langfuse/langfuse:2
restart: always
depends_on:
- langfuse-db
ports:
- "3000:3000"
environment:
DATABASE_URL: postgresql://langfuse:langfuse@langfuse-db:5432/langfuse
NEXTAUTH_SECRET: change-me
SALT: change-me
NEXTAUTH_URL: http://localhost:3000
TELEMETRY_ENABLED: "false"

volumes:
langfuse_db_data:
Loading
Loading