qa-pipeline

Compile Gherkin .feature files to runnable Playwright (web) or Maestro (mobile) scripts, execute them locally, and let Claude debug the failures. No SaaS test platforms, no test-management vendors — features live on disk as plain .feature files, run history lives in a local SQLite db.

Status: v0.1 alpha — not yet ready for public use.

Why

You already write Gherkin to describe what a test should do. Turning that into Playwright TS or Maestro YAML is repetitive — and when a test fails, triaging it across stdout, traces, and screenshots takes more time than fixing it. qa-pipeline does both jobs: it compiles each Gherkin step to a deterministic fragment, caches it by content hash, and only invokes the LLM when a fragment is missing or has gone stale. The first run is mostly AI. The third run is a deterministic regression test.

Install + quickstart

pnpm install
cp .env.example .env       # set ANTHROPIC_API_KEY
pnpm build:web             # one-time: build the React UI
pnpm seed:demo             # optional: pre-cache the bundled demos (no API key needed)
pnpm dashboard             # serves UI + API on http://127.0.0.1:3000

Two demo features ship with the project:

• features/demo-shop-pass.feature — opens the bundled demo shop, asserts the catalog renders, adds a product, checks the cart count.
• features/demo-shop-fail.feature — same opening steps, then asserts a heading that doesn't exist so Playwright captures a screenshot + trace + video. Use it to verify the dashboard's failure UI.

After pnpm seed:demo both demos run with zero LLM cost — every step has a cached compiled fragment, so the orchestrator goes straight to Replay. Click ⚡ Replay (or ▶ Run) on either test in the dashboard.

You can also drive the orchestrator from a script (no dashboard):

pnpm tsx -e "import { runHybrid } from './src/index.js'; \
  console.log(await runHybrid({ featurePath: 'demo-shop-pass.feature', target: 'playwright' }));"

Or via the open-ended agent for chat/debug flows:

pnpm cli "compile and run features/demo-shop-pass.feature against https://staging.example.com"

External tools

• Playwright — not a dep of this package. Install it in the project that uses qa-pipeline (or globally): pnpm add -D @playwright/test && npx playwright install.
• Maestro CLI — install from maestro.mobile.dev. Optional, only needed for mobile targets.

Architecture

Replay-first, AI-second. The orchestrator is a deterministic phase sequencer; the LLM is invoked only as the per-step compile unit-of-work.

runHybrid({ featurePath, target })
        │
        ▼
┌─ Phase 0: init ─────────────────────────────────────────────────┐
│  parse .feature → enumerate step hashes → look up cache         │
│  classify feature: 'replay' (all cached) | 'ai' (any miss)      │
└─────────────────────────────────────────────────────────────────┘
        │
        ▼
┌─ Phase 1: replay (only when classification = 'replay') ─────────┐
│  assemble script from cached fragments → execute → record run   │
│  on success: complete. on failure: forward error to ai phase.   │
└─────────────────────────────────────────────────────────────────┘
        │
        ▼
┌─ Phase 2: ai (when missing OR replay failed) ───────────────────┐
│  one structured LLM call → fragments for each missing step      │
│  persist each fragment → assemble full script → execute         │
└─────────────────────────────────────────────────────────────────┘
        │
        ▼
┌─ Phase 3: complete ─────────────────────────────────────────────┐
│  emit { type: 'completed', result } via the event stream        │
└─────────────────────────────────────────────────────────────────┘

Edit a Gherkin step's text → its hash changes → no cache hit → only that step recompiles. Replay fails (UI drift, etc.) → ai phase re-runs with the replay error in the prompt so the model fixes the actual broken step.

Layer responsibilities

Layer	Responsibility	Anti-pattern
lib/	Pure helpers (gherkin parser, step hash). No I/O.	DB / network / filesystem access.
helpers/	Reusable Playwright/Maestro snippets exposed to the compiler.	Per-feature one-offs.
orchestrator/	Phase sequencer + injected deps. Transport-agnostic.	HTTP types or Express handlers.
orchestrator/phases/	One file per phase. Each is a small, testable function.	Phases knowing about each other's internals.
dashboard/	Node http SSE adapter + static UI. Thin.	Business logic that doesn't depend on a request.
tools/	Anthropic Tool defs for the open-ended agent (debug/chat).	Direct calls from the orchestrator (use deps instead).

Rule: code calls down the stack only. orchestrator/ does not import from dashboard/. lib/ does not import from anywhere else inside the project.

Dashboard

Single-page React UI on 127.0.0.1:3000:

• Top: five stat cards — tests, plans, total runs, passed, failed.
• Three columns: Plans (folders of tests), Tests (.feature files), Recent runs. Click any item to expand it inline — Gherkin source paired with the compiled script for tests, the YAML for plans, per-step results + screenshots for runs.
• Run buttons open a side drawer that streams orchestrator events live: cache lookup, per-step compile, replay/AI execution, screenshot strip when something fails, cost chip.
• + New test / + New plan open compact modals.

A focused ~25 second slice on the agent compiling and running a passing test:

Tests view, expanded — every Gherkin step paired with its compiled Playwright fragment inline. The orange helper · openPage chip appears when a fragment uses a stdlib helper; uncached steps show not compiled.	Run drawer, replay-only — phase strip walks init → plan → replay → ai → done. Each step has a ready (cached) or needs compiling badge so you know up-front whether the LLM will be invoked.
Plan run — the orchestrator sequences each test in the plan, tagging the source as VIA AI or via cache per test. Active phase has a spinner; completed phases turn solid orange. The red Stop kills the running execa subprocess.	Failure detail — failed scenarios are bordered red with the assertion message inline; Playwright's screenshot, trace, and video are all surfaced underneath, click-to-lightbox.

Endpoints

Used by the UI; also handy for scripting.

Method	Path	Purpose
POST	/api/runs/hybrid	Run a feature through the orchestrator (SSE)
POST	/api/plans/run	Run a plan (SSE)
POST	/api/plans/create	Create a <name>.plan.yaml
POST	/api/tests/create	Generate a .feature from NL / Gherkin / hybrid
GET	/api/runs/:id/playwright-result	Per-test.step pass/fail breakdown
GET	/api/runs/:id/screenshots	All screenshots / videos for a run, recursive walk

Override host/port via env vars: HOST=0.0.0.0 PORT=8080 pnpm dashboard.

Programmatic use

import { runHybrid, runHybridGenerator } from 'qa-pipeline';

// Drain to a final result
const result = await runHybrid({
  featurePath: 'checkout.feature',
  target: 'playwright',
  baseUrl: 'https://staging.example.com',
});
console.log(result.status, result.cacheStats);

// Or stream phase events (e.g. for a live dashboard)
for await (const event of runHybridGenerator(input, defaultHybridDeps())) {
  switch (event.type) {
    case 'phase':            /* ... */ break;
    case 'replay_passed':    /* ... */ break;
    case 'ai_step_compiled': /* ... */ break;
  }
}

Substitute deps for tests:

import { runHybridGenerator } from 'qa-pipeline';
import { mockCompileMissing } from 'qa-pipeline/orchestrator/phases/ai-mock.js';

for await (const event of runHybridGenerator(input, {
  lookupCache:     async () => new Map(),
  persistFragment: async () => {},
  compileMissing:  mockCompileMissing,
  executeScript:   async () => ({ runId: 0, exitCode: 0, stdout: '', stderr: '', artifactsDir: '' }),
})) { /* ... */ }

Configuration

All runtime config is via env vars — see .env.example. This package does not load .env automatically; the caller should (e.g. dotenv/config in the entrypoint, or CI env injection).

Var	Required	Default	Purpose
ANTHROPIC_API_KEY	yes	—	Claude API access
QA_PIPELINE_MODEL_DEFAULT	no	claude-opus-4-7	Model for compile / debug
QA_PIPELINE_MODEL_SIMPLE	no	claude-sonnet-4-6	Model for simple CRUD
QA_PIPELINE_DB_PATH	no	./qa.db	SQLite run history
QA_PIPELINE_FEATURES_DIR	no	./features	Where .feature files live
QA_PIPELINE_COMPILED_DIR	no	./compiled	Where compiled specs are written
QA_PIPELINE_ARTIFACTS_DIR	no	./artifacts	Where run logs / screenshots go
QA_PIPELINE_LOG_LEVEL	no	info	pino log level

Tool catalog (open-ended agent)

The open-ended runQaAgent accepts these Anthropic tools. The orchestrator does not use them (it calls deps directly), but they remain useful for chat/debug flows where the model drives.

Tool	Purpose
list_features	List .feature files under the features dir
read_feature	Read a .feature file
write_playwright_script	Write a compiled .spec.ts to the compiled dir
write_maestro_flow	Write a compiled .yaml to the compiled dir
run_playwright	Execute a compiled spec via npx playwright test
run_maestro	Execute a compiled flow via maestro test
read_artifact	Read stdout / stderr / trace files from a run
list_runs	Query recent runs from SQLite
get_run	Get full details of one run by id
parse_feature_steps	Parse a .feature file into structured scenarios with step hashes
lookup_compiled_step	Check the cache for one or more step hashes
persist_compiled_step	Save a compiled fragment for a Gherkin step

License

Apache-2.0. See LICENSE.