diff --git a/docs/_diagrams/eval-test-types.excalidraw b/docs/_diagrams/eval-test-types.excalidraw index cd8b687..5d8ce11 100644 --- a/docs/_diagrams/eval-test-types.excalidraw +++ b/docs/_diagrams/eval-test-types.excalidraw @@ -94,7 +94,7 @@ "frameId": null, "roundness": null, "seed": 7942757, - "version": 1, + "version": 2, "versionNonce": 6169453, "isDeleted": false, "boundElements": null, @@ -102,13 +102,13 @@ "link": null, "locked": false, "index": "a2", - "text": "Isolated Workspace (Docker container or HOME-overridden temp dir)\nProject rsync + Setup overlays (base + per-eval) + Skill under test + Fixtures + Credentials", + "text": "Clean Working Directory (fresh empty HOME, env built from scratch)\nSkill under test + Case fixtures + state_prefix applied to the input", "fontSize": 16, "fontFamily": 1, "textAlign": "center", "verticalAlign": "top", "containerId": "el_1002", - "originalText": "Isolated Workspace (Docker container or HOME-overridden temp dir)\nProject rsync + Setup overlays (base + per-eval) + Skill under test + Fixtures + Credentials", + "originalText": "Clean Working Directory (fresh empty HOME, env built from scratch)\nSkill under test + Case fixtures + state_prefix applied to the input", "lineHeight": 1.25, "baseline": 13 }, diff --git a/docs/_diagrams/render.mjs b/docs/_diagrams/render.mjs index 92585c9..505ae3c 100644 --- a/docs/_diagrams/render.mjs +++ b/docs/_diagrams/render.mjs @@ -19,7 +19,7 @@ statSync(inPath); const sceneJson = JSON.parse(readFileSync(inPath, 'utf-8')); -const htmlPath = resolve(fileURLToPath(import.meta.url), '..', 'excalidraw_render.html'); +const htmlPath = resolve(fileURLToPath(import.meta.url), '..', 'render.html'); const htmlUrl = pathToFileURL(htmlPath).href; const browser = await chromium.launch({ headless: true }); diff --git a/docs/explanation/why-bmad-eval-runner.md b/docs/explanation/why-bmad-eval-runner.md index 544d9f0..d5112d3 100644 --- a/docs/explanation/why-bmad-eval-runner.md +++ b/docs/explanation/why-bmad-eval-runner.md @@ -7,7 +7,7 @@ The eval runner is built around a simple goal: produce results that reflect the ## Isolation -Every eval starts in a clean room. With Docker, the run executes inside a fresh container off `bmad-eval-runner:latest`. Without Docker, the runner falls back to a per-eval temp directory with `HOME` overridden so global memory and global `CLAUDE.md` cannot influence the result. Either way, two developers running the same eval get the same workspace state. +Every eval starts in a clean room. Each case runs from its own working directory with the skill under test staged into it, and the subprocess environment is built from scratch rather than inherited: `PATH`, a fresh empty `HOME` inside the case folder, `CLAUDE_CONFIG_DIR` pointing inside that `HOME`, the adapter's auth variable, and nothing else. No container, no terminal emulation, no credential file staging. Two developers running the same eval get the same workspace state. Why this matters: skills are sensitive to context. Your global `~/.claude/CLAUDE.md`, your auto-memory, an ancestor `CLAUDE.md` in the project tree, cached MCP settings. All of these reach a default `claude -p` invocation. The eval should measure the skill, not the bench it was tested on. @@ -35,4 +35,4 @@ After artifact runs complete, the runner spawns a grader subagent per eval (in p ## Next Steps -For a step-by-step run, see [Run Evals Against a Skill](/how-to/run-evals-against-a-skill.md). For the complete eval file schema, see [Eval Format](/reference/eval-format.md). For why Docker matters, see [Install Docker for Evals](/how-to/install-docker-for-evals.md). +For a step-by-step run, see [Run Evals Against a Skill](/how-to/run-evals-against-a-skill.md). For the complete eval file schema, see [Eval Format](/reference/eval-format.md). diff --git a/docs/how-to/install-docker-for-evals.md b/docs/how-to/install-docker-for-evals.md deleted file mode 100644 index 1ebf670..0000000 --- a/docs/how-to/install-docker-for-evals.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -title: 'Install Docker for Evals' -description: Install Docker Desktop so the eval runner can give you reproducible, hermetic test runs ---- - -Use Docker Desktop to give the eval runner a real isolation boundary. Without Docker, the runner falls back to local mode, which is best-effort and has known leak paths. - -## When to Use This - -- You plan to run trigger evals (local mode can leak host skills into the workspace) -- You want runs to be reproducible across machines -- You publish a module and want the same eval verdicts other developers see -- You want a guaranteed-empty `HOME` so global memory cannot influence results - -## When to Skip This - -- One-off iteration on artifact evals where local fallback is good enough for now -- A constrained environment where installing Docker is not feasible. The runner falls back to local mode and tells you it is doing so. - -:::note[Prerequisites] - -- Administrator access on your machine to install Docker Desktop -- A few GB of disk space for the Docker Desktop application and the eval-runner image -::: - -## Why Docker - -The eval runner needs to start each run from a clean slate. It is trying to measure the skill, not the host's accumulated state. Without isolation, three things contaminate the result. - -1. **Global memory and CLAUDE.md.** Your `~/.claude/CLAUDE.md` and auto-memory load on every Claude Code invocation. They influence outputs in ways the skill author cannot control. -2. **Ancestor configuration.** A `CLAUDE.md` anywhere above the skill in the directory tree gets discovered and loaded. -3. **Host-installed skills.** When `claude -p` runs in a directory with `.claude/skills/` somewhere up the tree, those skills are discoverable and can fire instead of (or alongside) the skill under test. This is especially harmful for trigger evals. - -Docker solves all three. The container has its own filesystem, its own `HOME`, and its own `.claude/`. Local mode patches `HOME` and creates a temp directory but cannot prevent ancestor discovery. - -## Step 1: Install Docker Desktop - -Download Docker Desktop for your platform: - -| Platform | Where to Get It | -| -------- | ----------------------------------------------------------------------------------------------------- | -| macOS | [docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop) | -| Windows | [docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop) | -| Linux | Docker Engine via your distribution's package manager, or Docker Desktop for Linux | - -Follow the installer's prompts. On macOS, drag the Docker app to Applications and launch it. On Windows, the installer enables WSL 2 if needed. - -## Step 2: Start Docker Desktop - -Launch Docker Desktop. Wait for the whale icon to indicate Docker is running. The eval runner shells out to the `docker` CLI; if Docker is not running, the runner falls back to local mode and tells you why. - -## Step 3: Verify Installation - -Confirm Docker is reachable from your terminal: - -```bash -docker info -``` - -A successful response means the eval runner can use Docker. An error means Docker is not running, or the CLI cannot reach the daemon. - -## Step 4: Let the Runner Build the Image - -The first time you invoke the eval runner with `--isolation docker` (or `auto` when Docker is available), the runner builds `bmad-eval-runner:latest` from a Dockerfile shipped with the skill. This takes a few minutes once. Subsequent runs reuse the cached image. - -The image is a minimal Node 20 base with Claude Code, Python 3, and standard tools. Nothing skill-specific or user-specific lives in the image. Your credentials are mounted in at run time, not baked in. - -:::tip[Credential Safety] -The Dockerfile contains no tokens, API keys, or credentials. Your authentication (macOS Keychain credential or `ANTHROPIC_API_KEY`) is staged into a per-run temp directory and mounted into the container as a read-only volume that disappears when the container exits. -::: - -## What You Get - -- Reproducible runs: the same eval produces the same workspace state on any machine with the image -- Real `HOME` isolation: the container's `/home/evaluator` is empty, not just overridden -- Trigger evals you can trust: only the synthetic skill staged for the test is discoverable, not your host's installed skills -- Network can be locked down per run if your evals do not need internet access - -## Tips - -- Rebuild the image with `uv run scripts/docker_setup.py --rebuild` if you ever need to reset it -- Per-eval container resource use is small (a few hundred MB). Parallel workers each spin up their own container. -- If `docker info` works in one terminal but not in your editor's integrated terminal, your shell PATH probably differs. Open a fresh terminal session. - -## Next Steps - -Run the eval runner against a skill: see [Run Evals Against a Skill](/how-to/run-evals-against-a-skill.md). For isolation internals, see the eval-runner skill's `references/isolation.md`. diff --git a/docs/how-to/run-evals-against-a-skill.md b/docs/how-to/run-evals-against-a-skill.md index 1d4371e..010d1c1 100644 --- a/docs/how-to/run-evals-against-a-skill.md +++ b/docs/how-to/run-evals-against-a-skill.md @@ -20,12 +20,11 @@ Use the `bmad-eval-runner` skill to run a skill's evals in a clean workspace and :::note[Prerequisites] - The skill you want to evaluate, with `evals.json` and/or `triggers.json` defined -- Either Docker Desktop installed (preferred) or willingness to run in best-effort local isolation. See [Install Docker for Evals](/how-to/install-docker-for-evals.md). - An Anthropic account authenticated through Claude Code (the runner reuses your existing credential) ::: :::tip[Quick Path] -Invoke the eval runner with the path to your skill: `bmad-eval-runner ./skills/my-skill`. The runner discovers your evals, picks isolation, runs everything in parallel, and tells you where the report lives. +Invoke the eval runner with the path to your skill: `bmad-eval-runner ./skills/my-skill`. The runner discovers your evals, runs each case in its own clean working directory, and tells you where the run folder lives. ::: ## Step 1: Confirm Eval Discovery @@ -40,17 +39,9 @@ The runner looks for evals in this order, taking the first match: If discovery fails, the runner halts. It does not invent evals. -## Step 2: Choose Isolation +## Step 2: Know What Isolation You Get -Pass `--isolation docker|local|auto`. Default is `auto`, which picks Docker when available and local when not. - -| Mode | When to Use | -| ------ | ------------------------------------------------------------------------ | -| docker | Trigger evals (host skills can leak in local mode); reproducible runs | -| local | Quick iteration when you have not installed Docker | -| auto | Default; lets the runner pick the best available option | - -The first time Docker is selected, the runner builds the `bmad-eval-runner:latest` image. This takes a few minutes once. Subsequent runs reuse the cached image. +There is nothing to choose here. Every case runs in its own clean working directory with the skill under test staged into it, and the subprocess environment is built from scratch instead of inherited: `PATH`, a fresh empty `HOME` inside the case folder, `CLAUDE_CONFIG_DIR` inside that `HOME`, and the adapter's auth variable. Your global `CLAUDE.md`, your auto-memory, and your host-installed skills are all out of reach, including for trigger evals. ## Step 3: Pick Mode @@ -67,7 +58,7 @@ Pass `--mode artifact|trigger|both`. Default is `both` if both eval files are fo Invoke the eval runner from your project. A typical invocation: ```bash -bmad-eval-runner ./src/skills/my-skill --isolation docker --workers 8 +bmad-eval-runner ./src/skills/my-skill --workers 8 ``` The runner stages each eval's workspace, executes `claude -p` against the prompt, captures the stream-JSON transcript, and rsyncs any files the skill wrote. After all evals complete, it spawns a grader subagent per eval (in parallel) and aggregates the verdicts. @@ -105,14 +96,13 @@ Run folders are never deleted automatically. Disk management is your call. - Pass `--eval-ids A1,B3` to run only specific evals while iterating - Pass `--workers 8` to parallelize aggressively (default is 4) - A specific eval can override the default timeout by setting `"timeout": 900` in its `evals.json` entry -- For trigger evals, prefer Docker. Local mode can let host-installed skills bleed in via cwd-based discovery and bias the fire rate. ## A Worked Example The `bmad-product-brief` skill in the BMad Method repository (`bmad-code-org/BMAD-METHOD`) ships a complete eval suite at `evals/bmm-skills/bmad-product-brief/`. To run it end-to-end: ```bash -bmad-eval-runner ./src/bmm-skills/1-analysis/bmad-product-brief --isolation docker --workers 8 +bmad-eval-runner ./src/bmm-skills/1-analysis/bmad-product-brief --workers 8 ``` The run produces 17 graded artifact evals (A1-A8 output grading, B1-B8 transcript grading, C1 configuration compliance), 15 trigger eval verdicts, and an aggregated HTML report. Use it as the model when writing evals for your own skills. diff --git a/docs/reference/eval-format.md b/docs/reference/eval-format.md index 6b4e5b2..2585fec 100644 --- a/docs/reference/eval-format.md +++ b/docs/reference/eval-format.md @@ -114,8 +114,8 @@ A query passes when: - `should_trigger=true` and `trigger_rate >= --trigger-threshold` (default 0.5) - `should_trigger=false` and `trigger_rate < --trigger-threshold` -:::caution[Trigger Evals Need Docker] -Local-mode trigger evals can be biased by host-installed skills that are discoverable via cwd-based skill discovery. The detector may see a real skill fire instead of the synthetic. Use Docker isolation for trigger evals whenever it is available. +:::note[Only the Synthetic Skill Is Discoverable] +Each query runs in a clean working directory under a fresh empty `HOME`, so host-installed skills cannot be discovered and fire in place of the synthetic one. The detector also counts only `tool_use` events naming the synthetic skill; whole-transcript substring matching is rejected, because the runtime's init event lists every discovered skill by name and would report a 100% trigger rate regardless of the description. ::: ## Setup Overlays diff --git a/website/public/img/eval-test-types.png b/website/public/img/eval-test-types.png index aaa4064..a1e0515 100644 Binary files a/website/public/img/eval-test-types.png and b/website/public/img/eval-test-types.png differ