feat(skill-optimizer): v1.4 chain implementation (draft, per RFC #52)

.claude-plugin/marketplace.json

@@ -14,7 +14,16 @@
         "name": "Fast"
       },
       "skills": [
-        "./skills/skill-optimizer"
+        "./skills/investigate-functionality",
+        "./skills/investigate-submissions",
+        "./skills/design-tests",
+        "./skills/write-tests",
+        "./skills/validate-tests",
+        "./skills/run-bench",
+        "./skills/analyze",
+        "./skills/improve",
+        "./skills/validate",
+        "./skills/autopilot"
       ]
     }
   ]

.codex/INSTALL.md

@@ -22,10 +22,20 @@ codex plugin marketplace add fastxyz/skill-optimizer --ref main
 
 ## Skill-Only Install
 
-Install the canonical skill with the open skills CLI:
+Install the 9 chain skills with the open skills CLI:
 
 ```bash
-npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a codex -y
+npx skills add fastxyz/skill-optimizer \
+  --skill investigate-functionality \
+  --skill investigate-submissions \
+  --skill design-tests \
+  --skill write-tests \
+  --skill validate-tests \
+  --skill run-bench \
+  --skill analyze \
+  --skill improve \
+  --skill validate \
+  -a codex -y
 ```
 
-Restart Codex if the skill does not appear immediately.
+Restart Codex if the skills do not appear immediately.

.cursor/INSTALL.md

@@ -2,14 +2,24 @@
 
 ## Skill install
 
-Install the skill into Cursor's project or global skill directory through the open skills CLI:
+Install the 9 chain skills into Cursor's project or global skill directory through the open skills CLI:
 
 ```bash
-npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a cursor -y
+npx skills add fastxyz/skill-optimizer \
+  --skill investigate-functionality \
+  --skill investigate-submissions \
+  --skill design-tests \
+  --skill write-tests \
+  --skill validate-tests \
+  --skill run-bench \
+  --skill analyze \
+  --skill improve \
+  --skill validate \
+  -a cursor -y
 ```
 
 Cursor can also import remote skills from GitHub in Settings -> Rules -> Project Rules -> Add Rule -> Remote Rule (Github).
 
 ## Plugin metadata
 
-This repository includes `.cursor-plugin/plugin.json` for Cursor-compatible plugin metadata. The canonical skill remains `skills/skill-optimizer/SKILL.md`.
+This repository includes `.cursor-plugin/plugin.json` for Cursor-compatible plugin metadata. The skills live at `skills/<chain-skill>/`; the plugin manifest exposes all 9 as a chain that triggers based on the user's intent (investigate, design tests, run bench, analyze, improve, validate).

.gitignore

@@ -55,6 +55,7 @@ temp/
 docs/superpowers/
 docs/plans/
 docs/specs/
+.superpowers/
 
 # Skill-optimizer generated artifacts
 .skill-optimizer/

.opencode/INSTALL.md

@@ -8,9 +8,9 @@ Add the plugin to `opencode.json` at user or project scope:
 }
 ```
 
-Restart OpenCode. The plugin registers the repository `skills/` directory so the native `skill` tool can load `skill-optimizer`.
+Restart OpenCode. The plugin registers the repository `skills/` directory so the native `skill` tool can discover the 9 chain skills (`investigate-functionality`, `investigate-submissions`, `design-tests`, `write-tests`, `validate-tests`, `run-bench`, `analyze`, `improve`, `validate`).
 
-Verify with the skill tool by listing skills or loading `skill-optimizer`.
+Verify by listing skills with the skill tool; each chain skill triggers on its own description (investigate, design tests, run bench, analyze, improve, validate).
 
 To pin a version, append a tag or commit ref:
 

AGENTS.md

@@ -20,7 +20,9 @@ npx tsx src/cli.ts run-suite --help
 - `src/cli.ts`: public CLI entrypoint
 - `src/workbench/`: workbench case loading, suite loading, Docker runner, Pi agent, graders, and traces
 - `docker/workbench-runner.Dockerfile`: generic non-root container image for setup, agent, grade, and cleanup phases
-- `skills/skill-optimizer/SKILL.md`: canonical distributable Agent Skill
+- `skills/<chain-skill>/`: the v1.4 chain of 9 user-invocable skills (investigate-functionality, investigate-submissions, design-tests, write-tests, validate-tests, run-bench, analyze, improve, validate)
+- `skills/shared/`: workflow overview, iteration protocol, subagent-dispatch rules, frontmatter discipline, workbench reference; loaded on-demand by chain skills
+- `skills/<chain-skill>/agents/`: prompt templates dispatched by chain skills via the Agent tool
 - `.claude-plugin/`, `.codex-plugin/`, `.cursor-plugin/`, `.opencode/`: cross-agent plugin manifests and install support
 - `.agents/plugins/marketplace.json`: Codex repo marketplace entry for the root plugin
 - `gemini-extension.json`, `GEMINI.md`: Gemini extension metadata and context file
@@ -47,7 +49,7 @@ Keep the README installation section aligned with packaged plugin metadata:
 - Cases use `graders: [{ name, command }]`; legacy `check:` and `artifacts:` are invalid.
 - Graders are the acceptance contract; evaluate outputs from `/work`, generated artifacts, `answer.json`, `trace.jsonl`, and result state.
 - The agent phase sees only `/work`, not `/case` or `/results`.
-- Keep plugin metadata pointed at the canonical `skills/skill-optimizer/SKILL.md`; do not create divergent skill copies.
+- Keep plugin metadata pointed at every chain skill under `skills/<chain-skill>/`; do not create divergent skill copies.
 - Codex plugin metadata lives in `.codex-plugin/plugin.json`; the repo marketplace lives in `.agents/plugins/marketplace.json` and points at `./`.
 - Provider install docs should link to the same canonical skill/plugin metadata, not separate skill copies.
 - Do not commit `.skill-eval/`, `.results/`, `.env`, or credentials.

CLAUDE.md

@@ -4,7 +4,7 @@
 
 `skill-optimizer` is a Docker workbench for running and grading agent skill eval cases. The current public CLI centers on `run-case` and `run-suite`.
 
-The workbench gives an agent an isolated Docker `/work` directory, captures traces, and grades deterministic local outcomes from files, command logs, generated artifacts, or other workspace state.
+The workbench drives a host-side ACP (Agent Client Protocol) client against per-trial Docker containers that run one of five agent CLIs (claude-agent-acp, codex-acp, gemini, opencode, pi-acp). Each trial gets an isolated `/work` directory; raw ACP messages are captured to `trace.jsonl`; graders evaluate deterministic local outcomes from files, command logs, generated artifacts, or other workspace state.
 
 ## Key Commands
 
@@ -20,10 +20,15 @@ npx tsx src/cli.ts run-suite --help
 ## Important Files
 
 - `src/cli.ts`: public CLI entrypoint
-- `src/workbench/`: workbench case loading, suite loading, Docker runner, Pi agent, graders, and traces
-- `docker/workbench-runner.Dockerfile`: generic non-root container image for setup, agent, grade, and cleanup phases
-- `skills/skill-optimizer/SKILL.md`: canonical distributable Agent Skill
-- `skills/skill-optimizer/references/workbench.md`: detailed workbench schema and usage reference
+- `src/workbench/`: case loading, suite loading, host-side Docker runner, graders
+- `src/workbench/acp/`: ACP transport, client wrapper, auth, skill deployment, MCP config writers, trace recorder
+- `src/workbench/agents/`: 5-agent registry (claude-agent-acp, codex-acp, gemini, opencode, pi-acp) + install snippets
+- `src/workbench/parse-trace.ts`: helpers for reading the raw ACP `trace.jsonl` (`iterMessages`, `iterToolCalls`, `computeMetrics`, etc.)
+- `docker/skill-optimizer-agent.Dockerfile`: container image with all 5 agent CLIs pre-baked
+- `docker/pi-acp-launcher.sh`: wrapper that bridges `SKILL_OPT_PROVIDER_*` env vars into pi-acp's `OPENROUTER_API_KEY`
+- `skills/<chain-skill>/`: the v1.4 chain of 9 user-invocable skills (investigate-functionality, investigate-submissions, design-tests, write-tests, validate-tests, run-bench, analyze, improve, validate)
+- `skills/shared/`: workflow overview, iteration protocol, subagent-dispatch rules, frontmatter discipline, and the workbench schema reference; loaded on-demand by chain skills
+- `skills/<chain-skill>/agents/`: prompt templates dispatched by chain skills via the Agent tool
 - `.claude-plugin/`, `.codex-plugin/`, `.cursor-plugin/`, `.opencode/`: cross-agent plugin manifests and install support
 - `.agents/plugins/marketplace.json`: Codex repo marketplace entry for the root plugin
 - `gemini-extension.json`, `GEMINI.md`: Gemini extension metadata and context file
@@ -45,12 +50,14 @@ Keep the README installation section aligned with packaged plugin metadata:
 ## Invariants
 
 - Keep evaluation static: extraction and matching are allowed; do not execute model-produced code outside the Docker workbench as part of evaluation.
-- `run-suite` uses models from `suite.yml`; do not add a `run-suite --models` override.
-- Keep OpenRouter model refs as `openrouter/...`; real model runs require `OPENROUTER_API_KEY`.
+- Every `case.yml` must declare `agent:` (one of `claude-agent-acp`, `codex-acp`, `gemini`, `opencode`, `pi-acp`); the loader fails loud on missing/unknown agents.
+- `suite.yml` declares its agent+model matrix via `runs: [{ agent, model }]`; legacy `models:` is rejected. `run-suite` uses what `suite.yml` declares; do not add a `--models` override.
+- Only `pi-acp` requires `openrouter/...` model refs; native ACP agents pass their own model strings through unchanged.
 - Cases use `graders: [{ name, command }]`; legacy `check:` and `artifacts:` are invalid.
 - Graders are the acceptance contract; evaluate outputs from `/work`, generated artifacts, `answer.json`, `trace.jsonl`, and result state.
 - The agent phase sees only `/work`, not `/case` or `/results`.
-- Keep plugin metadata pointed at the canonical `skills/skill-optimizer/SKILL.md`; do not create divergent skill copies.
+- `trace.jsonl` is raw ACP wire format (JSON-RPC envelopes); use `parse-trace.ts` helpers rather than parsing the file by hand. See [`skills/shared/acp-trace-format.md`](skills/shared/acp-trace-format.md).
+- Keep plugin metadata pointed at every chain skill under `skills/<chain-skill>/`; do not create divergent skill copies.
 - Codex plugin metadata lives in `.codex-plugin/plugin.json`; the repo marketplace lives in `.agents/plugins/marketplace.json` and points at `./`.
 - Provider install docs should link to the same canonical skill/plugin metadata, not separate skill copies.
 - Do not commit `.skill-eval/`, `.results/`, `.env`, or credentials.
@@ -59,6 +66,6 @@ Keep the README installation section aligned with packaged plugin metadata:
 
 - Run `npm run typecheck` after TypeScript changes.
 - Run `npm test` before finishing behavior changes.
-- For Docker runner or image changes, also run `docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .`.
+- For Docker runner or image changes, also run `docker build -t skill-optimizer-agent:local -f docker/skill-optimizer-agent.Dockerfile .`.
 - For CLI/docs changes, verify `npx tsx src/cli.ts --help` if touched docs mention CLI behavior.
 - For plugin/package metadata changes, run `npx tsx tests/smoke-skill-distribution.ts` and verify `npm pack --dry-run --json` includes required plugin files without result/cache directories.

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributing to skill-optimizer
 
-Thanks for contributing! This project is a small, opinionated Docker workbench for evaluating agent skills. Changes should preserve deterministic grading, isolated agent workspaces, and the canonical `skills/skill-optimizer/SKILL.md` distribution path.
+Thanks for contributing! This project is a small, opinionated Docker workbench for evaluating agent skills, plus a 9-step chain of Agent Skills (`skills/<chain-skill>/`) that orchestrates investigation, test design, bench runs, analysis, and improvement of a target skill, plus an `autopilot` driver that walks the chain end-to-end. Changes should preserve deterministic grading, isolated agent workspaces, and the chain skills' distribution paths.
 
 ## Installing The Skill
 
@@ -22,10 +22,13 @@ All three commands must pass before opening a PR when code changes are involved.
 ## Project layout
 
 - `src/cli.ts` — public CLI entry point for `run-case` and `run-suite`.
-- `src/workbench/` — case/suite loading, Docker runner, Pi agent wiring, graders, traces, metrics, MCP support, and trial aggregation.
-- `docker/workbench-runner.Dockerfile` — non-root container image for setup, agent, grade, and cleanup phases.
-- `skills/skill-optimizer/SKILL.md` — canonical distributable Agent Skill.
-- `skills/skill-optimizer/references/workbench.md` — detailed workbench schema and authoring reference.
+- `src/workbench/` — case/suite loading, host-side Docker runner, graders, metrics, MCP support, trial aggregation.
+- `src/workbench/acp/` — ACP transport, client wrapper, auth resolution, skill deployment, per-agent MCP config writer, trace recorder.
+- `src/workbench/agents/` — 5-agent registry + Dockerfile install snippet generator.
+- `docker/skill-optimizer-agent.Dockerfile` — container image with all 5 agent CLIs pre-baked.
+- `skills/<chain-skill>/` — the 9-step chain of user-invocable Agent Skills (investigate-functionality, investigate-submissions, design-tests, write-tests, validate-tests, run-bench, analyze, improve, validate) plus the `autopilot` driver.
+- `skills/shared/` — workflow overview, iteration protocol, subagent-dispatch rules, frontmatter discipline, workbench schema reference; loaded on-demand by chain skills.
+- `skills/<chain-skill>/agents/` — prompt templates dispatched by chain skills via the Agent tool.
 - `examples/workbench/` — packaged example suites.
 - `.claude-plugin/`, `.codex-plugin/`, `.cursor-plugin/`, `.opencode/`, `.agents/plugins/marketplace.json`, `gemini-extension.json`, `GEMINI.md` — cross-agent plugin and extension metadata.
 - `tests/` — hand-rolled smoke tests (`tsx tests/smoke-*.ts`).
@@ -42,23 +45,29 @@ All three commands must pass before opening a PR when code changes are involved.
 ## Workbench invariants
 
 - Keep evaluation static: extraction and matching are allowed; do not execute model-produced code outside the Docker workbench as part of evaluation.
-- Use only `openrouter/...` model refs; real model runs require `OPENROUTER_API_KEY`.
-- `run-suite` uses models from `suite.yml`; do not add a `run-suite --models` override.
+- Every `case.yml` must declare `agent:` (one of `claude-agent-acp`, `codex-acp`, `gemini`, `opencode`, `pi-acp`); the loader fails loud on missing/unknown.
+- `suite.yml` declares its `runs: [{ agent, model }]` matrix; legacy `models:` is rejected. `run-suite` uses what `suite.yml` declares; do not add a `--models` override.
+- Only `pi-acp` requires `openrouter/...` model refs; native ACP agents pass their own model strings through unchanged.
 - Cases use `graders: [{ name, command }]`; legacy `check:` and `artifacts:` are invalid.
 - The agent phase sees only `/work`, not `/case`, `/results`, graders, hidden answers, or hidden metadata.
-- Keep plugin metadata pointed at the canonical `skills/skill-optimizer/SKILL.md`; do not create divergent skill copies.
+- `trace.jsonl` is raw ACP wire format (JSON-RPC envelopes); use `src/workbench/parse-trace.ts` helpers (`iterMessages`, `iterToolCalls`, `computeMetrics`) rather than parsing the file by hand.
+- Keep plugin metadata pointed at every chain skill under `skills/<chain-skill>/`; do not create divergent skill copies.
 
 ## Testing guidance
 
 - Run `npm run typecheck` after TypeScript changes.
 - Run `npm test` before finishing behavior changes.
-- For Docker runner or image changes, also run `docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .`.
+- For Docker runner or image changes, also run `docker build -t skill-optimizer-agent:local -f docker/skill-optimizer-agent.Dockerfile .`.
 - For CLI/docs changes, verify `npx tsx src/cli.ts --help` if touched docs mention CLI behavior.
 - For plugin/package metadata changes, run `npx tsx tests/smoke-skill-distribution.ts` and verify `npm pack --dry-run --json` includes required plugin files without result/cache directories.
 
+## Agent capabilities and limitations (v1)
+
+MCP server support is available for `claude-agent-acp`, `codex-acp`, `gemini`, and `opencode` agents. `pi-acp` MCP support is deferred to v1.1 pending documentation of pi-acp's native MCP config path.
+
 ## Adding workbench capabilities
 
-Keep new capabilities small and deterministic. Add validation in the relevant loader, tests in `tests/smoke-workbench-*.ts`, and docs in `skills/skill-optimizer/references/workbench.md` or `docs/workbench.md` when users need to author new YAML fields or understand new runtime behavior.
+Keep new capabilities small and deterministic. Add validation in the relevant loader, tests in `tests/smoke-workbench-*.ts`, and docs in `skills/shared/workbench.md` or `docs/workbench.md` when users need to author new YAML fields or understand new runtime behavior.
 
 ## Commit style
 

GEMINI.md

@@ -1,5 +1,5 @@
 @./AGENTS.md
 @./README.md
 @./CONTRIBUTING.md
-@./skills/skill-optimizer/SKILL.md
-@./skills/skill-optimizer/references/workbench.md
+@./skills/shared/workflow.md
+@./skills/shared/workbench.md