v1.1

.github/workflows/ci.yml

@@ -1,12 +1,13 @@
-name: CI
+name: Build & Test
 
 on:
   pull_request:
   push:
-    branches: [main]
+    branches: [main, development, staging]
 
 jobs:
   build-test:
+    name: "Build & Test (node ${{ matrix.node }})"
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false

.gitignore

@@ -48,8 +48,10 @@ coverage/
 tmp/
 temp/
 
-# Superpowers planning artifacts
+# Superpowers planning artifacts and internal dev docs
 docs/superpowers/
+docs/plans/
+docs/specs/
 
 # Skill-optimizer generated artifacts
 .skill-optimizer/

CHANGELOG.md

@@ -1,6 +1,33 @@
 # Changelog
 
-## Unreleased
+## 1.1.0 — 2026-04-16
+
+### Breaking Changes
+
+The following public API exports have been removed. Update imports to use the canonical names:
+
+| Removed | Replacement |
+|---------|-------------|
+| `CodeModeConfig` | `SdkSurfaceConfig` |
+| `McpModeConfig` | `McpSurfaceConfig` |
+| `ExpectedTool` | `ExpectedAction` |
+| `ToolMatch` | `ActionMatch` |
+| `LEGACY_PROJECT_CONFIG_NAME` | hard-code `"skill-optimizer.json"` |
+| `toLegacyOptimizeManifest` | removed with no replacement |
+| `SurfaceSnapshotArg` | removed with no replacement |
+
+`TaskResult` fields renamed: `toolMatches` → `actionMatches`, `hallucinatedCalls` → `hallucinatedActions` (on `metrics`), `unnecessaryCalls` → `unnecessaryActions` (on `metrics`). `loadReport` does not validate old field names — old report JSON files may produce unexpected output in detail views. Re-run the benchmark to generate a current-format report.
+
+Existing `tasks.json` files using `expected_tools` (instead of `expected_actions`) or `method` (instead of `name`) on action entries will now fail to load with an error. Update affected task files: rename `expected_tools` to `expected_actions` and rename each action's `method` field to `name`.
+
+The config file `skill-benchmark.json` is no longer auto-detected. Rename it to `skill-optimizer.json`.
+
+### Added
+- **prompt surface type** — benchmark and optimize prompt templates, Claude Code skills, and agent instructions. Discovers phases and capabilities from markdown, evaluates output quality with content-based criteria.
+
+### Fixed
+
+- **benchmark:** Strip provider prefix from model ID when using direct `anthropic` or `openai` formats. Previously, `anthropic/claude-sonnet-4-6` was sent as-is to the Anthropic API, which expects `claude-sonnet-4-6`. The `pi` format is unaffected.
 
 ## 1.0.0 — 2026-04-14
 

CLAUDE.md

@@ -4,7 +4,7 @@
 
 `skill-optimizer` measures whether LLMs pick the right SDK methods, CLI commands, or MCP tools from docs and task prompts, and can run a benchmark-driven optimization loop over an allowed target repo.
 
-The repo now has four important layers:
+The repo has five layers:
 
 - `src/project/`: unified `skill-optimizer.json` config loading, validation, and path resolution
 - `src/runtime/pi/`: shared Pi auth/model/runtime helpers
@@ -77,7 +77,7 @@ npx tsx src/cli.ts optimize --config ./.tmp/mock-repos/mcp-tracker-demo/skill-op
 - Prefer small changes in the existing architecture over broad refactors.
 - When updating config or project types, also update the README examples and any scaffolding in `src/benchmark/init.ts` if needed.
 - When changing optimizer behavior, verify both the loop and the unified project defaults still agree.
-- Code-first surface discovery is now active for `sdk`, `cli`, and `mcp` via `target.discovery.sources`. Explicit manifests/declared surface metadata still exist as transitional internal fallbacks, but new public examples should prefer code-first discovery.
+- Code-first surface discovery is the preferred mode for `sdk`, `cli`, and `mcp` via `target.discovery.sources`. Explicit manifest files (`target.cli.commands`, `target.mcp.tools`, `target.discovery.fallbackManifest`) are supported for projects that cannot use code-first discovery.
 - Be careful around mock repo references: code may support template names that are not currently present in the working tree.
 
 ## Testing Guidance
@@ -86,6 +86,26 @@ npx tsx src/cli.ts optimize --config ./.tmp/mock-repos/mcp-tracker-demo/skill-op
 - Run `npm test` before finishing when behavior changes may affect extraction, evaluation, reporting, or optimizer flow.
 - For CLI-only or docs-only changes, at minimum verify `npx tsx src/cli.ts --help` still works if the touched docs reference CLI behavior.
 
+## Model ID Convention
+
+Model IDs use a provider-prefixed format. The prefix determines how the request is routed:
+
+```
+openrouter/<provider>/<model-slug>   — routed through OpenRouter (format: "pi")
+anthropic/<model-slug>               — direct Anthropic API    (format: "anthropic")
+openai/<model-slug>                  — direct OpenAI API       (format: "openai")
+```
+
+**Version segments use hyphens, not dots**, regardless of provider. Examples:
+- `openrouter/anthropic/claude-sonnet-4-6` ✓ (not `claude-sonnet-4.6`)
+- `openrouter/google/gemini-2-5-flash` ✓ (not `gemini-2.5-flash`)
+- `openrouter/deepseek/deepseek-v3-2` ✓ (not `deepseek-v3.2`)
+- `openai/gpt-5-4` ✓ (not `openai/gpt-5.4`)
+
+`src/project/validate.ts` warns on dot-notation version segments for all non-`openai/` model IDs (`model-id-bad-format`) and `src/project/fix.ts` auto-corrects them. (OpenAI's own slugs use dots in some cases, so `openai/` is exempt.) When adding new model presets to `src/init/scaffold.ts`, `src/init/wizard.ts`, or `src/benchmark/init.ts`, always use hyphens in the model ID path.
+
+Display names (`name:` / `label:` fields) are human-readable and should keep dots (e.g. `'Claude Sonnet 4.6'`, `'Gemini 2.5 Flash'`).
+
 ## Environment Notes
 
 - Do not commit `.env` or secrets.

CONTRIBUTING.md

@@ -5,7 +5,7 @@ Thanks for contributing! This project is a small, opinionated tool — changes s
 ## Local workflow
 
 ```bash
-git clone https://github.com/bucurdavid/skill-optimizer
+git clone https://github.com/fastxyz/skill-optimizer
 cd skill-optimizer
 npm install
 npm run typecheck
@@ -35,11 +35,11 @@ All three commands must pass before opening a PR.
 
 ## Adding a surface type
 
-Follow the shape of existing discoverers in `src/project/snapshot.ts`. A surface discoverer returns `ActionDefinition[]`. Then:
+A surface discoverer returns `ActionDefinition[]`. To add one:
 
 1. Extend `BenchmarkSurface` in `src/benchmark/types.ts`.
 2. Add a branch to `src/project/validate.ts` and `src/project/resolve.ts`.
-3. Register the discoverer in `src/project/snapshot.ts`.
+3. Implement the new code-first discoverer in `src/actions/discover.ts`, then register it in `src/project/snapshot.ts` (the dispatcher that routes surfaces to discoverers).
 4. Add a discovery smoke test.
 
 ## Adding an LLM provider

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2026 David Bucur
+Copyright (c) 2026 Pi Squared Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -4,7 +4,7 @@ Benchmark and self-optimize SDK, CLI, and MCP guidance so every agent model can
 
 skill-optimizer runs your SDK / CLI / MCP docs against multiple LLMs, measures whether they call the right actions with the right arguments, and iteratively rewrites your `SKILL.md` / docs until a floor score is met across every model.
 
-**Requirements:** Node.js 20+, an [OpenRouter](https://openrouter.ai) API key.
+**Requirements:** Node.js 20+, plus either an [OpenRouter](https://openrouter.ai) API key or a local Codex login when using direct OpenAI models.
 
 ## Installation
 
@@ -22,10 +22,24 @@ npm link        # makes `skill-optimizer` available globally
 export OPENROUTER_API_KEY=sk-or-...
 ```
 
+For direct OpenAI runs you can also use your local Codex browser login instead of exporting `OPENAI_API_KEY`:
+
+```json
+{
+  "benchmark": {
+    "format": "pi",
+    "authMode": "codex",
+    "models": [
+      { "id": "openai/gpt-5-4", "name": "GPT-5.4", "tier": "flagship" }
+    ]
+  }
+}
+```
+
 **Step 1 — Scaffold config** (run from your project root):
 
 ```bash
-npx skill-optimizer init cli       # or: init sdk, init mcp
+npx skill-optimizer init cli       # or: init sdk, init mcp, init prompt
 ```
 
 The wizard asks for your repo path, models to benchmark, and where your `SKILL.md` lives. It creates a `skill-optimizer/` directory:
@@ -72,7 +86,17 @@ npx skill-optimizer init --answers answers.json
 {
   "surface": "cli",
   "repoPath": "/absolute/path/to/your-repo",
-  "models": ["openrouter/anthropic/claude-sonnet-4.6", "openrouter/openai/gpt-4o"],
+  "models": [
+    "openrouter/anthropic/claude-sonnet-4-6",
+    "openrouter/deepseek/deepseek-v3-2",
+    "openrouter/google/gemini-2-5-flash",
+    "openrouter/qwen/qwen3-5-397b-a17b",
+    "openrouter/moonshotai/kimi-k2-5",
+    "openrouter/z-ai/glm-5-1",
+    "openrouter/minimax/minimax-m2-7",
+    "openrouter/google/gemma-4-31b-it",
+    "openrouter/meta-llama/llama-4-maverick"
+  ],
   "maxTasks": 20,
   "maxIterations": 5,
   "entryFile": "src/cli.ts"
@@ -86,11 +110,28 @@ npx skill-optimizer init --answers answers.json
 | `target.repoPath` | Root of the project being benchmarked | Absolute or relative path to your repo |
 | `target.discovery.sources` | Source files to scan for callable methods/commands/tools | e.g. `["../src/index.ts"]` or `["../src/server.ts"]` |
 | `target.skill` | Docs file the optimizer will edit | Path to your `SKILL.md` or equivalent guidance doc |
-| `benchmark.models` | Models to benchmark | Valid [OpenRouter](https://openrouter.ai/models) model IDs |
+| `benchmark.models` | Models to benchmark | Model IDs with provider prefix: `openrouter/<provider>/<model>` (via OpenRouter), `anthropic/<model>` (direct Anthropic), `openai/<model>` (direct OpenAI) |
+| `benchmark.authMode` | How model auth is resolved | `env` (default), `codex`, or `auto` |
+
+### Prompt templates / Claude Code skills
+
+Benchmark how well models follow your prompt templates:
+
+```bash
+skill-optimizer init prompt
+skill-optimizer benchmark
+```
+
+The prompt surface discovers phases and capabilities from your SKILL.md,
+generates scenario-based tasks, and evaluates output quality — not just
+tool calls. It scores responses on required sections, format patterns,
+forbidden keywords, and structural elements (code blocks, numbered lists,
+tables). This lets you optimize prompt templates the same way you optimize
+SDK/CLI/MCP guidance.
 
 ## How it works
 
-1. **Discover** callable surface (SDK methods / CLI commands / MCP tools) via tree-sitter or a manifest.
+1. **Discover** callable surface (SDK methods / CLI commands / MCP tools / prompt phases) via tree-sitter, manifest, or markdown parsing.
 2. **Scope** the surface with `target.scope.include` / `target.scope.exclude` globs.
 3. **Generate tasks** — one prompt per in-scope action, coverage-guaranteed.
 4. **Benchmark** — every configured model attempts every task; static evaluator checks action calls + args.
@@ -150,7 +191,7 @@ No per-failure LLM calls — feedback is deterministic (structured failure detai
 
 ## Dependencies
 
-The optimizer's coding agent is powered by `@mariozechner/pi-coding-agent` — a small OSS wrapper around OpenRouter that handles agent sessions and tool loops. Models are accessed through [OpenRouter](https://openrouter.ai/) — you need one API key for everything.
+The optimizer's coding agent is powered by `@mariozechner/pi-coding-agent`. OpenRouter-backed runs still use your configured API key env var. Direct OpenAI runs can use either `OPENAI_API_KEY` or the browser-login tokens that Codex stores in `~/.codex/auth.json`.
 
 ## Troubleshooting
 
@@ -159,14 +200,14 @@ The optimizer's coding agent is powered by `@mariozechner/pi-coding-agent` — a
 export OPENROUTER_API_KEY=sk-or-...
 ```
 
+**Using Codex auth**: Set `benchmark.authMode` (and optionally `optimize.authMode`) to `"codex"` or `"auto"` and use direct OpenAI model refs such as `openai/gpt-5-4`. Codex auth only applies to the `openai` provider and reads either a browser-login access token or `OPENAI_API_KEY` from `~/.codex/auth.json`. Alternatively, set `benchmark.format` to `"openai"` with `authMode: "codex"` and `openai/...` model IDs — the client bridges to the Pi/Codex path automatically.
+
 **Dirty git**: The optimizer requires a clean git state in the target repo (`requireCleanGit: true` by default). Commit or stash uncommitted changes before running. Note: the optimizer never writes to the target repo's skill file — it works from local versioned copies in `.skill-optimizer/`.
 
 **`maxTasks < scope_size`**: `benchmark.taskGeneration.maxTasks` must be >= the number of in-scope actions. Run `npx skill-optimizer --dry-run --config ./skill-optimizer.json` to see the count without making LLM calls.
 
 **Empty scope**: `target.scope.include` matched nothing. Check your glob patterns — remember `*` matches everything including dots.
 
-**Legacy `skill-benchmark.json`**: Rename it to `skill-optimizer.json`. The loader will tell you if it finds the old name.
-
 ## Contributing
 
 See [CONTRIBUTING.md](./CONTRIBUTING.md).

SKILL/SKILL.md

@@ -0,0 +1,67 @@
+---
+name: skill-optimizer
+description: >
+  Benchmark and optimize SDK, CLI, MCP, and prompt documentation so every LLM
+  model can reliably call the right actions with correct arguments. Use when
+  setting up skill-optimizer for a project, running benchmarks, interpreting
+  results, optimizing SKILL.md files, or diagnosing configuration issues. Also
+  use when working inside the skill-optimizer repository itself — for running
+  against mock repos, testing changes, or understanding the codebase.
+---
+
+# skill-optimizer
+
+Benchmark your SDK / CLI / MCP / prompt docs against multiple LLMs, measure whether they call the right actions with the right arguments, and iteratively rewrite your guidance until a quality floor is met across every model.
+
+## Context Detection
+
+Before doing anything, figure out where you are:
+
+1. **Look for `skill-optimizer.json`** (in CWD or parent directories). If found, you are in a **configured target project**. Use that file path as `<config-path>` in all commands below.
+
+2. **Look for `src/cli.ts` and a `package.json` with `"name": "skill-optimizer"`**. If found, you are in the **optimizer repo itself**. You can use dev commands directly (`npm run build`, `npm test`, `npx tsx src/cli.ts`). To benchmark a target, either use the mock repos in `mock-repos/` or point `--config` at an external project's config.
+
+3. **Neither found** — you are in an **unconfigured target project**. Read `references/setup.md` to scaffold a config before proceeding.
+
+## Quick Reference
+
+| Task | Command |
+|------|---------|
+| Init config | `npx skill-optimizer init cli\|sdk\|mcp\|prompt` |
+| Init (non-interactive) | `npx skill-optimizer init cli --yes` |
+| Import CLI commands | `npx skill-optimizer import-commands --from ./src/cli.ts` |
+| Import (binary scrape) | `npx skill-optimizer import-commands --from my-cli --scrape` |
+| Diagnose config | `npx skill-optimizer doctor --config <config-path>` |
+| Auto-fix config | `npx skill-optimizer doctor --fix --config <config-path>` |
+| Dry run (no LLM calls) | `npx skill-optimizer run --dry-run --config <config-path>` |
+| Run benchmark | `npx skill-optimizer run --config <config-path>` |
+| Generate tasks only | `npx skill-optimizer generate-tasks --config <config-path>` |
+| Run optimizer | `npx skill-optimizer optimize --config <config-path>` |
+| Compare two runs | `npx skill-optimizer compare --baseline a.json --current b.json` |
+
+`<config-path>` is the path to your `skill-optimizer.json` — typically `./skill-optimizer/skill-optimizer.json` after running `init`, or wherever you placed it.
+
+## What Do You Need?
+
+Read the reference file that matches your current goal:
+
+| Goal | Reference |
+|------|-----------|
+| Set up skill-optimizer for a project (first time) | Read `references/setup.md` |
+| Run a benchmark or understand results | Read `references/benchmark.md` |
+| Automatically optimize a SKILL.md | Read `references/optimize.md` |
+| Understand config options | Read `references/config.md` |
+
+If you are in an **unconfigured project** (context detection case 3), start with `references/setup.md`.
+
+## Key Concepts
+
+**Surfaces** — The callable interface of your project: SDK methods, CLI commands, MCP tools, or prompt templates. Skill-optimizer discovers these via tree-sitter code analysis, manifest files, or markdown parsing.
+
+**Static evaluation** — Benchmark evaluation never executes generated code. Actions are extracted from model responses via pattern matching and compared structurally against expected calls. This makes benchmarks safe and repeatable.
+
+**Verdict gates** — Two thresholds must both pass for a benchmark to receive a PASS verdict: `perModelFloor` (each model individually meets a minimum score) and `targetWeightedAverage` (the weighted mean across all models meets a target). A single model below the floor fails the entire run.
+
+**Safety boundary** — The optimizer never modifies your original SKILL.md. It creates versioned copies in `.skill-optimizer/skill-v{N}.md` and only accepts mutations that improve scores without dropping any model below the floor. It does not modify tracked source files, but the generated artifacts appear under `.skill-optimizer/` — add that directory to your `.gitignore`.
+
+**LLM routing** — By default (`format: "pi"`), all benchmark calls route through [OpenRouter](https://openrouter.ai) and need `OPENROUTER_API_KEY`. You can also call providers directly: `format: "anthropic"` uses the Anthropic API directly (`ANTHROPIC_API_KEY`), and `format: "openai"` uses the OpenAI API directly (`OPENAI_API_KEY`), with optional Codex browser-login auth via `authMode: "codex"`. The model ID prefix must match the format — see `references/config.md` for the full mapping.