Switch Stage-2 review harness from Claude Code to pi

README.md

@@ -8,7 +8,7 @@ Org-wide GitHub defaults and shared reusable workflows.
 two-stage AI code review on a pull request:
 
 1. **Codex first pass** — reviews the diff against the PR's JIRA ticket and writes findings.
-2. **Claude verify & synthesize** — validates Codex's findings, reviews the diff
+2. **pi verify & synthesize** — validates Codex's findings, reviews the diff
    independently (grepping for callers/tests, factoring in the existing PR discussion),
    checks the change against the ticket's acceptance criteria, and decides a single verdict.
 
@@ -17,10 +17,10 @@ The **BiggiePockets** service account then submits the resulting `approve` /
 ticket can't be fetched), the review degrades gracefully to a diff-based review instead of
 failing.
 
-Codex and Claude run as separate GitHub Actions jobs. Codex uploads the reviewed commit's
-diff, ticket/discussion context, and findings as a short-lived artifact; Claude downloads
-that immutable handoff. If Claude is rate-limited, use **Re-run failed jobs** on the workflow
-run. GitHub reruns only the Claude job, reusing the completed Codex pass instead of invoking
+Codex and pi run as separate GitHub Actions jobs. Codex uploads the reviewed commit's
+diff, ticket/discussion context, and findings as a short-lived artifact; pi downloads
+that immutable handoff. If the pi pass is rate-limited, use **Re-run failed jobs** on the workflow
+run. GitHub reruns only the pi job, reusing the completed Codex pass instead of invoking
 Codex again.
 
 The review logic lives centrally in this repo. Each consuming repo only adds a thin
@@ -30,13 +30,13 @@ The review logic lives centrally in this repo. Each consuming repo only adds a t
 
 Do this once per repo you want BiggiePockets to review.
 
-#### 1. Install the Claude GitHub app
+#### 1. Install nothing
 
-The Claude verification stage uses [`anthropics/claude-code-action`](https://github.com/anthropics/claude-code-action).
-Install the official [Claude GitHub app](https://github.com/apps/claude) for the organization
-once; you do **not** need to reinstall it per repo. Claude's model requests authenticate
-through OpenRouter using the shared `OPENROUTER_API_KEY` described below, so no personal
-Claude Code OAuth token is required.
+The Stage-2 verification stage runs [`@earendil-works/pi-coding-agent`](https://www.npmjs.com/package/@earendil-works/pi-coding-agent),
+a plain npm CLI that the workflow installs onto the runner itself (`npm install -g`).
+There is no GitHub app, no OIDC, and no per-repo installation. pi authenticates to
+OpenRouter with the shared `OPENROUTER_API_KEY` described below — no OAuth login or
+personal token is required.
 
 #### 2. Add the caller workflow
 
@@ -57,12 +57,11 @@ on:
         required: true
         type: string
 
-# The reusable workflow's jobs need these scopes. Declare them explicitly so the
+# The reusable workflow's jobs need read scopes. Declare them explicitly so the
 # caller works regardless of the repo's default token permissions.
 permissions:
   contents: read
   pull-requests: read
-  id-token: write
 
 jobs:
   review:
@@ -83,7 +82,7 @@ jobs:
 #### 3. Make the secrets available
 
 The reusable workflow consumes several secrets via `secrets: inherit`: credentials for the
-the AI review provider (`OPENROUTER_API_KEY`, shared by both the Codex and Claude stages),
+the AI review provider (`OPENROUTER_API_KEY`, shared by both the Codex and pi stages),
 an Atlassian email + API token to fetch the PR's JIRA ticket for intent, and a personal access
 token for the BiggiePockets service account that submits the review. Configure them as
 **organization secrets** (recommended — set once, available to every repo) or as per-repo
@@ -92,8 +91,8 @@ secrets if you prefer to scope them.
 It also reports per-review traces to the `biggiepockets-review` app in Datadog LLM
 Observability via `secrets.DATADOG_API_KEY`: verdict, timing, prompt template and version
 (tracked as prompts, see below), the model each
-stage ran (`CODEX_MODEL`/`CLAUDE_MODEL` env vars in the workflow — both are OpenRouter
-model slugs and must be set), and the actual findings text from Codex and the summary Claude wrote,
+stage ran (`CODEX_MODEL`/`PI_MODEL` env vars in the workflow — both are OpenRouter
+model slugs and must be set), and the actual findings text from Codex and the summary pi wrote,
 so review quality is inspectable, not just counted. This secret is optional — reviews still
 run and post normally without it, but no metrics are reported.
 
@@ -114,21 +113,25 @@ OpenRouter, whose slugs look like `openai/gpt-5.6-sol`, so `scripts/llm-usage.py
 splits the slug into `model_name: gpt-5.6-sol` / `model_provider: openai` and records
 the routing as a `gateway:openrouter` tag.
 
-Cost itself is never computed here, and there is no rate table in the repository: a
-list price committed to a file goes stale silently and would be reported with the same
-confidence as a real one. Instead each span carries whichever of the two things
-Datadog needs. For a model in the catalog, token counts are enough. For one it does
-not carry, the span reports a `total_cost` metric — the amount OpenRouter says it
-charged, taken from the `cost` field it returns on every response, where that figure
-survives into what the tool wrote to disk.
-
-The two passes record their usage differently. Claude Code writes a `usage` object to
-its execution output. Codex writes running token counters to a session rollout on its
-own runner, and since its action exposes no usage output, the workflow reads that
-rollout in the Codex job and hands the totals to the reporting job. In both cases only
-usage objects are read — never message content, transcripts, prompts, or diffs. Claude
-Code's own `total_cost_usd` is ignored: it is computed against Anthropic's list prices,
-while these passes are billed by OpenRouter for a non-Anthropic model.
+Cost is never computed in the reporting script: `scripts/llm-usage.py` holds no rate
+table. Instead each span carries whichever of the two things Datadog needs. For a model
+in the catalog, token counts are enough. For one it does not carry, the span reports a
+`total_cost` metric taken at face value.
+
+The two passes record their usage differently. pi runs in `--mode json` and writes a
+JSON event stream; every assistant message carries a usage object with token counts and
+`cost.total` — a price computed by pi from the model's OpenRouter list rates, the same
+rates OpenRouter bills against, so it is the amount the pass is charged. The Stage-2
+model and its rates are pinned in `scripts/pi/models.json` (the catalog pi ships
+predates the model, and the live catalog refresh is a background fetch, not a startup
+step — a committed pin is what makes a fresh runner deterministic); if you roll the
+Stage-2 model, update that file in the same commit. Codex writes running token counters
+to a session rollout on its own runner, and since its action exposes no usage output,
+the workflow reads that rollout in the Codex job and hands the totals to the reporting
+job. In both cases only usage objects are read — never message content, transcripts,
+prompts, or diffs. (The Claude Code harness that previously ran Stage 2 translated
+usage into Anthropic's schema, so OpenRouter's reported cost regularly did not survive
+to the span — the pi pass records usage itself and is what this repo now trusts.)
 
 Two **organization-level variables** (`vars`, not secrets — **Settings → Secrets and
 variables → Actions → Variables** at the org level) configure where the trace lands.
@@ -141,22 +144,18 @@ Both are optional, and nothing here is committed to the repository:
   Defaults to `biggiepockets-review`.
 
 Cost tracking is best-effort and never fails a review. With no `DATADOG_API_KEY` the
-whole reporting step is skipped, and a missing execution file, an absent rollout, or
+whole reporting step is skipped, and a missing event stream, an absent rollout, or
 malformed usage data degrades to fewer metrics on the span.
 
 #### 4. Set workflow permissions
 
-The reusable workflow's jobs need `pull-requests: read` and `id-token: write` (OIDC
-authentication for `claude-code-action`). The caller YAML declares these in its
-`permissions` block, but GitHub caps those declarations at whatever the repo's default
-token setting allows. Repos set to **"Read repository contents"** (the restrictive
-default) deny `id-token: write` regardless of what the YAML says — the workflow fails
-with `startup_failure` before any job runs.
-
-Go to **Settings → Actions → General → Workflow permissions** on the target repo and set:
-
-- **"Read and write permissions"**
-- **"Allow GitHub Actions to create and approve pull requests"** (checked)
+With the move off `claude-code-action`, the review no longer needs OIDC (`id-token`),
+so the default **"Read repository contents"** setting is fine — repos that previously
+had to loosen their workflow permissions for the review can leave them restrictive.
+The reusable workflow's jobs declare `contents: read` and `pull-requests: read` only;
+the review itself is submitted with the `BIGGIEPOCKETS_PAT` secret, not the repo's
+`GITHUB_TOKEN`. Callers that already declare `id-token: write` in their caller file can
+remove it, but leaving it is harmless.
 
 #### 5. Give BiggiePockets access
 
@@ -190,16 +189,16 @@ prompts/
 ```
 
 - **Templates + shared blocks.** Each prompt references the shared rule blocks via
-  `{{@prompts/_shared/<name>.md}}`, so the Codex and Claude prompts can never drift out of
+  `{{@prompts/_shared/<name>.md}}`, so the Codex and Stage-2 prompts can never drift out of
   sync. Prompts resolve `{{PR}}`, `{{PROMPT_NAME}}`, `{{PROMPT_VERSION}}` too.
 - **Content-derived versions.** `prompt_version` is a content hash of the template plus the
   shared blocks it includes — it changes only when that prompt's text changes, not per PR
   or per arm, so Datadog LLM Obs can attribute quality to the exact prompt text that ran.
-- **One arm per pull request.** Two Claude prompts sit in the registry — `control` and
+- **One arm per pull request.** Two Stage-2 prompts sit in the registry — `control` and
   `thesis-first` — and each review runs exactly one of them. `scripts/resolve-prompts.sh`
   hashes `<repo>:<pr>` into a bucket 0-99 and assigns the PR to the experiment arm when
   that bucket falls under `experiment_split_percent` (50 today), so the arms split traffic
-  evenly and every review costs a single Claude pass. Assignment is a pure function of
+  evenly and every review costs a single Stage-2 pass. Assignment is a pure function of
   repo and PR number: re-running a review reuses the same arm, and one PR never sees two
   review styles.
 - **The assigned arm decides.** Whichever arm a PR draws writes the posted summary *and*
@@ -223,7 +222,7 @@ prompts/
 - **Datadog.** Each review is one trace, tagged with the arm that ran it:
 
   ```
-  biggiepockets.review → codex.review, claude.synthesize
+  biggiepockets.review → codex.review, pi.synthesize
   ```
 
   A tag key resolves to one value per submitted payload, so an `arm` tag is only

prompts/registry.json

@@ -1,7 +1,7 @@
 {
   "//": "Prompt registry config for the BiggiePockets review workflow.",
   "//": "",
-  "//": "Each review runs ONE Claude synthesize pass. Which arm's prompt it runs is",
+  "//": "Each review runs ONE Stage-2 synthesize pass (pi). Which arm's prompt it runs is",
   "//": "decided per pull request: scripts/resolve-prompts.sh hashes '<repo>:<pr>' into a",
   "//": "bucket 0-99, and a PR below experiment_split_percent is assigned the experiment",
   "//": "arm instead of control_arm. The assigned arm writes the summary that gets posted",
@@ -14,7 +14,7 @@
   "//": "",
   "//": "Comparison is BETWEEN pull requests: Datadog tags each review with its assigned",
   "//": "arm, so verdict rates and latency group by arm across PRs. No PR is ever reviewed",
-  "//": "by two arms, which is what keeps Claude token spend at one pass per review.",
+  "//": "by two arms, which is what keeps Stage-2 token spend at one pass per review.",
   "//": "",
   "//": "prompts/<name>.md are versioned templates; shared rule blocks live in",
   "//": "prompts/_shared/ and are injected via {{@path}} markers. Prompt versions are",

scripts/llm-usage.py

@@ -10,15 +10,21 @@
 
 Where the numbers come from:
 
-- Both passes reach models through OpenRouter, which reports what it charged in the
-  `cost` field of every response's usage object. Where that figure survives into what
-  the tool wrote to disk, it is the authoritative cost for the pass and is emitted as
-  `total_cost` — a real charge, not an estimate.
+- Both passes reach models through OpenRouter, which reports what it charged in
+  the `cost` field of every response's usage object. Where that figure survives into
+  what the tool wrote to disk, it is the authoritative cost for the pass and is
+  emitted as `total_cost` — a real charge, not an estimate.
 - The Codex pass writes token counters to its session rollout. Its model is in the
   catalog, so those counts are enough for Datadog to price it.
+- The pi pass records a usage object on every run (input/output/cacheRead/cacheWrite
+  plus `cost.total`, a price computed by pi from the model's OpenRouter list rates
+  pinned in scripts/pi/models.json — the same rates OpenRouter bills against, so
+  `cost.total` is the amount the pass is charged; verified against a live run
+  where 17,515 input tokens at $0.019/M priced exactly to what OpenRouter charges).
 - Claude Code's own `total_cost_usd` is deliberately ignored. It is computed against
   Anthropic's list prices, and these passes are billed by OpenRouter for a non-
-  Anthropic model, so it describes a bill nobody was sent.
+  Anthropic model, so it describes a bill nobody was sent. (The Claude Code harness
+  itself is no longer used for Stage 2, but the parser stays for historical files.)
 
 The workflow names models the way OpenRouter routes them ("openai/gpt-5.6-sol").
 Datadog's catalog keys on the bare model and its originating provider, so
@@ -30,6 +36,7 @@
 
 Usage: llm-usage.py claude <model-slug> [execution-file]
        llm-usage.py codex  <model-slug> [rollout-dir]
+       llm-usage.py pi     <model-slug> [event-stream-file]
 Prints a JSON object on stdout for the workflow's jq to splice into a span:
   {"model_name": ..., "model_provider": ..., "gateway": ..., "metrics": {...}}
 `metrics` carries only what is actually known; it is `{}` when nothing is.
@@ -62,6 +69,16 @@
     "cached_input_tokens": "cache_read_input_tokens",
 }
 
+# pi's normalized usage object, as written to its JSON event stream (--mode json),
+# mapped to Datadog's metric names. `cost.total` (computed by pi from the model's
+# OpenRouter list rates) is handled separately below.
+PI_USAGE_FIELDS = {
+    "input": "input_tokens",
+    "output": "output_tokens",
+    "cacheRead": "cache_read_input_tokens",
+    "cacheWrite": "cache_write_input_tokens",
+}
+
 
 def split_model(slug):
     """"openai/gpt-5.6-sol" -> ("gpt-5.6-sol", "openai"). A slug with no provider
@@ -189,6 +206,53 @@ def codex_usage(directory):
     return (collect(totals, CODEX_USAGE_FIELDS), openrouter_cost(totals))
 
 
+def pi_cost(usage):
+    """The amount pi computed for the pass from the model's OpenRouter list rates
+    (usage.cost.total), or None when it is absent, non-numeric, or zero. Zero is
+    treated as "nothing reported" — a free call or an absent rate table both
+    report no cost, matching the claude/codex conventions."""
+    if not isinstance(usage, dict):
+        return None
+    cost = usage.get("cost")
+    if not isinstance(cost, dict):
+        return None
+    total = read_number(cost.get("total"))
+    if total is None or total <= 0:
+        return None
+    return total
+
+
+def pi_usage(path):
+    """pi's JSON event stream (--mode json) -> (token counts, reported cost or None).
+    Every assistant message carries the cumulative usage of the turn so far; the last
+    one that finished cleanly is the whole pass. Events of interest are
+    message_end/turn_end (message under `message`) and agent_end (messages array)."""
+    candidates = []
+    for event in read_messages(path):
+        message = event.get("message")
+        if event.get("type") == "agent_end":
+            messages = event.get("messages")
+            if isinstance(messages, list) and messages:
+                message = messages[-1]
+        if not isinstance(message, dict) or message.get("role") != "assistant":
+            continue
+        if not isinstance(message.get("usage"), dict):
+            continue
+        candidates.append(message)
+    if not candidates:
+        return ({}, None)
+    # Prefer the last message that finished cleanly; fall back to the last one with
+    # any usage; last resort is the last assistant message overall.
+    chosen = next(
+        (m for m in reversed(candidates) if m.get("stopReason") == "stop"),
+        next(
+            (m for m in reversed(candidates) if sum(
+                read_number(m["usage"].get(k)) or 0 for k in ("input", "output")) > 0),
+            candidates[-1]))
+    usage = chosen.get("usage") or {}
+    return (collect(usage, PI_USAGE_FIELDS), pi_cost(usage))
+
+
 def build_span_fields(slug, counts, cost):
     """Pure assembly of the span fields the workflow splices in: a catalog-matching
     model identity, whichever token counts are known, and a reported cost when there
@@ -216,6 +280,8 @@ def main(argv):
         counts, cost = codex_usage(source or DEFAULT_ROLLOUT_DIR)
     elif pass_name == "claude":
         counts, cost = claude_usage(source)
+    elif pass_name == "pi":
+        counts, cost = pi_usage(source)
     else:
         counts, cost = ({}, None)
     print(json.dumps(build_span_fields(slug, counts, cost)))

scripts/pi/models.json

@@ -0,0 +1,38 @@
+{
+  "//": "pi config for the Stage 2 review pass (BiggiePockets/.github).",
+  "//": "",
+  "//": "pi resolves models from the config dir pointed at by PI_CODING_AGENT_DIR. This file",
+  "//": "declares the OpenRouter provider and the single pinned Stage 2 model with its",
+  "//": "OpenRouter list rates, so a FRESH CI runner (no cached models-store.json) resolves",
+  "//": "the model deterministically. The model is absent from the catalog pi ships and the",
+  "//": "live catalog refresh is a background fetch, not a startup step we can rely on.",
+  "//": "",
+  "//": "Rates: $/1M tokens as published by OpenRouter for deepseek/deepseek-v4.1-flash at",
+  "//": "the time of pinning (input 0.15, output 0.6, input cache read 0.003; no cache write",
+  "//": "rate). pi computes cost.total from these, so it is exactly what OpenRouter bills for",
+  "//": "standard usage. If you roll the model, update the id AND these rates together.",
+  "//": "",
+  "//": "The apiKey interpolation reads OPENROUTER_API_KEY from the workflow environment",
+  "//": "(the same org secret the Codex stage uses) — no interactive login or OAuth is",
+  "//": "involved. api/api identify the OpenAI-compatible endpoint pi uses.",
+  "providers": {
+    "openrouter": {
+      "baseUrl": "https://openrouter.ai/api/v1",
+      "apiKey": "$OPENROUTER_API_KEY",
+      "api": "openai-completions",
+      "models": [
+        {
+          "id": "deepseek/deepseek-v4.1-flash",
+          "name": "DeepSeek V4.1 Flash",
+          "reasoning": true,
+          "cost": {
+            "input": 0.15,
+            "output": 0.6,
+            "cacheRead": 0.003,
+            "cacheWrite": 0
+          }
+        }
+      ]
+    }
+  }
+}

scripts/resolve-prompts.sh

@@ -6,7 +6,7 @@
 # experiment arm gets. This script picks ONE arm per review — hash('<repo>:<pr>')
 # mod 100, below experiment_split_percent means the experiment arm — and emits only
 # that arm's prompt. The assigned arm both posts its summary and decides the review,
-# so arms are compared BETWEEN pull requests and each review costs one Claude pass.
+# so arms are compared BETWEEN pull requests and each review costs one Stage-2 pass.
 #
 # Shared rule blocks (privacy, migration-data, perf) are stored once in
 # prompts/_shared/ and injected into every prompt that references them via {{@path}}