Skip to content

Switch Stage-2 review harness from Claude Code to pi - #56

Merged
bockets merged 1 commit into
mainfrom
switch-stage2-to-pi
Sep 10, 2026
Merged

Switch Stage-2 review harness from Claude Code to pi#56
bockets merged 1 commit into
mainfrom
switch-stage2-to-pi

Conversation

@bockets

@bockets bockets commented Sep 10, 2026

Copy link
Copy Markdown
Member

What & why

Stage 2 of the BiggiePockets review (verify & synthesize) ran on anthropics/claude-code-action. Claude Code translates OpenRouter's usage into its own schema on the way to the terminal result message, so the cost OpenRouter reported regularly did not survive to the Datadog span — that's the pricing data loss this PR fixes.

The stage now runs pi (@earendil-works/pi-coding-agent), a plain npm CLI: no GitHub app, no OIDC, no per-repo install. It's authenticated to OpenRouter with the same OPENROUTER_API_KEY the Codex stage uses (env var — nothing to configure).

Verified live before shipping:

  • pi resolves deepseek/deepseek-v4.1-flash in a fresh config dir (pinned via models.json — the model is absent from the catalog pi ships and the live catalog refresh is a background fetch).
  • Headless --mode json pass executes read/grep/glob/bash/write tools, writes verdict.json, and exits cleanly.
  • Usage is recorded on every pass: token counts + cost.total, computed from the model's OpenRouter list rates — the same rates OpenRouter bills against (confirmed: 17,515 input tokens x $0.019/M = $0.0003328, matching OpenRouter exactly). That's the cost that used to die in Claude Code.

What changed

  • Workflow (.github/workflows/biggiepockets-review.yml): claude-code-action step -> Install pi + Configure pi (copies scripts/pi/models.json into PI_CODING_AGENT_DIR) + one headless pi --mode json pass with the same read,grep,glob,bash,write allowlist and --no-context-files --no-skills --no-extensions for hermetic running. The step fails when verdict.json is missing (pi can exit 0 on a retried-but-failed session), so Re-run failed jobs still retries just this pass.
  • New scripts/pi/models.json: pins the OpenRouter provider + Stage-2 model and its OpenRouter list rates (fetched from OpenRouter's API at pin time: input $0.15/M, output $0.60/M, cache-read $0.003/M).
  • scripts/llm-usage.py: new pi pass reading the last finished assistant message's usage from pi-output.jsonl (input/output/cacheRead/cacheWrite -> input_tokens/etc., cost.total -> total_cost). Legacy claude/codex parsers kept.
  • scripts/review-diagnostics.py: parses pi's event stream (terminal message only — tool calls/args/results never published, secrets redacted) and attaches a redacted stderr tail.
  • Telemetry renamed with the harness: span claude.synthesize -> pi.synthesize, CLAUDE_MODEL -> PI_MODEL, tag claude_model -> pi_model. If you filter Datadog on the old span name, this is the change to know about.
  • README: "Install the Claude GitHub app" -> "Install nothing" (the workflow installs the npm package on the runner); workflow-permissions section drops the OIDC requirement (restrictive "Read repository contents" is now fine); cost-tracking section documents the pi pass and the pinned model config.
  • Tests: +9 (pi usage parsing, pi diagnostics, stderr redaction); 38 total, all passing. actionlint clean.

Things to know

  • Stage-2 model roll = edit PI_MODEL and scripts/pi/models.json together (id + rates).
  • Cost reporting now trusts pi's recorded usage rather than whatever survived Claude Code's schema translation — that's the fix.
  • One semantic note: cost.total comes from the pinned OpenRouter list rates. Wiring exact reported usage.cost through a pi provider extension can be added later if you ever want billed amounts for catalog-less models, but standard usage bills at list rates, so this matches.

Verification

  • python3 -m unittest discover -s tests -> 38 OK
  • actionlint .github/workflows/biggiepockets-review.yml -> clean
  • resolve-prompts.sh deterministic per PR (untouched logic, comment-only registry change)
  • Live pi runs on this machine against the org's OpenRouter key: exact command shape from the workflow, verdict written, usage+cost recorded.

Claude Code translated OpenRouter usage into its own schema on the way to
the terminal result message, so the cost OpenRouter reported regularly did
not survive to the Datadog span — the pricing loss this replaces.

The Stage-2 job now runs the pi CLI (@earendil-works/pi-coding-agent, an
npm package — no GitHub app, no OIDC) the same way Claude ran:

- pi is installed on the runner and runs one headless `--mode json` pass
  with the same read/grep/glob/bash/write tool allowlist. It writes its
  JSON event stream to pi-output.jsonl and the verdict.json the submit
  step consumes. Failing the pi step gates on verdict.json (pi can exit 0
  on a retried-but-failed session), so "Re-run failed jobs" still retries
  just this pass.
- scripts/pi/models.json pins the OpenRouter provider and the Stage-2
  model with its list rates. A fresh runner needs this: the model is
  absent from the catalog pi ships and the live catalog refresh is a
  background fetch. Pinned rates are OpenRouter's published ones, so
  pi's usage.cost.total is the amount the pass is charged (verified on a
  live run: 17,515 input tokens at $0.019/M).
- scripts/llm-usage.py gains a pi pass that reads the last finished
  assistant message's usage (tokens + cost.total). review-diagnostics.py
  parses pi's event stream (terminal message only — no tool transcripts)
  and attaches a redacted stderr tail.
- Telemetry is renamed with the harness: span codex.review -> pi.synthesize,
  env CLAUDE_MODEL -> PI_MODEL, tag claude_model -> pi_model. Legacy
  claude execution-file parsing stays in both scripts for historical files.
- README: install step becomes "install nothing" (npm package installed by
  the workflow), workflow-permissions section drops the OIDC requirement,
  cost-tracking section documents the pi pass and the pinned model config.

Verified live: pi resolves the pinned model in a fresh config dir, executes
read/write tools headlessly, writes verdict.json, and records usage on every
pass. 38 unit tests pass; actionlint clean.
@bockets
bockets merged commit 4b67a46 into main Sep 10, 2026
1 check passed
@bockets
bockets deleted the switch-stage2-to-pi branch September 10, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant