Switch Stage-2 review harness from Claude Code to pi - #56
Merged
Conversation
Claude Code translated OpenRouter usage into its own schema on the way to the terminal result message, so the cost OpenRouter reported regularly did not survive to the Datadog span — the pricing loss this replaces. The Stage-2 job now runs the pi CLI (@earendil-works/pi-coding-agent, an npm package — no GitHub app, no OIDC) the same way Claude ran: - pi is installed on the runner and runs one headless `--mode json` pass with the same read/grep/glob/bash/write tool allowlist. It writes its JSON event stream to pi-output.jsonl and the verdict.json the submit step consumes. Failing the pi step gates on verdict.json (pi can exit 0 on a retried-but-failed session), so "Re-run failed jobs" still retries just this pass. - scripts/pi/models.json pins the OpenRouter provider and the Stage-2 model with its list rates. A fresh runner needs this: the model is absent from the catalog pi ships and the live catalog refresh is a background fetch. Pinned rates are OpenRouter's published ones, so pi's usage.cost.total is the amount the pass is charged (verified on a live run: 17,515 input tokens at $0.019/M). - scripts/llm-usage.py gains a pi pass that reads the last finished assistant message's usage (tokens + cost.total). review-diagnostics.py parses pi's event stream (terminal message only — no tool transcripts) and attaches a redacted stderr tail. - Telemetry is renamed with the harness: span codex.review -> pi.synthesize, env CLAUDE_MODEL -> PI_MODEL, tag claude_model -> pi_model. Legacy claude execution-file parsing stays in both scripts for historical files. - README: install step becomes "install nothing" (npm package installed by the workflow), workflow-permissions section drops the OIDC requirement, cost-tracking section documents the pi pass and the pinned model config. Verified live: pi resolves the pinned model in a fresh config dir, executes read/write tools headlessly, writes verdict.json, and records usage on every pass. 38 unit tests pass; actionlint clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Stage 2 of the BiggiePockets review (verify & synthesize) ran on
anthropics/claude-code-action. Claude Code translates OpenRouter's usage into its own schema on the way to the terminal result message, so the cost OpenRouter reported regularly did not survive to the Datadog span — that's the pricing data loss this PR fixes.The stage now runs pi (
@earendil-works/pi-coding-agent), a plain npm CLI: no GitHub app, no OIDC, no per-repo install. It's authenticated to OpenRouter with the sameOPENROUTER_API_KEYthe Codex stage uses (env var — nothing to configure).Verified live before shipping:
deepseek/deepseek-v4.1-flashin a fresh config dir (pinned viamodels.json— the model is absent from the catalog pi ships and the live catalog refresh is a background fetch).--mode jsonpass executes read/grep/glob/bash/write tools, writesverdict.json, and exits cleanly.cost.total, computed from the model's OpenRouter list rates — the same rates OpenRouter bills against (confirmed: 17,515 input tokens x $0.019/M = $0.0003328, matching OpenRouter exactly). That's the cost that used to die in Claude Code.What changed
.github/workflows/biggiepockets-review.yml):claude-code-actionstep ->Install pi+Configure pi(copiesscripts/pi/models.jsonintoPI_CODING_AGENT_DIR) + one headlesspi --mode jsonpass with the sameread,grep,glob,bash,writeallowlist and--no-context-files --no-skills --no-extensionsfor hermetic running. The step fails whenverdict.jsonis missing (pi can exit 0 on a retried-but-failed session), so Re-run failed jobs still retries just this pass.scripts/pi/models.json: pins the OpenRouter provider + Stage-2 model and its OpenRouter list rates (fetched from OpenRouter's API at pin time: input $0.15/M, output $0.60/M, cache-read $0.003/M).scripts/llm-usage.py: newpipass reading the last finished assistant message's usage frompi-output.jsonl(input/output/cacheRead/cacheWrite ->input_tokens/etc.,cost.total->total_cost). Legacyclaude/codexparsers kept.scripts/review-diagnostics.py: parses pi's event stream (terminal message only — tool calls/args/results never published, secrets redacted) and attaches a redacted stderr tail.claude.synthesize->pi.synthesize,CLAUDE_MODEL->PI_MODEL, tagclaude_model->pi_model. If you filter Datadog on the old span name, this is the change to know about.actionlintclean.Things to know
PI_MODELandscripts/pi/models.jsontogether (id + rates).cost.totalcomes from the pinned OpenRouter list rates. Wiring exact reportedusage.costthrough a pi provider extension can be added later if you ever want billed amounts for catalog-less models, but standard usage bills at list rates, so this matches.Verification
python3 -m unittest discover -s tests-> 38 OKactionlint .github/workflows/biggiepockets-review.yml-> cleanresolve-prompts.shdeterministic per PR (untouched logic, comment-only registry change)