[bench] models: upgrade to Sonnet 5 / Opus 4.8#11
Conversation
Upgrade the pipeline judges to current models and make future model changes a clean single-file edit. - Challenger/Defender: claude-sonnet-4-6 -> claude-sonnet-5 - Oracle: claude-opus-4-7 -> claude-opus-4-8 - Decouple CLAUDE.md and README from literal model IDs: docs now name the utils/api.py constants (README also shows a non-authoritative "current model" snapshot). This removes the code/doc duplication that made a model change an inherently multi-file edit and repeatedly tripped the pipeline's own divergence veto. - Raise call_model max_tokens default 4096 -> 8192: Sonnet 5 runs adaptive thinking by default, which shares the output budget with the JSON body. Guards against PARSE_FAILURE on the anthropic/openrouter paths (the claude_code CLI caps its own output, so it is unaffected). - Add tests/test_model_ids.py: structural checks on the constants plus an opt-in live resolution smoke test (BENCH_LIVE_SMOKE=1) so a bad model string cannot silently disable the pipeline. Every edit was governed by Bench's own pipeline (PASS) and recorded in the ledger, committed here alongside the code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7578627c75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| CHALLENGER_MODEL: str = "claude-sonnet-5" | ||
| DEFENDER_MODEL: str = "claude-sonnet-5" |
There was a problem hiding this comment.
With the default Anthropic provider, switching Challenger/Defender to Sonnet 5 makes these stages use a model whose default adaptive thinking can return a thinking block before the final text (Anthropic docs). _anthropic_call still reads only response.content[0].text, so any governed change that triggers thinking will parse an empty string, retry once, then surface PARSE_FAILURE; pipeline/runner.py turns that stage PIPELINE_ERROR into a fail-open PASS. Scan response.content for text blocks before enabling Sonnet 5 here.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch — fixed in 2d34d9d. _anthropic_call now selects type == "text" blocks with non-empty text (skipping thinking/tool_use), so Sonnet 5 adaptive-thinking blocks can no longer blank the reply into a PARSE_FAILURE that the runner fails open. Added regression tests for thinking-before-text, multi-text concat, and a non-text block carrying .text.
Codex P1 on PR #11. On the anthropic provider, Sonnet 5 runs adaptive thinking by default, so response.content[0] can be a thinking block. _anthropic_call read content[0].text, which is then empty, forcing a PARSE_FAILURE that pipeline/runner.py fails open into a PASS. That would silently disable Challenger/Defender adjudication on the anthropic path (the claude_code CLI path was unaffected, so tests missed it). Fix: scan response.content and concatenate blocks whose type == "text" with non-empty text, so thinking/tool_use/other block types cannot leak into or blank out the governed reply body. Add regression tests: thinking-block-before-text, multi-text concat, and a non-text block carrying a .text attribute (must be ignored). Update the one existing anthropic mock to set type="text" to match real block shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d34d9dfac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # provider before shipping a change. | ||
| CHALLENGER_MODEL: str = "claude-sonnet-5" | ||
| DEFENDER_MODEL: str = "claude-sonnet-5" | ||
| ORACLE_MODEL: str = "claude-opus-4-8" |
There was a problem hiding this comment.
When BENCH_PROVIDER=openrouter, _openrouter_call simply prefixes this constant, so the Oracle request is sent as anthropic/claude-opus-4-8; OpenRouter's published slug for Opus 4.8 is anthropic/claude-opus-4.8 (see https://openrouter.ai/anthropic/claude-opus-4.8). In that provider configuration the Oracle stage returns API_ERROR, and pipeline/runner.py fails open on Oracle pipeline errors, so every governed edit is allowed without a binding Oracle verdict. Please add a provider-specific mapping or use an OpenRouter-compatible alias before shipping this model change.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed against openrouter.ai/api/v1/models and fixed in e0420b3. Added an explicit _OPENROUTER_SLUGS map: claude-opus-4-8 -> anthropic/claude-opus-4.8 (dotted), claude-sonnet-5 -> anthropic/claude-sonnet-5, with a bare-prefix fallback for unlisted ids. Tests cover both mapped cases and the fallback. Note both of your P1s share a root cause: runner.py fails open to PASS on stage errors (a deliberate "broken governance must not block work" policy). Removing the triggers here; the fail-open policy itself is a separate design decision left to the maintainer.
Second Codex P1 on PR #11. _openrouter_call prefixed the first-party id as "anthropic/claude-opus-4-8", but OpenRouter's published slug is dotted: "anthropic/claude-opus-4.8" (verified against openrouter.ai/api/v1/models). On BENCH_PROVIDER=openrouter the Oracle would return API_ERROR, which pipeline/runner.py fails open into a PASS, so every governed edit would be allowed without a binding Oracle verdict. claude-sonnet-5 maps cleanly, so only Opus was affected; the active claude_code provider is unaffected. Fix: add an explicit _OPENROUTER_SLUGS map (verified slugs) consumed by _openrouter_call, with a bare-prefix fallback for unlisted ids. Tests cover the mapped Opus/Sonnet cases and the fallback. Update the CLAUDE.md OpenRouter routing note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0420b36f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| CHALLENGER_MODEL: str = "claude-sonnet-5" | ||
| DEFENDER_MODEL: str = "claude-sonnet-5" | ||
| ORACLE_MODEL: str = "claude-opus-4-8" |
There was a problem hiding this comment.
With BENCH_PROVIDER=claude_code (the checked-in .claude/settings.json default), _claude_cli_call passes these constants directly to claude -p --model ...; Claude Code's model-config docs state, “Sonnet 5 requires Claude Code v2.1.197 or later. Opus 4.8 requires v2.1.154 or later.” (https://code.claude.com/docs/en/model-config). On older installed CLIs, the new IDs make stages exit non-zero, become API_ERROR/PIPELINE_ERROR, and the runner fail-opens to PASS, so governance is bypassed until the CLI is upgraded; add a version check/fallback or enforce the minimum before switching the defaults.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Verified and documented. The version floors are accurate (Sonnet 5 needs Claude Code v2.1.197+, Opus 4.8 needs v2.1.154+, per code.claude.com/docs/en/model-config) and are now a documented deployment prerequisite in README (e0420b3 follow-up). Note the installed CLI already recognizes both IDs: the governance runs on this very PR executed through claude_code on Sonnet 5 / Opus 4.8 and returned coherent verdicts. I did not add a hardcoded version gate because it would not prevent the bypass (the error still fail-opens) and would add brittle version parsing; the real lever is the shared root cause of all three findings, the runner fail-open-to-PASS policy. That is a deliberate constitutional choice ("a broken governance layer must not block work"), so I have flagged it for the maintainer to decide rather than changing it autonomously.
…ex P2) Codex P2 on PR #11. On BENCH_PROVIDER=claude_code, an older Claude Code CLI that does not recognize claude-sonnet-5 / claude-opus-4-8 fails the stage, which the runner fails open into a PASS. Document the verified minimum CLI versions (Sonnet 5: v2.1.197+, Opus 4.8: v2.1.154+, per Claude Code's model-config docs) as a deployment prerequisite. No code gate: it would not prevent the fail-open bypass and version thresholds are brittle; the systemic fix is the runner's fail-open policy, a maintainer decision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Governance receipts appended after the PR #11 commits were staged (the model-upgrade session, including generating the work-report artifact). Committing them keeps the ledger continuous with the changes that produced them. No source changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Summary
CLAUDE.md/README.mdfrom literal model IDs so the docs name theutils/api.pyconstants. Future model changes become a single-file edit, which stops re-tripping the pipeline's own multi-file divergence veto.README.mdkeeps a clearly non-authoritative "Current model" snapshot column for readers.call_modelmax_tokensdefault 4096 -> 8192. Sonnet 5's default-on adaptive thinking shares the output budget with the JSON body; this guards against PARSE_FAILURE on the anthropic/openrouter paths (the claude_code CLI caps its own output).tests/test_model_ids.py: structural checks on the model constants plus an opt-in live resolution smoke test (BENCH_LIVE_SMOKE=1) so a bad model string cannot silently disable the pipeline.Every edit was governed by Bench's own pipeline (all PASS) and is recorded in the committed ledger.
Test plan
python -m unittest discover -s tests -v-> 324 pass, 1 skipped (the opt-in live smoke test).python -m cli verify-> ledger chain VALID.BENCH_LIVE_SMOKE=1 python -m unittest tests.test_model_ids(makes real API calls).🤖 Generated with Claude Code