Skip to content

[bench] models: upgrade to Sonnet 5 / Opus 4.8 #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dburks-svg merged 4 commits into from
Jul 16, 2026
Merged

[bench] models: upgrade to Sonnet 5 / Opus 4.8 #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7578627
[bench] models: upgrade to Sonnet 5 / Opus 4.8
dburks-svg Jul 16, 2026
2d34d9d
[bench] api: extract text blocks so Sonnet 5 thinking does not fail open
dburks-svg Jul 16, 2026
e0420b3
[bench] api: map OpenRouter slug for Opus 4.8 (Codex P1)
dburks-svg Jul 16, 2026
096ed3e
[bench] docs: note Claude Code CLI version floor for claude_code (Cod…
dburks-svg Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 29 additions & 19 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ governance pipeline. This is non-negotiable.
## Architecture

```
PreToolUse Hook -> Challenger (Sonnet 4.6) -> Defender (Sonnet 4.6) -> Oracle (Opus 4.7) -> Ledger
PreToolUse Hook -> Challenger (Sonnet) -> Defender (Sonnet) -> Oracle (Opus) -> Ledger
```

- Hook intercepts Write/Edit/MultiEdit tool calls
Expand All @@ -35,9 +35,9 @@ bench/
hooks/
pre-tool-use.py # Hook entry point
pipeline/
challenger.py # Adversarial analysis (Sonnet 4.6)
defender.py # Soundness argument (Sonnet 4.6)
oracle.py # Binding verdict (Opus 4.7)
challenger.py # Adversarial analysis (Sonnet)
defender.py # Soundness argument (Sonnet)
oracle.py # Binding verdict (Opus)
constitution.py # Load, snapshot, hash
runner.py # Sequential orchestration
ledger/
Expand All @@ -59,12 +59,21 @@ bench/

## Models

| Role | Model | Purpose |
|------------|------------|----------------------------------|
| Challenger | Sonnet 4.6 | Find problems in proposed change |
| Defender | Sonnet 4.6 | Argue soundness of the change |
| Oracle | Opus 4.7 | Issue binding PASS or VETO |
| Utility | Haiku 4.5 | Reserved for future summarization (utils/formatting.py is currently stdlib-only) |
| Role | Constant (in `utils/api.py`) | Purpose |
|------------|------------------------------|----------------------------------|
| Challenger | `CHALLENGER_MODEL` | Find problems in proposed change |
| Defender | `DEFENDER_MODEL` | Argue soundness of the change |
| Oracle | `ORACLE_MODEL` | Issue binding PASS or VETO |
| Utility | `UTILITY_MODEL` | Reserved for future summarization (utils/formatting.py is currently stdlib-only) |

`utils/api.py` is the single source of truth for model IDs. This document names
the constants rather than restating version strings, so a model change is a
single-file edit to `utils/api.py` and cannot drift from the docs. Each constant
holds the exact first-party Anthropic model ID. For current-generation models
these are bare aliases (for example `claude-sonnet-5`, `claude-opus-4-8`), which
are complete as-is; a dated suffix is used only for models that publish dated
snapshots (as with the reserved `UTILITY_MODEL`). Confirm each ID resolves on the
target provider(s) before shipping.

Models are Anthropic by default. The wrapper in utils/api.py also supports
OpenRouter as a routing backend (selected via the BENCH_PROVIDER env var) so
Expand Down Expand Up @@ -127,8 +136,9 @@ The constitution lives in bench.json. Current constraints:
## API Configuration

The LLM wrapper lives at `utils/api.py` and exposes a single
`call_model(model, system_prompt, user_content, max_tokens=4096) -> dict`
function. The provider is selected at call time by the `BENCH_PROVIDER`
`call_model(model, system_prompt, user_content, max_tokens=...) -> dict`
function (the `max_tokens` default is defined in `utils/api.py`, the single
source of truth). The provider is selected at call time by the `BENCH_PROVIDER`
environment variable; the function signature is identical across all backends.

```python
Expand All @@ -142,8 +152,9 @@ client = openai.OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
)
# Model strings are auto-prefixed with "anthropic/" on this path,
# e.g. "claude-sonnet-4-6" -> "anthropic/claude-sonnet-4-6".
# Model strings are routed to their OpenRouter slug on this path. Most map to
# "anthropic/<id>", but ids whose OpenRouter slug differs (e.g. the dotted
# "anthropic/claude-opus-4.8") are translated via a small map in utils/api.py.
# The openai SDK is a soft dependency — install it only if you set
# BENCH_PROVIDER=openrouter.

Expand All @@ -157,11 +168,10 @@ client = openai.OpenAI(
# adds no dependency. Per-stage timeout is BENCH_CLAUDE_TIMEOUT seconds
# (default 120).

# Model strings (same across all providers; routing handles any prefix)
CHALLENGER_MODEL = "claude-sonnet-4-6"
DEFENDER_MODEL = "claude-sonnet-4-6"
ORACLE_MODEL = "claude-opus-4-7"
UTILITY_MODEL = "claude-haiku-4-5-20251001"
# Model strings live in utils/api.py as CHALLENGER_MODEL, DEFENDER_MODEL,
# ORACLE_MODEL, and UTILITY_MODEL (the single source of truth). This section
# does not restate the literal IDs, so they cannot drift. Routing handles any
# prefix (the openrouter path prepends "anthropic/").
```

## Hook Response Format
Expand Down
24 changes: 18 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,12 +123,24 @@ Not all tool inputs are simple text edits. Bench handles three edge cases:

## Models

| Role | Model | Purpose |
|------------|------------|-----------------------------|
| Challenger | Sonnet 4.6 | Adversarial analysis |
| Defender | Sonnet 4.6 | Soundness argument |
| Oracle | Opus 4.7 | Binding verdict |
| Utility | Haiku 4.5 | Reserved for future summarization (formatting is currently stdlib-only) |
| Role | Constant (in `utils/api.py`) | Current model | Purpose |
|------------|------------------------------|------------------|-----------------------------|
| Challenger | `CHALLENGER_MODEL` | Claude Sonnet 5 | Adversarial analysis |
| Defender | `DEFENDER_MODEL` | Claude Sonnet 5 | Soundness argument |
| Oracle | `ORACLE_MODEL` | Claude Opus 4.8 | Binding verdict |
| Utility | `UTILITY_MODEL` | Claude Haiku 4.5 | Reserved for future summarization (formatting is currently stdlib-only) |

`utils/api.py` is the single source of truth for model IDs. The "Current model"
column is an illustrative snapshot for readers, not an authoritative record;
refresh it when you change a model. The exact IDs live in the constants named
above.

With `BENCH_PROVIDER=claude_code` (set by the repo's `.claude/settings.json`),
the local Claude Code CLI must be recent enough to recognize these IDs: Claude
Sonnet 5 needs Claude Code v2.1.197+ and Claude Opus 4.8 needs v2.1.154+ (per
Claude Code's model-config docs; run `claude update` to upgrade). An older CLI
fails the stage, and under the runner's fail-open policy that surfaces as a
flagged `pipeline_error` rather than a hard block, so keep the CLI current.

## Built With

Expand Down
Loading
Loading