Rebuild the eval engine on inspect_ai, keeping real Claude Code as the agent - #19

Open

rominf wants to merge 33 commits into

rominf:inspect-engine

Commits on Sep 11, 2026

Add inspect_ai engine behind --engine flag
rominf
committed
d564a66
Record what a graded run spends, and compare engines
rominf
committed
3f76a2f
Add the Claude Code verification leg
rominf
committed
fd52c3e
Count legacy model calls the same way in both commands
rominf
committed
ef3365f
Document the engine choice
rominf
committed
a97d447
Make the sandbox provider selectable, and stop it eating the skill's config
rominf
committed
ff499f7
Say in the report whether the run was isolated
rominf
committed
7b54f68
Check the model answers before starting containers
rominf
committed
991ac6d
Stop a broken sandbox looking like an idle agent
rominf
committed
4df2504
Give a containerised case a working directory
rominf
committed
6a3719d
Let the judge read the artifacts, and hear what the agent said
rominf
committed
1400d90
Stop truncation hiding the action a check turns on
rominf
committed
27660ae
Use neutral values in the header-parsing fixture
rominf
committed
98ca77f
Exercise the inspect engine in CI, on both sandboxes
rominf
committed
de64e59
Fix what CI found: platform-blind tests and a misplaced fixture root
rominf
committed
f7d5124
Do not pin a mock model to opus under CI
rominf
committed
00d1110
Stop the wiring run paying for a mock that cannot finish
rominf
committed
86283df
Say what to install when a sandbox provider will not resolve
rominf
committed
b987db7
Grade the answer the agent submitted, not the sentence introducing it
rominf
committed
24f476d
Document what podman actually needs
rominf
committed
f983879

Commits on Sep 14, 2026

Resolve a declared compose file beside machine.yml
rominf
committed
bb3a32b
Contain a skill that cannot be run to that skill
rominf
committed
b6fc197
Show the judge everything the agent said, not just the last thing
rominf
committed
8938682
Run the real CLI inside inspect's framework
rominf
committed
facfb2e
Let the benchmark compare any two engines
rominf
committed
ecf0825
Run Claude Code where the scorers look
rominf
committed
dbe49a1

Commits on Sep 15, 2026

Name the engine that was actually asked for
rominf
committed
00e5438

Commits on Sep 16, 2026

Send claude-cli to the engine it asked for
rominf
committed
8da6ee6
Launch the CLI on Windows, where it is not an executable
rominf
committed
22b214f
Give the CLI the skill it is supposed to be testing
rominf
committed
a7ada23

Commits on Sep 17, 2026

Let the shell probe survive a guest with no bash
rominf
committed
928414c
Drop the live sample buffer on Windows
rominf
committed
9015640
Make a timeout say where it got to
rominf
committed
3c3bd5e