feat(bench): Anthropic tool-use adapter for the multi-turn loop (#102)#121
Merged
Conversation
Phase 4 (Anthropic only): make AnthropicModel a ToolModel so a real Claude model can drive the agent loop. OpenAI/Gemini adapters land separately (#107). - models.py: AnthropicModel.step() plus pure, module-level converters between the neutral loop format and the Anthropic wire format (_anthropic_tools, _anthropic_messages, _step_from_anthropic). The message converter coalesces consecutive same-role turns so a tool_result turn followed by the forced-answer nudge stays a single, valid user message. Assistant turns are echoed back via Step.provider_msg so history round-trips losslessly. - tests: 4 offline tests for the converters + response parsing (the riskiest part), using a fake response — no network in the suite. Live smoke (haiku, multi, cascade-quota-batcher, C0/C1/C2, N=1) confirms the round-trip and already shows the effect: C0 (no doc) reads the hidden window.py and is correct; C1 (stale doc) answers in one turn without verifying and is misled. The real signal awaits N>=10 + the oracle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 (Anthropic only): make
AnthropicModelaToolModelso a real Claude model can drive the multi-turn agent loop built in #119. This is what turns the offline harness into something that can actually measure verification suppression. OpenAI + Gemini adapters land separately (#107).Builds directly on merged #119 (phases 1–3).
What changed
models.py—AnthropicModel.step()plus three pure, module-level converters between the neutral loop format (agent.run_agent) and the Anthropic wire format:_anthropic_tools— neutralTOOL_SPECS→ Anthropicinput_schematools._anthropic_messages— neutral history → Anthropic messages, coalescing consecutive same-role turns so atool_resultuser-turn followed by the forced-answer nudge stays one valid user message. Assistant turns are echoed back viaStep.provider_msgso history round-trips losslessly._step_from_anthropic— response (text +tool_useblocks, usage, stop_reason) → neutralStep.Verification
uv run pytest(inbench/) — 24 passed (4 new), no network.cargo fmt --all --check— clean (no Rust touched).Live smoke (haiku ·
--mode multi·cascade-quota-batcher-code· C0/C1/C2 · N=1) confirms the real tool round-trip and already previews the headline:window.pyWith no doc the agent explores and checks the hidden dependency; with a confident stale doc it answers immediately and is misled. The real signal awaits N≥10 + the oracle tripwires (#105/#117).
Scope
Part of #102 (completes the loop so a real model can drive it). Not a full close: OpenAI/Gemini are Connorrmcd6/surface-bench#16 and the verification/compounding metrics + reporting are Connorrmcd6/surface-bench#13.
🤖 Generated with Claude Code