feat(bench): OpenAI + Gemini tool-use adapters (#107)#122
Merged
Connorrmcd6 merged 1 commit intoJun 14, 2026
Conversation
Cross-provider adapters for the multi-turn generalization sweep, so the "capability doesn't buy rot-resistance" finding isn't Claude-only. Code only — live smokes come later; exact model ids / pricing / per-model params are flagged TODO at smoke time. - models.py: OpenAIModel.step (Chat Completions + function tools) and GeminiModel.step (google-genai generate_content + function calling), each behind the same ToolModel protocol, with lazy SDK imports + key guards mirroring AnthropicModel. Pure, module-level converters for both (_openai_*/_gemini_*), incl. Gemini's function_response name resolution (it keys by name, not id). build_model gains openai/gemini via a small provider->class map. - config.toml: [models.gpt] + [models.gemini] with placeholder ids/pricing (TODO at smoke). - pyproject.toml: [providers] optional extra (openai, google-genai) — not needed for the Anthropic-only or offline paths; converters stay import-free so the suite runs without the SDKs. - tests: 6 offline tests for the converters + response parsing using fake responses (no network). Suite: 30 passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cross-provider adapters for the multi-turn generalization sweep (#107), so "a more capable model is no more rot-resistant" doesn't rest on three Claude models. Code only — live smokes come later (as discussed); exact model ids, pricing, and any per-model param quirks are flagged
TODO at smoke time.What changed
models.py—OpenAIModel.step()(Chat Completions + function tools) andGeminiModel.step()(google-genaigenerate_content+ function calling), each behind the sameToolModelprotocol, with lazy SDK imports + key guards mirroringAnthropicModel(so nothing imports the SDKs unless that provider is actually used). Pure, module-level converters for both — including Gemini's function_response name resolution (Gemini keys tool results by function name, not id, so the converter maps id→name from the preceding call).build_modelgainsopenai/geminivia a small provider→class map.config.toml—[models.gpt]+[models.gemini]with placeholder ids/pricing.pyproject.toml+uv.lock— a[providers]optional extra (openai,google-genai); not needed for the Anthropic-only or offline paths.Verification
uv run pytest(inbench/) — 30 passed (6 new). Converters import and run with the provider SDKs not installed.cargo fmt --all --check— clean (no Rust touched).TODO.Scope
Implements the adapter half of #107. The cross-provider sweep + aggregation run is the other half of Connorrmcd6/surface-bench#16; verification/compounding metrics are Connorrmcd6/surface-bench#13.
🤖 Generated with Claude Code