Problem
AOBench cannot call Google Gemini. Its function-calling declarations differ structurally
from OpenAI's, so the existing openai adapter (which reaches any OpenAI-compatible
endpoint) does not cover it. That means no Gemini results on the leaderboard.
Desired result
aobench run task --task JOB_USR_001 --env env_01 --adapter gemini:<model>
Read this first
If the LiteLLM adapter (#33) lands, this may reduce to a documentation page plus a regression
test. Please check #33 before starting, and comment here so we do not do the work
twice.
Where to look
| Path |
Why |
src/aobench/adapters/openai_adapter.py |
The shape to mirror |
src/aobench/adapters/base.py |
The contract: BaseAdapter.run(context) -> Trace |
src/aobench/adapters/__init__.py |
Register it |
pyproject.toml |
Optional extra, following openai/anthropic |
Implementation hints
- SDK:
google-genai.
- The real work is the tool-schema translation. Gemini's function declarations are
shaped differently from OpenAI's; the ToolRegistry schemas have to be converted, and
the tool results have to be converted back into the Trace shape.
- Import the SDK inside the function; raise an actionable "install
aobench[gemini]"
error rather than letting an ImportError escape.
- Populate token counts — the efficiency scorer and CLEAR cost axis depend on them.
Acceptance criteria
Tests
tests/unit/ with a mocked client, following the existing OpenAI adapter tests.
No network calls in tests.
Difficulty
Roughly a day.
Getting started
Comment before starting. Questions welcome.
Problem
AOBench cannot call Google Gemini. Its function-calling declarations differ structurally
from OpenAI's, so the existing
openaiadapter (which reaches any OpenAI-compatibleendpoint) does not cover it. That means no Gemini results on the leaderboard.
Desired result
Read this first
If the LiteLLM adapter (#33) lands, this may reduce to a documentation page plus a regression
test. Please check #33 before starting, and comment here so we do not do the work
twice.
Where to look
src/aobench/adapters/openai_adapter.pysrc/aobench/adapters/base.pyBaseAdapter.run(context) -> Tracesrc/aobench/adapters/__init__.pypyproject.tomlopenai/anthropicImplementation hints
google-genai.shaped differently from OpenAI's; the
ToolRegistryschemas have to be converted, andthe tool results have to be converted back into the
Traceshape.aobench[gemini]"error rather than letting an
ImportErrorescape.Acceptance criteria
aobench run task --adapter gemini:<model> ...produces a scoreableTraceopenaiadapteraobench list adaptersincludes itdocs/guides/evaluating-your-own-agent.mdupdatedTests
tests/unit/with a mocked client, following the existing OpenAI adapter tests.No network calls in tests.
Difficulty
Roughly a day.
Getting started
Comment before starting. Questions welcome.