Skip to content

Google Gemini adapter #34

Description

@MSKazemi

Problem

AOBench cannot call Google Gemini. Its function-calling declarations differ structurally
from OpenAI's, so the existing openai adapter (which reaches any OpenAI-compatible
endpoint) does not cover it. That means no Gemini results on the leaderboard.

Desired result

aobench run task --task JOB_USR_001 --env env_01 --adapter gemini:<model>

Read this first

If the LiteLLM adapter (#33) lands, this may reduce to a documentation page plus a regression
test.
Please check #33 before starting, and comment here so we do not do the work
twice.

Where to look

Path Why
src/aobench/adapters/openai_adapter.py The shape to mirror
src/aobench/adapters/base.py The contract: BaseAdapter.run(context) -> Trace
src/aobench/adapters/__init__.py Register it
pyproject.toml Optional extra, following openai/anthropic

Implementation hints

  • SDK: google-genai.
  • The real work is the tool-schema translation. Gemini's function declarations are
    shaped differently from OpenAI's; the ToolRegistry schemas have to be converted, and
    the tool results have to be converted back into the Trace shape.
  • Import the SDK inside the function; raise an actionable "install aobench[gemini]"
    error rather than letting an ImportError escape.
  • Populate token counts — the efficiency scorer and CLEAR cost axis depend on them.

Acceptance criteria

  • aobench run task --adapter gemini:<model> ... produces a scoreable Trace
  • Tool calls and results recorded in the same shape as the openai adapter
  • Token usage populated
  • Missing dependency and missing credentials each produce their own clear message
  • aobench list adapters includes it
  • docs/guides/evaluating-your-own-agent.md updated

Tests

tests/unit/ with a mocked client, following the existing OpenAI adapter tests.
No network calls in tests.

Difficulty

Roughly a day.

Getting started

Comment before starting. Questions welcome.

Activity

  1. added
    enhancementNew feature or request
    help wantedExtra attention is needed
    area: adaptersModel/agent adapters (OpenAI, LiteLLM, Gemini, Bedrock, MCP)
    on Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: adaptersModel/agent adapters (OpenAI, LiteLLM, Gemini, Bedrock, MCP)effort: mediumRoughly a dayenhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions