LiteLLM adapter — one file, ~100 providers #33

Activity

1. MSKazemi

   added this to the v0.6 — Infrastructure milestone on Aug 10, 2026
2. MSKazemi

   added

   enhancementNew feature or requestNew feature or request

   help wantedExtra attention is neededExtra attention is needed

   effort: mediumRoughly a dayRoughly a day

   area: adaptersModel/agent adapters (OpenAI, LiteLLM, Gemini, Bedrock, MCP)Model/agent adapters (OpenAI, LiteLLM, Gemini, Bedrock, MCP)

   on Aug 10, 2026
3. hoti-code commented on Sep 1, 2026

   

   hoti-code

   on Sep 1, 2026

   More actions

   Hi, I’m taking this one. I’ll follow the existing adapter patterns and start with the scope described in the issue. I’ll open a PR once I have the adapter and tests working.
4. hoti-code commented on Sep 1, 2026

   

   hoti-code

   on Sep 1, 2026

   More actions

   Quick scope check before I start: I plan to use litellm:<model> and pass everything after litellm: directly to LiteLLM, including the provider prefix when needed—for example, litellm:anthropic/claude-sonnet-4-5. Does that match the intended format?
5. MSKazemi commented on Sep 8, 2026

   

   MSKazemi

   on Sep 8, 2026

   OwnerAuthor

   More actions

   Yes — that is exactly the right format, and thank you for checking before writing code
   rather than after.

   litellm:anthropic/claude-sonnet-4-5 will work as written. The dispatch in
   _build_adapter (src/aobench/cli/run_cmd.py) splits the prefix off the first colon
   only and hands the remainder to the factory untouched:

   prefix = name.split(":")[0]          # -> "litellm"
   factory = _REGISTRY.get(prefix)
   ...
   model = n.split(":", 1)[1]           # -> "anthropic/claude-sonnet-4-5"

   So a provider prefix containing /, and any further colons in the model tag, both pass
   through intact. Register "litellm": _make_litellm in _REGISTRY alongside the existing
   four and you inherit the CLI surface for free.

   Three things worth knowing before you start, none of them blocking:

   1. _REGISTRY is not the only list. aobench list adapters reads its own table in
      src/aobench/cli/list_cmd.py, and the "Unknown adapter" error message in run_cmd.py
      spells the known prefixes out by hand in two places. A new adapter that updates only
      _REGISTRY works but is invisible and is missing from the error text. Please update
      all of them — or, if you would rather fix the underlying duplication, that is welcome
      too.
   2. Keep the import lazy. Every other factory imports its adapter inside the
      function, so that someone without litellm installed can still run direct_qa.
      Please put it behind an optional extra in pyproject.toml and raise a clear message
      naming the extra when the import fails.
   3. No Anthropic key on my side. I can review the code and run the tests, but I cannot
      verify an anthropic/* route end-to-end. Please make the tests mock the LiteLLM call
      rather than requiring a live key, and mention in the PR which routes you actually
      exercised against a real endpoint — I will take your word for it and say so in the
      changelog.

   Also worth a look: openai_adapter.py already has an OpenAI-compatible base-URL path used
   for Ollama, so there is prior art for "one adapter, many providers" in the tree.

   The issue is yours — thank you for taking it. Shout if anything in the trace or scoring
   plumbing is unclear and I will answer quickly; a scope question like the one you just
   asked is never a bother.

   Optional, and no strings: if the project looks useful to you, a ⭐ helps it get found —
   and when the adapter lands, do post about it under your own name. "Added ~100-provider
   support to an HPC agent benchmark" reads well, and the credit should be yours.
6. MSKazemi commented on Sep 9, 2026

   

   MSKazemi

   on Sep 9, 2026

   OwnerAuthor

   More actions

   A brief follow-up on credit rather than on the code — no action needed from you.

   I audited every account that has ever interacted with this repository, looking for people
   whose contribution was real but who appeared on none of the project's credit surfaces. Two
   people fell through, and you were one of them.

   You are now on the contributor
   wall and in
   AUTHORS.md under Work in
   flight, credited specifically for the thing that is easy to overlook: you asked whether
   litellm:<model> with the provider prefix passed through was the intended format before
   writing any code. That is why this adapter will not need redoing in review, and it is a
   better contribution than most first patches — it just does not look like one, because
   nothing had to be fixed afterwards.

   Two things that follow from it, and both are commitments rather than requests:

   • Your claim on #33 is recorded and
     protected. It is in a live claims table on
     #20, and the stated policy on both the
     wall and that issue is now that nothing gets merged over a claim. That is written down
     because it was broken once — a contributor claimed an issue and a parallel PR landed on
     top of them four and a half hours later, and I never even replied. That will not happen
     to you.
   • There is no deadline. Take the time you need, and if you have changed your mind or
     hit something awkward, just say so — that frees it for someone else and costs you
     nothing. If you would rather not be listed on the wall at all, say the word and I will
     remove the entry.

   If any part of the trace plumbing, the token accounting or the scoring path is unclear, ask
   here and I will answer quickly — the scope question you asked earlier was a good use of my
   time, not an imposition, and so is the next one.

   One optional aside and I will not raise it again: when the adapter lands, your wall entry
   and the changelog line will both be public and linkable, and "added ~100-provider support
   to an open HPC agent benchmark" reads well under your own name. Link your own pull request
   rather than the repo — the credit is yours.