context_report.run.runner.PROVIDERS is currently ("anthropic", "claude-cli", "openai-compatible") — the Anthropic API with a key, the local Claude Code CLI with its login,
and anything speaking the OpenAI-compatible shape. A manifest naming any other provider is
rejected with "no backend for judge provider".
Gemini is worth adding: Google's own API, not its OpenAI-compatible shim. The native shape
differs enough that the compatible path loses information, and a judge whose usage accounting is
approximate undermines the cost.* rows.
What to do: add the backend beside the existing ones. The contract is small — the Asker
protocol in src/context_report/efficacy/backends.py is essentially ask(prompt) -> str, with
optional last_usage and last_model. openai_backend.py and cli_backend.py are the two
worked examples.
The thing to get right: a report says how it was obtained and cannot overstate it. Whatever
you add must carry its provenance through to the statement, so a reader can see which backend
produced a row. A backend that makes its rows indistinguishable from another's defeats the point
of the format.
Testing: the existing tests inject a fake asker so no network call is ever made in CI. Follow
that pattern. Do not add a test that needs a live key — it will fail for every contributor who
does not have one, which is most of them.
Sized to an afternoon if you have credentials to try it against; the plumbing is small and the
care is in the provenance. Files: src/context_report/run/runner.py, judge.py,
efficacy/backends.py, and their tests.
context_report.run.runner.PROVIDERSis currently("anthropic", "claude-cli", "openai-compatible")— the Anthropic API with a key, the local Claude Code CLI with its login,and anything speaking the OpenAI-compatible shape. A manifest naming any other provider is
rejected with "no backend for judge provider".
Gemini is worth adding: Google's own API, not its OpenAI-compatible shim. The native shape
differs enough that the compatible path loses information, and a judge whose usage accounting is
approximate undermines the
cost.*rows.What to do: add the backend beside the existing ones. The contract is small — the
Askerprotocol in
src/context_report/efficacy/backends.pyis essentiallyask(prompt) -> str, withoptional
last_usageandlast_model.openai_backend.pyandcli_backend.pyare the twoworked examples.
The thing to get right: a report says how it was obtained and cannot overstate it. Whatever
you add must carry its provenance through to the statement, so a reader can see which backend
produced a row. A backend that makes its rows indistinguishable from another's defeats the point
of the format.
Testing: the existing tests inject a fake asker so no network call is ever made in CI. Follow
that pattern. Do not add a test that needs a live key — it will fail for every contributor who
does not have one, which is most of them.
Sized to an afternoon if you have credentials to try it against; the plumbing is small and the
care is in the provenance. Files:
src/context_report/run/runner.py,judge.py,efficacy/backends.py, and their tests.