context_report.run.runner.PROVIDERS is currently ("anthropic", "claude-cli", "openai-compatible"). A manifest naming any other provider is rejected with "no backend for judge
provider".
Vertex is worth adding: the other managed path to a hosted model, and the one GCP-resident
users are usually required to take. Same argument as Bedrock — a report format nobody can run
inside their own boundary is a report format they will not run.
What to do: add the backend beside the existing ones. The contract is small — the Asker
protocol in src/context_report/efficacy/backends.py is essentially ask(prompt) -> str, with
optional last_usage and last_model. openai_backend.py and cli_backend.py are the two
worked examples.
The thing to get right: a report says how it was obtained and cannot overstate it. Carry the
real model identity and usage accounting through to the statement rather than approximating it —
the cost.* rows are only worth reading if they came from the run they claim to describe.
Worth checking first: if the gemini backend (#61) lands before this, a good part of the
request shaping may already exist and this becomes mostly auth and endpoint work. Read that issue's
state before starting.
Testing: the existing tests inject a fake asker so no network call is ever made in CI. Follow
that pattern; do not add a test that needs live GCP credentials.
Sized to an afternoon if you have a project to try it against. Files:
src/context_report/run/runner.py, judge.py, efficacy/backends.py, and their tests.
context_report.run.runner.PROVIDERSis currently("anthropic", "claude-cli", "openai-compatible"). A manifest naming any other provider is rejected with "no backend for judgeprovider".
Vertex is worth adding: the other managed path to a hosted model, and the one GCP-resident
users are usually required to take. Same argument as Bedrock — a report format nobody can run
inside their own boundary is a report format they will not run.
What to do: add the backend beside the existing ones. The contract is small — the
Askerprotocol in
src/context_report/efficacy/backends.pyis essentiallyask(prompt) -> str, withoptional
last_usageandlast_model.openai_backend.pyandcli_backend.pyare the twoworked examples.
The thing to get right: a report says how it was obtained and cannot overstate it. Carry the
real model identity and usage accounting through to the statement rather than approximating it —
the
cost.*rows are only worth reading if they came from the run they claim to describe.Worth checking first: if the gemini backend (#61) lands before this, a good part of the
request shaping may already exist and this becomes mostly auth and endpoint work. Read that issue's
state before starting.
Testing: the existing tests inject a fake asker so no network call is ever made in CI. Follow
that pattern; do not add a test that needs live GCP credentials.
Sized to an afternoon if you have a project to try it against. Files:
src/context_report/run/runner.py,judge.py,efficacy/backends.py, and their tests.