feat(evaluation): support OpenAI models in jobbench and deep-swe harnesses

[David Koleczek (DavidKoleczek)]

What

Both benchmark harnesses were Anthropic-only on the model-under-test path. Provider selection is now derived from --model at runtime, so an OpenAI model and claude-sonnet-5 can both be run from the same tree without editing code between runs.

Scope is the amplifier-agent, amplifier-foundation, and opencode-vanilla arms. The opencode-amplifier arm is untouched.

Why

Running a benchmark against a different provider previously required hand-editing four separate layers per harness, each of which fails differently when missed. Two of those failures are silent: a missing base_url falls back to the vendor's public endpoint via an SDK default, and a missing rate-card entry reports cost as unavailable rather than wrong. Deriving everything from one input removes that class of mistake.

How

Per harness:

• New providers.py maps a model id to a provider family and holds the per-family constants: credential env var names, provider module and source, opencode provider id and npm package.
• The amplifier-agent arm selects the provider module by family and pins base_url explicitly rather than relying on SDK environment fallback.
• The amplifier-foundation arm emits a family-appropriate settings.yaml. The Anthropic-only caching and long-context keys are dropped on the OpenAI path, where they are not consumed and would be silently inert.
• The opencode-vanilla arm uses the family's ai-sdk package and provider id, with the base URL in provider options. The Anthropic-specific /v1 normalisation no longer runs for OpenAI, whose base URL already carries it.
• Rate cards gain gpt-5.6-terra, using the published list rates that amplifier-module-provider-openai carries.

deep-swe additionally:

• agent_env() forwards only the selected family's credentials, rather than handing the container a key for a provider nobody asked to benchmark.
• network_allowlist() opens the selected family's API host. This one is load-bearing: deep-swe tasks run no-network behind an egress proxy, so without it every request is blocked before reaching the model, and the failure surfaces as a network error rather than a model error.
• The preflight credential check keys off the selected family, so an OpenAI run no longer requires an unused Anthropic key to be exported.

jobbench additionally:

• The trial launch profile passes OpenAI credentials through to the container.

Reasoning effort

Effort is pinned to high for all three arms on the OpenAI path, from a single REASONING_EFFORT constant per harness. The arms previously inherited different defaults from their respective stacks, which made reasoning-token volume incomparable between them. Changing the one constant re-pins all three together.

The Anthropic path is deliberately excluded: it uses a thinking-token budget rather than an effort level, and pinning it is a separate change.

Compatibility

The claude-sonnet-5 path is unchanged. Generated configuration, environment, allowlists, and run commands are byte-identical to before for all three arms, verified by diffing generated output against the pre-change tree.

Testing

• jobbench: 127 tests pass. deep-swe: 27 tests pass. ruff check and ruff format --check clean on both.
• Both harnesses exercised end to end against a live OpenAI endpoint, one task per benchmark across all three arms. Every arm completed, produced a graded result, and reported real cost figures rather than "not available".

Known follow-ups, not in this change

• jobbench forwards both families' credentials into every trial container via a static passthrough block, where deep-swe now selects per family. Rendering that block per trial would make the two harnesses consistent.
• The deep-swe allowlist appends the vendor's default API host even when a base URL override is set, which leaves the public endpoint reachable on a run intended for a proxy.
• Hand-declared opencode models default to a context limit of zero, which affects that arm's auto-compaction behaviour.