Bring Your Own Model — Ollama, OpenAI, Anthropic, OpenCode engines

[spboyer]

Migrated from spboyer/waza#290

Summary

Allow users to run evals against local models (Ollama) and direct API providers (OpenAI, Anthropic) plus OpenCode — instead of only through the Copilot SDK.

Motivation

Users want to evaluate skills against models running locally for privacy, cost, or speed. Also enables testing against providers not available through Copilot SDK.

Approach

New engine implementations in internal/execution/ — all implement existing AgentEngine interface:

Engine	Provider	API
ollama	Ollama (local)	REST /api/chat
openai	OpenAI-compatible	Chat Completions API (covers OpenAI, Azure OpenAI, any compatible endpoint)
anthropic	Anthropic	Messages API
opencode	OpenCode	TBD — research integration surface

• Engine selection via eval.yaml engine: field or CLI flag --engine ollama
• Model selection via model: field (e.g., llama3.2, gpt-4o, claude-sonnet-4)
• Configuration: environment variables for API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY), Ollama base URL, OpenCode config
• Update schema: schemas/waza-config.schema.json to include new engine types
• Build all four engines in parallel — no sequential dependencies

Key Files

• internal/execution/ollama.go
• internal/execution/openai.go
• internal/execution/anthropic.go
• internal/execution/opencode.go
• internal/execution/engine.go — update engine factory/registry
• schemas/waza-config.schema.json — add engine types
• cmd/waza/cmd_run.go — wire up engine selection

Priority

P1 — Platform

Size

Large

[spboyer]

Originally by @richardpark-msft on 2026-02-23:

• opencode and claude code we could get by just supporting ACP. I've done a spike on that some azd work, and it'd be a fit here - it lets you talk to any agent. copilot CLI also supports it, but I think we'd stick with the SDK for max richness with the API.

• chat-completions compatible endpoints, like Azure Foundry, or *llama style runners, would be easy to support as literally just chat completions (ie, using the standard OpenAI SDK).

[spboyer]

Audit note: this is partially covered by current open PRs, but should stay open until the provider story is complete.

• feat: wire BYOK providers #240 adds a custom-provider lane for Copilot SDK-backed runs.
• feat: add codex executor config support #233 adds Codex executor support.
• Remaining gap: direct Ollama/OpenAI/Anthropic/OpenCode engine support is not fully implemented across the requested providers.

[spboyer]

Audit note: this is partially covered by open PRs #233 and #240. PR #233 adds Codex executor support, and PR #240 adds a custom-provider lane through the Copilot SDK executor. The remaining gap is full support for the requested provider matrix (Ollama, OpenAI, Anthropic, OpenCode) with clear configuration, docs, and tests.

[spboyer]

Follow-up plan from issue audit

Open PRs #233 and #240 cover part of BYOM/provider support: Codex executor support and a custom-provider lane through the Copilot SDK executor. The full provider matrix remains open.

Missing items

• Merge or supersede PR feat: add codex executor config support #233 for Codex executor support.
• Merge or supersede PR feat: wire BYOK providers #240 for custom provider configuration through the Copilot SDK lane.
• Define the supported provider matrix and config surface for Ollama, OpenAI, Anthropic, OpenCode, and Codex.
• Add provider-specific auth/config validation and clear error messages.
• Add docs and examples for local model and direct API provider eval runs.
• Add integration or mocked engine tests that verify model selection, transcript capture, tool telemetry, and result JSON behavior across providers.