Skip to content

feat: add LiteLLM as an AI provider - #149

Closed
prodmanpd wants to merge 1 commit into
control-theory:mainfrom
prodmanpd:feat/add-litellm-provider
Closed

prodmanpd wants to merge 1 commit into
control-theory:mainfrom
prodmanpd:feat/add-litellm-provider

Conversation

@prodmanpd

@prodmanpd prodmanpd commented Sep 8, 2026

Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a first class AI provider (--ai-provider litellm),
  • A LiteLLM proxy is OpenAI wire compatible, so this reuses the existing, tested OpenAIClient verbatim and only differs in its defaults and environment variables. One provider entry reaches 100+ backends (OpenAI, Anthropic, Gemini, AWS Bedrock, Vertex AI, Azure, Groq, etc.) through a single endpoint with unified auth, routing, cost tracking, and observability.
  • Additive only: no existing provider changes, no new dependency.

Changes

  • internal/ai/litellm.go: NewLiteLLMClient(model) reuses OpenAIClient (chat, /models discovery, validation, retry) with LiteLLM defaults: LITELLM_API_KEY (required), LITELLM_API_BASE (default http://localhost:4000/v1), ServiceName "LiteLLM".
  • internal/ai/factory.go: ProviderLiteLLM type, NewClient case, and ValidProviders().
  • cmd/gonzo/main.go: --ai-provider help text now lists litellm.
  • internal/ai/litellm_test.go: unit tests.

Tests

1. Unit tests (go test ./internal/ai/ -run 'LiteLLM|ValidProviders|NewClient_LiteLLM' -v):

--- PASS: TestNewLiteLLMClient_NoAPIKeyReturnsUnvalidated (0.00s)
--- PASS: TestNewLiteLLMClient_DefaultBaseURL (0.09s)
--- PASS: TestNewLiteLLMClient_CustomBaseURL (0.03s)
--- PASS: TestNewClient_LiteLLMProvider (0.00s)
--- PASS: TestValidProviders_IncludesLiteLLM (0.00s)
PASS
ok  	github.com/control-theory/gonzo/internal/ai

2. Build and vet: go build ./... and go vet ./internal/ai/ both clean. New files are gofmt clean.

3. Live end to end through the real provider (ai.NewClient(ProviderLiteLLM, model) then AnalyzeLog(...)) against a running LiteLLM proxy, on two different upstreams routed by the same provider:

[E2E] service=LiteLLM model=gpt-4.1-mini      validated=true  err=""
[E2E] analysis (first 200 chars): 1. **What happened:** ...

[E2E] service=LiteLLM model=gemini-2.5-flash  validated=true  err=""
[E2E] analysis (first 200 chars): Here's an analysis of the log message: ...

validated=true confirms /v1/models discovery through the proxy; the analysis output confirms /v1/chat/completions. Full chain: factory to OpenAIClient to the LiteLLM proxy to the upstream provider to a parsed response.

Risk / Compatibility

  • Additive; openai and claude-code behavior unchanged, ProviderAuto default unchanged.
  • No new dependency (reuses OpenAIClient).
  • Inert unless a user passes --ai-provider litellm.

Example usage

CLI:

# Run a LiteLLM proxy (e.g. litellm --config config.yaml --port 4000), then:
export LITELLM_API_KEY=sk-...        # your proxy key
export LITELLM_API_BASE=http://localhost:4000/v1   # optional, this is the default
gonzo --ai-provider litellm --ai-model gpt-4o-mini

Programmatic (the internal/ai API, the same client the CLI builds):

import "github.com/control-theory/gonzo/internal/ai"

// LITELLM_API_KEY / LITELLM_API_BASE are read from the environment; the
// client validates itself via the proxy's /models discovery on construction.
client, err := ai.NewClient(ai.ProviderLiteLLM, "gpt-4o-mini")
if err != nil {
    return err
}
summary, err := client.AnalyzeLog(logLine, "ERROR", timestamp, map[string]string{"service": "api"})

@lucasMesquitaBorges

Copy link
Copy Markdown
Collaborator

Hi @prodmanpd! Thank you for the PR!
This is a request for a functionality we already support.
LiteLLM can be used via OpenAPI env vars. Example:

OPENAI_API_KEY=sk-litellm-... OPENAI_API_BASE=http://localhost:4000/v1 gonzo --ai-model gpt-4o-mini

I'm closing the PR, however, feel free to submit other PRs!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants