Skip to content

feat: add LiteLLM as AI gateway provider#18

Open
RheagalFire wants to merge 1 commit into
nduckmink:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#18
RheagalFire wants to merge 1 commit into
nduckmink:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

Summary

Adds LiteLLM as an AI provider for embedding, LLM, and vision capabilities. Users select litellm as their provider and use provider-prefixed model IDs (e.g. anthropic/claude-sonnet-4-6) to access 100+ LLM providers through a unified Python SDK. drop_params=True is always passed so provider-unsupported kwargs are silently dropped.

Follows the existing provider pattern exactly: LiteLLMEmbedding, LiteLLMChat, and LiteLLMVision extend the abstract base classes, registered in the registry dispatch functions.

Changes

  • app/ai/providers/litellm_provider.py - LiteLLMEmbedding, LiteLLMChat, LiteLLMVision (all three capabilities)
  • app/ai/providers/base.py - added LITELLM = "litellm" to ProviderType enum
  • app/ai/registry.py - registered litellm in _get_embedding_class, _get_llm_class, _get_vision_class + added label
  • pyproject.toml - added litellm>=1.80.0,<2.0 under [project.optional-dependencies].litellm

Verification

Live E2E against Azure Foundry (Claude via LiteLLM SDK):

$ python3 -c "
config = ProviderConfig(
    provider=ProviderType.LITELLM,
    api_key=os.environ['ANTHROPIC_FOUNDRY_API_KEY'],
    model_id='anthropic/claude-sonnet-4-6',
    base_url=os.environ['ANTHROPIC_FOUNDRY_BASE_URL'],
)
llm = LiteLLMChat(config)
result = await llm.generate('Say OK and nothing else.', temperature=0)
"

content: 'OK'
test_connection: ok=True, msg=OK — model=anthropic/claude-sonnet-4-6, response='OK'

Lint clean:

$ ruff check app/ai/providers/litellm_provider.py app/ai/providers/base.py app/ai/registry.py
All checks passed!

Usage

from app.ai.providers.base import ProviderConfig, ProviderType
from app.ai.providers.litellm_provider import LiteLLMChat

config = ProviderConfig(
    provider=ProviderType.LITELLM,
    api_key="sk-ant-...",                    # or leave empty for env-var auth
    model_id="anthropic/claude-sonnet-4-6",  # any litellm model ID
    base_url=None,                           # optional: litellm proxy URL
)

llm = LiteLLMChat(config)
result = await llm.generate("Summarize this document...")

# Tool calling works too:
turn = await llm.generate_with_tools(messages, tools)

Notes

  • Additive only. No existing providers modified.
  • litellm is an optional extra - lazy-imported inside function bodies.
  • generate_with_tools is implemented with full tool-call parsing, matching the OpenAI provider's AssistantTurn output format.
  • Vision provider includes retry logic (3 attempts with backoff), matching the OpenAI vision pattern.
  • No catalog entries added - the maintainer can add specific litellm model specs as needed.

@RheagalFire

Copy link
Copy Markdown
Author

cc @nduckmink

@nduckmink

Copy link
Copy Markdown
Owner

Thank you for your PR, I will review it soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants