+
-### Helm Chart
+To connect the database, switch the image to `docker.litellm.ai/berriai/litellm-database` and add `DATABASE_URL` and `LITELLM_MASTER_KEY` to the Secret; nothing else in the manifest changes.
-:::info
+## Deploy with Terraform (AWS and GCP)
-[BETA] Helm Chart is BETA. If you run into an issues/have feedback please let us know [https://github.com/BerriAI/litellm/issues](https://github.com/BerriAI/litellm/issues)
+The official modules deploy the full microservices stack (network, database, Redis, object storage, secrets, compute, load balancer, and a migrations job that runs before the services start) and are published to the Terraform Registry:
-:::
-
-The canonical chart lives at [`deploy/charts/litellm-helm`](https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm) in the litellm repo and is published as an OCI artifact at `oci://ghcr.io/berriai/litellm-helm`, the published chart versions carry litellm release numbers (for example, `1.90.2`). This section covers a local quickstart; for a production install on EKS, GKE, or AKS with managed Postgres and Redis, see [Deploy to Cloud](./deploy_cloud.md#deploy-with-helm).
+- [`BerriAI/litellm/aws`](https://registry.terraform.io/modules/BerriAI/litellm/aws/latest)
+- [`BerriAI/litellm/google`](https://registry.terraform.io/modules/BerriAI/litellm/google/latest)
Full manifest (ConfigMap, Secret, Deployment, Service)
```yaml apiVersion: v1 @@ -192,12 +207,11 @@ metadata: name: litellm-config-file data: config.yaml: | - model_list: + model_list: - model_name: gpt-4o litellm_params: - model: azure/gpt-4o-ca - api_base: https://my-endpoint-canada-berri992.openai.azure.com/ - api_key: os.environ/CA_AZURE_OPENAI_API_KEY + model: openai/gpt-4o + api_key: os.environ/OPENAI_API_KEY --- apiVersion: v1 kind: Secret @@ -205,7 +219,7 @@ type: Opaque metadata: name: litellm-secrets data: - CA_AZURE_OPENAI_API_KEY: bWVvd19pbV9hX2NhdA== # your api key in base64 + OPENAI_API_KEY: bWVvd19pbV9hX2NhdA== # your api key in base64 --- apiVersion: apps/v1 kind: Deployment @@ -214,7 +228,7 @@ metadata: labels: app: litellm spec: - replicas: 1 + replicas: 2 selector: matchLabels: app: litellm @@ -269,502 +283,124 @@ spec: type: NodePort ``` -Port-forward to reach the proxy locally: - -```bash -kubectl port-forward service/litellm-service 4000:4000 -``` - -To connect a database (Virtual Keys, spend tracking), use the `docker.litellm.ai/berriai/litellm-database` image and add `DATABASE_URL` and `LITELLM_MASTER_KEY` to the Secret; nothing else in the manifest changes. See [Deploy with Database](#deploy-with-database). To run more than one instance behind a load balancer, see [Multi-region and scaling](./multi_region.md). - -:::info -To avoid issues with predictability, difficulties in rollback, and inconsistent environments, pin a version or SHA digest (for example, `litellm:main-v1.90.2` or `litellm@sha256:12345abcdef...`) instead of `litellm:latest`. -::: - +
+
+
**Option 2: Without UI (API-only deployment)**
If you don't need the admin UI, you can run with minimal configuration:
@@ -435,7 +354,7 @@ securityContext:
The proxy will log a warning about the UI but API endpoints will work normally.
-#### Environment Variables for Read-Only Filesystems
+Environment variables for read-only filesystems:
| Variable | Purpose | Default |
|----------|---------|---------|
@@ -445,23 +364,38 @@ The proxy will log a warning about the UI but API endpoints will work normally.
| `PRISMA_BINARY_CACHE_DIR` | Prisma binary cache | System default |
| `XDG_CACHE_HOME` | General cache directory | System default |
-#### Important Notes
+Notes: always set `LITELLM_MIGRATION_DIR` to a writable emptyDir path, and set `PRISMA_BINARY_CACHE_DIR` and `XDG_CACHE_HOME` to writable paths. If using a custom `server_root_path`, you must pre-process UI files in your Dockerfile as the proxy cannot modify files at runtime with a read-only filesystem. The UI is automatically detected as pre-restructured if it contains a `.litellm_ui_ready` marker file (created by the official Docker images).
-1. **Migrations**: Always set `LITELLM_MIGRATION_DIR` to a writable emptyDir path
-2. **Prisma Cache**: Set `PRISMA_BINARY_CACHE_DIR` and `XDG_CACHE_HOME` to writable paths
-3. **Server Root Path**: If using a custom `server_root_path`, you must pre-process UI files in your Dockerfile as the proxy cannot modify files at runtime with read-only filesystem
-4. **Automatic Detection**: The UI is automatically detected as pre-restructured if it contains a `.litellm_ui_ready` marker file (created by the official Docker images)
+## Verify production readiness
-## Extras
-### Expected Performance in Production
+### Expected performance
-See benchmarks [here](../benchmarks#performance-metrics)
+See benchmarks [here](../benchmarks#performance-metrics).
-### Verifying Debugging logs are off
+### Confirm debug logging is off
+
+You should only see the following level of details in logs on the proxy server:
-You should only see the following level of details in logs on the proxy server
```shell
# INFO: 192.168.2.205:11774 - "POST /chat/completions HTTP/1.1" 200 OK
# INFO: 192.168.2.205:34717 - "POST /chat/completions HTTP/1.1" 200 OK
# INFO: 192.168.2.205:29734 - "POST /chat/completions HTTP/1.1" 200 OK
```
+
+## Deployment FAQ
+
+**Q: Is Postgres the only supported database, or do you support other ones (like Mongo)?**
+
+A: We explored MySQL but that was hard to maintain and led to bugs for customers. Currently, PostgreSQL is our primary supported database for production deployments.
+
+Because LiteLLM talks to the database through Prisma over the PostgreSQL wire protocol, any Postgres-wire-compatible distributed SQL database works as a drop-in replacement. [YugabyteDB](https://www.yugabyte.com/) is used in production this way; point `DATABASE_URL` at its YSQL endpoint (`postgresql://Full Deployment manifest (initContainer, env, securityContext, volumes)
+ ```yaml apiVersion: apps/v1 kind: Deployment @@ -419,6 +336,8 @@ spec: sizeLimit: 64Mi ``` +}){kind=icon}
+ );
+}
+
+function Clients({ label = 'Clients (OpenAI SDK, LangChain, curl)' }: { label?: string }) {
+ return (
+
+
+ );
+}
+
+function Node({
+ icon,
+ title,
+ subtitle,
+ accent,
+ small,
+}: {
+ icon: string;
+ title: string;
+ subtitle?: string;
+ accent?: 'blue' | 'green';
+ small?: boolean;
+}) {
+ const cls = [
+ styles.node,
+ accent === 'blue' ? styles.nodeAccent : '',
+ accent === 'green' ? styles.nodeGreen : '',
+ small ? styles.nodeSmall : '',
+ ].join(' ');
+ return (
+
+
+
+ {label}
+
+
+ );
+}
+
+function StepNode({
+ n,
+ title,
+ subtitle,
+ accent,
+}: {
+ n?: number;
+ title: string;
+ subtitle?: string;
+ accent?: 'blue' | 'green';
+}) {
+ const cls = [
+ styles.node,
+ styles.nodeSmall,
+ accent === 'blue' ? styles.nodeAccent : '',
+ accent === 'green' ? styles.nodeGreen : '',
+ ].join(' ');
+ return (
+
+ {title}
+ {subtitle && {subtitle}}
+
+
+ {n !== undefined && {n}}
+
+ );
+}
+
+function PillsNode({
+ title,
+ subtitle,
+ pills,
+}: {
+ title: string;
+ subtitle?: string;
+ pills: string[];
+}) {
+ return (
+
+ {title}
+ {subtitle && {subtitle}}
+
+
+
+ );
+}
+
+function ConnectorDown({ label }: { label?: string }) {
+ if (!label) return ;
+ return (
+
+
+
+ {title}
+ {subtitle && {subtitle}}
+
+
+ {pills.map((p) => (
+
+ {p}
+
+ ))}
+
+
+ {label}
+
+
+ );
+}
+
+function Branch({ legs }: { legs: number }) {
+ const positions =
+ legs === 2 ? ['30%', '70%'] : legs === 3 ? ['20%', '50%', '80%'] : ['50%'];
+ return (
+
+
+
+ {positions.map((p) => (
+
+ ))}
+
+ );
+}
+
+function Box({
+ name,
+ badge,
+ badgeColor,
+ children,
+}: {
+ name: string;
+ badge?: string;
+ badgeColor?: 'blue' | 'green' | 'orange';
+ children: React.ReactNode;
+}) {
+ const badgeCls =
+ badgeColor === 'green'
+ ? styles.badgeGreen
+ : badgeColor === 'orange'
+ ? styles.badgeOrange
+ : styles.badgeBlue;
+ return (
+
+
+ );
+}
+
+/* ────────────────────── Life of a Request ────────────────────── */
+
+export function RequestFlowDiagram() {
+ return (
+
+ {name}
+ {badge && {badge}}
+
+ {children}
+
+
+ );
+}
+
+/* ────────────────────── Router: fallbacks and retries ────────────────────── */
+
+export function RouterFlowDiagram() {
+ return (
+
+
+
+
+
+
+
+
+ After the response is returned to the client, spend logging, rate-limit accounting, and
+ logging callbacks all run as asynchronous background tasks; no database write sits in
+ the request path.
+
+
+
+
+ );
+}
+
+/* ────────────────────── Image URL handling ────────────────────── */
+
+export function ImageFlowDiagram() {
+ return (
+
+
+
+
+ Retries stay inside the model group that failed; fallbacks leave it for the next group
+ in your fallbacks configuration.
+
+
+
+
+ );
+}
+
+/* ────────────────────── Tenancy hierarchy ────────────────────── */
+
+export function TenancyDiagram() {
+ return (
+
+
+
+
+
+
+
+
+
+
+ );
+}
diff --git a/src/components/CloudArchitecture/index.tsx b/src/components/CloudArchitecture/index.tsx
index 50c2c2dd1..76a4446ca 100644
--- a/src/components/CloudArchitecture/index.tsx
+++ b/src/components/CloudArchitecture/index.tsx
@@ -3,3 +3,9 @@ export {
CloudArchitectureSelector,
MultiRegionArchitecture,
} from './CloudArchitecture';
+export {
+ RequestFlowDiagram,
+ RouterFlowDiagram,
+ ImageFlowDiagram,
+ TenancyDiagram,
+} from './ConceptDiagrams';
diff --git a/src/components/CloudArchitecture/styles.module.css b/src/components/CloudArchitecture/styles.module.css
index 857672bba..f2b7c2a93 100644
--- a/src/components/CloudArchitecture/styles.module.css
+++ b/src/components/CloudArchitecture/styles.module.css
@@ -500,3 +500,17 @@
gap: 0.6rem;
}
}
+
+.stepChip {
+ width: 26px;
+ height: 26px;
+ border-radius: 50%;
+ background: var(--ca-accent);
+ color: #fff;
+ display: flex;
+ align-items: center;
+ justify-content: center;
+ font-size: 0.75rem;
+ font-weight: 700;
+ flex-shrink: 0;
+}
+
+
+
+
+
+
+
+
+
+
+
+ Every request's spend is attributed to its key, user, team, and organization at once,
+ and budgets are enforced at each level; a request is blocked when any level on its path
+ is over budget. Teams are the top-level boundary in open source; Organizations add the
+ outer layer and are an enterprise feature.
+
+
+