📟 LLM inference is fundamentally different. Standard backend logic won't cut it.

---

Real production workloads require:
➢ Multiple endpoints
➢ Multiple model families
➢ Parallel async inference
➢ High availability
➢ Load balancing
➢ Fault tolerance
➢ Efficient GPU memory sharing
➢ Caching

That's where Tunnel Engine comes in. It provides a single endpoint link to access multiple LLM models. By simply changing the model name, we can easily maintain all model providers.

vLLM enables efficient multi-model serving with continuous batching, PagedAttention, precise GPU memory utilization, KV-cache, and separate URL endpoints per instance.

LMCache acts as the extender for vLLM. To store the KV-cache precisely, we need LMCache. It takes the KV-cache from vLLM and saves it to cheaper memory (local RAM or storage, instead of GPU memory only) so it can be reused later. LMCache also supports distributed cache synchronization, allowing multiple vLLM nodes to share caches.

LiteLLM is used for intelligent load balancing. While vLLM runs on multiple separate ports (e.g., ports 8000, 8001, 8002), LiteLLM wraps all of these ports into a single endpoint URL (e.g., port 4000) for all our microservices to call. Additionally, if a model in a specific service crashes (e.g., Out of Memory), we can set up alternative fallback models to handle the requests seamlessly without pausing the service.

Model weights don't have to live on local disk. With the Run:ai Model Streamer, an instance can stream its weights straight from any S3-compatible bucket (Wasabi, Cloudflare R2, Backblaze B2, MinIO) into VRAM at startup — no HF download, no disk copy (a 9.3 GB model streams in ~15s). Set model: s3://bucket/path + load_format: runai_streamer in configs/models.yaml and the S3 credentials in .env; switching storage providers is an .env-only change. See docs/s3-llm.md for the provider evaluation and setup guide.

Architecture (what we need):

   Our services        ┌───────────────────────────┐
   call this     ────▶ │   Apache APISIX Gateway   │  ← Edge TLS, Global API Key Auth, WAF
                       └─────────────┬─────────────┘
                                     │ (Internal VPC)
                                     ▼
                        ┌───────────────────────────┐  
   Rate limiting ────▶  │    LiteLLM Proxy :4000    │ ──(Tracks Latency/Token Counts)  ────▶  [ Prometheus ]
                        │ (Routing / Load-Balancer) │                                                ▲
                        └─────────────┬─────────────┘                                                │
                        ▲             │(Calls Hook)                                                  │
      (Checks/Sanitizes)│   ┌─────────▼─────────┐                                                    │
                        └───│  XGuard :8002     │  ← In-proxy content safety (internal, ~49ms)       │
                            │  guard_hook       │    pre-call always; post-call if check_output      │
                            └─────────┬─────────┘                                                    │
                                      │ (Validated request)                                          │
                        ┌─────────────▼─────────────┐                                                │
                        │  Kubernetes Service Mesh  │  ← K8s Load Balancer    | [Optional rn]        │
                        └─────────────┬─────────────┘                                                │
┌─────────────────────┐               │                                                              │
│  S3 Model Storage   │               │                                                              │
│ (Wasabi/R2/B2/MinIO)│               │                                                              │
└───▲─────────────────┘               │                                                              │
    | runai_streamer┌─────────────────▼─────────────────┐                                            │
    |               │                                   │                                            │
    |  ┌────────────▼──────────┐             ┌──────────▼────────────┐                               │
    |  │   vLLM Pod 1 :8000    │             │   vLLM Pod 2 :8001    │ ← Autoscaled via KEDA         │
    └─ │    (Model A v2)       │             │     (Model A v2)      │ ──(vllm:num_requests_waiting)─┘
       └────────────┬──────────┘             └──────────┬────────────┘
                    │                                   │
                    └─────────────────┬─────────────────┘
                                      ▼
                    ┌───────────────────────────────────┐
                    │      LMCache + Distributed Redis  │  ← Ultra-low TTFT across pods
                    └───────────────────────────────────┘

Running models manual via vLLM:

# Instance 1: Qwen 0.8B 
vllm serve Qwen/Qwen3.5-0.8B \
  --port 8000 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.35 \
  --max-model-len 65536

# Instance 2: MiniCPM 1B 
vllm serve openbmb/MiniCPM5-1B \
  --port 8001 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.45 \
  --max-model-len 65536

Quick Start

The whole workflow is driven by make. Run make help any time to list every target. New to the codebase? Read docs/PLAYBOOK.md - the fast-onboarding guide covering the repo layout, the multi-tenancy features, common workflows, and known gotchas.

1. One-time setup

make install                 # install all dependencies

cp .env.example .env         # then edit .env and set at least:
                             #   HF_TOKEN=hf_...           (for gated models)
                             #   LITELLM_MASTER_KEY=...    (the gateway ADMIN key)
                             #   PG_PASSWORD + DATABASE_URL (virtual keys / spend
                             #   tracking; see .env.example comments)

Secrets live only in .env. They are never written into any file under configs/ (the registry references the key via os.environ/LITELLM_MASTER_KEY, resolved at boot).

2. Configure your models

configs/models.yaml is the single source of truth. Add or swap a model there, then:

make check       # validate the registry: YAML parses + GPU budget not exceeded (writes nothing)
make generate    # rebuild derived configs (configs/litellm/ + configs/prometheus/)
make list        # show registered instances, their ports, and the proxy port

Always run make check first after editing models.yaml - it catches YAML mistakes and over-budget GPU splits before you ever launch a model.

3. Run the engine (pick ONE style)

Option A - single model, foreground (dev / benchmarking):

# terminal 1: load one model (stays in foreground, streams its logs)
make serve ID=<instance-id>     # use an id from `make list`

# terminal 2: once the model is loaded, health-gate and start the proxy on :4000
make start

Option B - whole fleet, background (one command):

make up          # launches every instance in the background, waits until all are
                 # healthy, then starts the proxy. Use `make down` to stop it all.

4. Verify and test

make health      # poll every instance + show GPU memory usage
make test        # full test suite

5. Stop and clean everything

make down         # stops ALL instances (started via `serve` OR `up`) and the proxy,
                  # then frees the GPU. Safe to run any time before switching models.

make down sweeps both tracked background instances and untracked foreground make serve processes, plus the LiteLLM proxy - so nothing is left holding the GPU or a port.

6. Retire ONE model while others keep serving

To stop a single model without touching the rest of the fleet or the gateway - for example, decommissioning one model the next day while another is still serving live production traffic:

make stop ID=<instance-id>       # stops just that instance, frees its GPU slice

This is safe with zero downtime for the surviving models: every instance is a separate process on its own port, and the LiteLLM proxy is a separate process, so stopping one never interrupts the others. The proxy keeps running and the remaining models keep answering.

Note

The proxy still lists the stopped model in /v1/models, so calls routed to it will error until you bring it back (make serve ID=<id>). To also stop the gateway from advertising it, remove its block from configs/models.yaml, run make generate, and restart the proxy during a maintenance window (a proxy restart briefly interrupts the surviving models too).

Golden rules (avoid conflicts)

1. Edit only configs/models.yaml, then make generate. Never hand-edit files under configs/litellm/ or configs/prometheus/ - they are auto-generated and get overwritten.
2. Always make down before swapping models or re-running. A leftover vLLM keeps the GPU memory and the port bound; the next launch will fail with an out-of-memory or port-in-use error. When unsure, confirm the GPU is clear with nvidia-smi (used memory should read 0 MiB).
3. Every instance needs a unique id and port. The registry rejects duplicate ids, duplicate ports, and any port that collides with the proxy - make check surfaces these instantly.
4. Do not mix make serve and make up for the same model. serve is foreground (one terminal per model); up is background (the whole fleet). Running both binds the same port twice.

Command reference

Command	What it does
make install	Install all dependencies
make check	Validate the registry without writing anything
make generate	Rebuild derived LiteLLM + Prometheus configs
make list	List registered instances and ports
make serve ID=<id>	Launch one instance in the foreground
make up	Launch the whole fleet in the background + proxy
make start	Health-gate running instances, then start the proxy
make health	Poll instance health + GPU memory
make stop ID=<id>	Stop ONE instance, leave the others and the proxy running
make down	Stop every instance and the proxy, free the GPU
make db-up / make db-down	Start/stop Postgres for virtual keys + spend logs
make keys-sync	Reconcile LiteLLM virtual keys with registry services
make keys-list	Per-service key status + spend
make obs-up / make obs-down	Start/stop Prometheus (:9092) + Grafana (:3000)
make loadtest	Open-loop load generator (RATE/DURATION/MIX/TIER_MIX env vars)
make loadtest-plots	Render analysis PNGs from loadgen results
make test	Run the full test suite
make view-models	List locally cached HuggingFace models

Virtual keys, tiers, and rate limiting

Every consumer declared under services: in configs/models.yaml gets its own LiteLLM virtual key with a model allowlist, tier rate limits (rpm/tpm), an optional budget, and a scheduling priority (enterprise requests preempt lower tiers at the vLLM engine under load). Spend is tracked per service at the cost: rates in the registry, so per-service costs are directly comparable.

make db-up       # start Postgres (Docker, :5433) - stores keys + spend logs
make keys-sync   # create/reconcile one virtual key per registry service
make keys-list   # per-service key status, tier, allowed models, and spend

Calling Tunnel Engine from your service

Tunnel Engine exposes one OpenAI-compatible endpoint for every model. Your service always points at the same URL and key; to use a different model you change only the model field to a registered instance id. No new endpoint per model, no client code changes - that is the whole point of the gateway.

What your client needs	Value
LLM endpoint (base_url)	http://<tunnel-host>:4000/v1 (e.g. http://localhost:4000/v1)
LLM API key	your service's virtual key from .tunnel/keys.env (SVC_<ID>=sk-...)
Model provider	openai - the gateway speaks the OpenAI API
LLM model	a registered instance id, e.g. lfm2.5-8b-a1b - swap this to switch models

Discover the available model ids with make list, or GET /v1/models on the gateway.

curl

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $SVC_MY_SERVICE" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lfm2.5-8b-a1b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",   # the ONE endpoint
    api_key="sk-...",                       # your service's virtual key
)

resp = client.chat.completions.create(
    model="lfm2.5-8b-a1b",                  # swap this id to use another model
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

To route to a different model, change only model="..." to another id from make list. The URL, key, and code stay identical. Tool calling works through the standard OpenAI tools parameter for any instance that has a tool_parser set in models.yaml.

Framework config (e.g. a tool-bench service)

If your service takes provider settings, map them like this:

provider   = openai
base_url    = http://<tunnel-host>:4000/v1
api_key     = <your service's SVC_* key from .tunnel/keys.env>
model       = <instance-id>   # e.g. lfm2.5-8b-a1b, swap per call

So yes: one endpoint, one key, and you swap models by name.