model-router

Self-hosted AI gateway for Node. Unify every model behind one API, route by cost/quality, and never get locked to one provider.

Point your app at one OpenAI-compatible endpoint. model-router picks the model for each request, falls back automatically when a provider fails, and logs exactly what you spend — so you can adopt new models the day they launch without touching app code.

Why

Hardcoding model: "gpt-4o" locks you in, overpays for cheap tasks, and takes your feature down when the provider does. A router is one layer of indirection that fixes all three.

• One endpoint, every model — OpenAI-compatible, so switching is a one-line base_url change
• No lock-in — swap or add providers behind the same contract
• Automatic fallback — a provider outage or rate-limit escalates to the next model in the chain (streaming falls back on connect failure)
• Streaming — OpenAI-compatible SSE (stream: true), token deltas normalized across providers
• Cost visibility — every request logged with model, tokens, cost, and latency
• Response caching — exact-match cache in front of the chain; pluggable store (in-memory or Upstash)

Quickstart

npm install
cp .env.example .env   # fill in ROUTER_KEY + provider keys
npm run dev

Call it with any OpenAI client — just set model to a route name, not a provider model:

curl http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $ROUTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cheap",
    "messages": [{ "role": "user", "content": "Classify: is this spam?" }]
  }'

Stream tokens by adding "stream": true — you get OpenAI-style data: chunks ending in data: [DONE]:

curl -N http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $ROUTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "smart", "stream": true, "messages": [{ "role": "user", "content": "Write a haiku" }] }'

Testing

Unit tests mock every provider fetch, so the suite runs with no API keys and no network — safe for CI and contributors:

npm test            # run once
npm run test:watch

Covers route resolution, chain fallback + spend logging, the SSE parser, and each adapter's request/response mapping.

Caching

An exact-match cache sits in front of the fallback chain: identical non-streaming requests (same messages, params, and route chain) return the stored response without calling any provider, and the hit is logged to the ledger at costUsd: 0 so savings show up in your spend data.

CACHE_TTL=3600   # seconds; set 0 to disable

The store is pluggable via the CacheStore interface. It defaults to an in-memory cache (per instance). To share a cache across instances, set Upstash Redis REST credentials — no SDK, just fetch, so it works on every runtime:

UPSTASH_REDIS_REST_URL=https://...upstash.io
UPSTASH_REDIS_REST_TOKEN=...

Any other backend (Cloudflare KV, Redis, Postgres) drops in behind the same interface — implement get/set and inject it via your Platform. Streaming responses are not cached yet.

Routing

Routes are defined in config/routes.json. A route name maps to an ordered fallback chain — the first model is primary, the rest are tried in order if it fails:

{
  "cheap": ["gemini-3.1-flash-lite", "gpt-5.4-nano", "claude-haiku-4-5"],
  "smart": ["claude-opus-4-8", "gpt-5.6-sol", "gemini-3.5-flash"],
  "frontier": ["claude-fable-5", "gpt-5.5-pro"]
}

Callers pick a route via model, or pass metadata.use_case to steer routing without changing model.

Automatic routing

Set model (or metadata.use_case) to auto and the router picks the tier for you from cheap local signals — prompt length, code, and reasoning-oriented language — with no extra model call and no added latency. Simple prompts get the cheap chain, harder ones get smart:

curl http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $ROUTER_KEY" -H "Content-Type: application/json" \
  -d '{ "model": "auto", "messages": [{ "role": "user", "content": "refactor this function" }] }'

This is the first rung of predictive routing; an LLM classifier or a model trained on the ledger can replace the heuristic in src/classify.ts without changing the API.

Supported providers

Provider	Status
OpenAI	✅
Anthropic	✅
Gemini	✅

Adding a provider = one file implementing the ProviderAdapter interface in src/adapters/.

Runtimes — deploy anywhere, no vendor lock

The core is just a standard Web fetch handler (app.fetch). Any runtime that can serve one can run this — there is no vendor-specific code in the repo. Deploy it on bare metal, Docker, or any PaaS; put it behind whatever you want.

Runtime	Entry	Run
Node (default)	src/index.ts	npm run dev / npm start
Bun (native, faster)	src/bun.ts	bun run src/bun.ts
Bun (via Node entry)	src/index.ts	bun run src/index.ts — works unchanged

Node is the only runtime that needs a shim to serve a fetch handler (@hono/node-server), and that shim lives in exactly one file. Everything host-specific sits behind the Platform interface (env, loadRoutes, ledger), and the single default platform uses only process.env + the filesystem — APIs that exist on Node, Bun, and Deno alike.

Behavior is env-flag driven, not vendor-flag driven: e.g. ROUTES_JSON supplies routes inline instead of reading config/routes.json, for hosts without a mountable filesystem.

Want to run it on a host that lacks fs/process? Write one Platform implementation against src/platform/types.ts and add a three-line entry. The core doesn't change, and no such adapter ships here — so nobody inherits a dependency on a platform they don't use.

Architecture

              ┌───────────── shared core (runtime-agnostic) ─────────────┐
entry ──────► │  app → router → execute → adapters(openai|anthropic)     │
(node|worker) │                    │                                     │
              └────────────────────┼─────────────────────────────────────┘
                                   ▼
                            Platform (injected)
                  env · loadRoutes · ledger   ← default.ts (or your own)

• ingress / app (src/app.ts) — OpenAI-compat endpoint, virtual-key auth
• router (src/router.ts) — route name → ordered model chain
• execute (src/execute.ts) — walk chain, retry/fallback, record spend
• adapters (src/adapters/*) — translate canonical ⇄ provider schema
• platform (src/platform/*) — the one boundary to runtime-specific code

Roadmap

• Streaming (SSE) responses
• Exact-match response caching (in-memory / Upstash)
• Semantic caching
• Predictive routing — heuristic difficulty classifier (auto route)
• LLM classifier, cascade escalation, offline eval harness
• Postgres-backed keys, budgets, and spend attribution
• Gemini adapter
• Tool-calling normalization

License

MIT