Interactive animated architecture flows for LLM-D distributed inference on Kubernetes
This site visualizes how llm-d (CNCF Sandbox) enables Kubernetes-native distributed LLM inference.
llm-d is a cloud-native distributed inference orchestration system for serving large language models in production on Kubernetes. It provides:
- llm-d Inference Scheduler (v0.7.1) — GPU topology-aware pod placement
- llm-d Workload Variant Autoscaler (v0.6.0) — Queue-based autoscaling for prefill/decode
llm-d integrates with:
- KServe (v0.17.0) — User-facing LLMInferenceService CRD
- LeaderWorkerSet — Kubernetes-native leader+worker pod orchestration (kubernetes-sigs)
- vLLM (v0.18.0) — High-performance inference engine with Ray distributed backend
- Istio + Gateway API — Service mesh and ingress routing
- Disaggregated Serving — Prefill/decode split with KV cache reuse
- Unified Serving — Single-pod inference with tensor parallelism
- Multi-Node Tensor Parallelism — Leader+worker pods via LeaderWorkerSet
Click any component in the visualizer to see:
- Role and responsibilities
- Configuration details
- Integration with other components
- Version information
Visual style matches the AI Gateway Flow Visualizers (RHAI 3.4 GA) with:
- GitHub-inspired dark mode
- Animated request flows
- Step-by-step inspector
- Clean, technical aesthetic
┌─────────────────────────────────────────────────────────┐
│ User API: LLMInferenceService CRD (KServe) │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Control Plane: KServe llmisvc-controller │
│ Translates to → LeaderWorkerSet + Service + HTTPRoute │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Orchestration: │
│ - llm-d Inference Scheduler (GPU topology aware) │
│ - llm-d Workload Variant Autoscaler (queue-based) │
│ - LeaderWorkerSet Controller (leader+worker pods) │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Service Mesh: Istio + Gateway API (HTTPRoute) │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Runtime: vLLM (v0.18.0) with Ray distributed backend │
└─────────────────────────────────────────────────────────┘
Client → Gateway → vLLM Leader Pod → (Ray) → Worker Pods → Aggregated Response
Client → Gateway → llm-d Router → Prefill Pod → KV Cache Storage
↓
Decode Pod → Response
This is a static HTML site with no build process:
# Option 1: Open directly
open index.html
# Option 2: Simple HTTP server
python3 -m http.server 8000
# Option 3: npm script
npm run dev# Install Vercel CLI
npm i -g vercel
# Deploy
vercel
# Deploy to production
vercel --prod- Push to GitHub
- Go to Settings → Pages
- Select branch
main - Site live at
https://your-username.github.io/repo-name
See DEPLOYMENT.md for more options (Netlify, Cloudflare Pages).
.
├── index.html # Landing page with cards
├── llmd-architecture.html # Interactive flow visualizer
├── llmd-flow.html # Original simple visualizer
├── README.md # This file
├── DEPLOYMENT.md # Deployment guide
├── CONTRIBUTING.md # Contribution guide
├── LICENSE # Apache 2.0
├── package.json # npm scripts
└── vercel.json # Vercel config
- llm-d: https://github.com/llm-d (CNCF Sandbox)
- KServe: https://kserve.github.io
- LeaderWorkerSet: https://lws.sigs.k8s.io
- vLLM: https://docs.vllm.ai
- OpenDataHub: https://opendatahub.io
Apache 2.0
Website: This repo
Docs: See llm-d GitHub
RHAI: Red Hat AI on Kubernetes (includes llm-d integration)