RAG Pipeline — Qdrant + vLLM + Embeddings

Full Retrieval-Augmented Generation stack on Convox: ingest documents, embed them, store in Qdrant, query with Llama 3.1 8B — all in your own cloud with source citations.

Architecture: 4 services (API gateway + vLLM GPU + embedding CPU + Qdrant vector DB) GPU: 1x A10G or L4 for LLM · Cold start: 4-8 min (full stack)

Prerequisites

• Convox rack v3.24.6+ with GPU-capable nodes (g5.xlarge recommended)
• A HuggingFace account that has accepted the Llama 3.1 license
• A HuggingFace access token

Five-Minute Quickstart

git clone https://github.com/convox-examples/inference-examples.git
cd inference-examples/rag-pipeline-qdrant

convox apps create rag
convox env set HUGGING_FACE_HUB_TOKEN=hf_your_token_here -a rag
convox deploy -a rag

Get the Endpoint

convox services -a rag

The api service is the public endpoint. vllm, embed, and qdrant are internal services.

Test It

1. Ingest documents:

ENDPOINT=$(convox services -a rag | awk '$1 == "api" {print $2}')

curl -s "https://$ENDPOINT/v1/ingest" \
  -H "Content-Type: application/json" \
  -d '{
    "documents": [
      "Convox is an open-source platform for deploying applications on AWS, GCP, and Azure.",
      "Convox uses Kubernetes under the hood but provides a simpler deployment interface.",
      "GPU workloads on Convox support NVIDIA GPUs with automatic scheduling and cost controls."
    ],
    "metadata": [
      {"source": "docs"},
      {"source": "architecture"},
      {"source": "gpu-guide"}
    ]
  }' | jq .

2. Query with RAG:

curl -s "https://$ENDPOINT/v1/query" \
  -H "Content-Type: application/json" \
  -d '{
    "question": "How does Convox handle GPU workloads?",
    "top_k": 3
  }' | jq .

The response includes the LLM's answer with [Source N] citations and the matched source documents with relevance scores.

Architecture

┌──────────┐     ┌──────────┐     ┌──────────┐
│   api    │────►│  embed   │     │  qdrant  │
│  :8080   │     │  :8001   │     │  :6333   │
│ (public) │     │(internal)│     │(internal)│
└────┬─────┘     └──────────┘     └──────────┘
     │                                  ▲
     │           ┌──────────┐           │
     └──────────►│   vllm   │     vector search
                 │  :8000   │───────────┘
                 │(internal)│
                 │   GPU    │
                 └──────────┘

• api — FastAPI gateway: orchestrates ingest + query flows
• vllm — Llama 3.1 8B for answer generation (GPU, internal)
• embed — BGE-small-en-v1.5 for document/query embedding (CPU, internal)
• qdrant — Vector database for similarity search (CPU, internal)

API Endpoints

Endpoint	Method	Purpose
/v1/ingest	POST	Embed and store documents
/v1/query	POST	RAG query with citations
/healthz	GET	Health check

Scaling

• api: Scales horizontally (CPU, stateless)
• vllm: GPU-bound; scale for throughput
• embed: CPU-bound; scale for ingest throughput
• qdrant: Single replica for small datasets; shard for large

For production, set vllm.scale.min: 1 to avoid the LLM cold-start penalty on every query.

Budget Controls

convox budget set rag --monthly-cap-usd 500 --at-cap-action alert-only

GPU Observability

Enable GPU telemetry in Rack Settings to surface per-app GPU utilization, memory, temperature, and inference throughput in the Console GPU Dashboard. In this stack, only the vllm service uses GPU -- the other services are CPU-only.

Troubleshooting

Symptom	Cause	Fix
vllm service not ready	Model still downloading or loading	Check convox logs -a rag --filter vllm; first deploy pulls ~16 GB
Embedding timeout	embed service not reachable from api	Verify all services show healthy in convox ps -a rag
Qdrant connection refused	Qdrant still initializing	Wait for Qdrant health check; initial startup takes 15-30s
Empty search results	Documents not ingested	Call /v1/ingest first; verify with /healthz