qvllm

qvllm is a compact single-GPU inference engine for Qwen3, Qwen3.5, and Qwen3-ASR. The Python runtime follows the offline engine shape of nano-vLLM: requests enter through LLM.generate, are scheduled by a small prefill/decode scheduler, and use paged KV cache plus GDN recurrent state for hybrid models. Model-critical operators use a common interface with interchangeable handwritten CUDA, PyTorch, and Triton backends.

Supported local checkpoints:

• /home/vhagor/model_repo/qwen3-0.6B
• /home/vhagor/model_repo/Qwen3.5-0.8B (Qwen/Qwen3.5-0.8B via ModelScope)
• /home/vhagor/model_repo/Qwen3-ASR-0.6B (Qwen/Qwen3-ASR-0.6B via ModelScope)

Relationship to nano-vLLM

qvllm uses and modifies code from nano-vLLM, including its offline LLM.generate API shape, request lifecycle, scheduler, sequence and block management, model runner, model/layer organization, prefix caching, and paged KV cache flow. qvllm retains that compact Python control plane while adding a selectable operator backend and project-local CUDA and Triton implementations.

nano-vLLM is distributed under the MIT License, copyright (c) 2025 Xingkai Yu. The original copyright and permission notice are retained in this repository's LICENSE. qvllm is distributed under the same MIT License.

Scope

Supported:

• Single GPU inference only.
• Qwen3 decoder-only and Qwen3.5 hybrid (GDN linear attention + full attention) models.
• Qwen3.5 visual preprocessing and vision-tower merge for image prompts.
• Qwen3-ASR audio preprocessing (Whisper mel features) and AuT encoder merge for speech prompts. The audio tower uses PyTorch Conv2d / SDPA; no new CUDA or Triton ops.
• Offline LLM.generate API, modeled after nano-vLLM.
• bf16 weights, activations, KV cache, GDN state, and attention output.
• Paged KV cache with prefix caching (disabled on hybrid GDN models).
• Selectable cuda, torch, and triton operator backends.
• Matching activation, add, attention, embedding, GDN, layernorm, matmul, and RoPE interfaces.
• Custom FlashAttention subset plus Gated DeltaNet conv / recurrent kernels.
• Single-node tensor parallelism (tensor_parallel_size > 1) via nano-vLLM-style weight sharding and NCCL. All three backends share this path because GEMM/attention run on local shards.

Out of scope for the current implementation:

• HTTP/OpenAI-compatible server.
• Pipeline parallelism, data parallelism, or multi-node execution.
• fp16/fp32 inference paths.
• Attention features such as dropout, alibi, sliding window, non-causal masks, or training.

Repository Layout

qvllm/
  csrc/                 # PyTorch CUDA extension entry points and handwritten CUDA ops
  triton_src/           # Triton implementations matching the native operator surface
  include/              # Common C++/CUDA headers and utility macros
  python/               # Offline engine, scheduler, model, layers, and utilities
  example/              # Minimal examples and local experiments
  tests/                # Python tests for engine components and CUDA op references
  docs/                 # Architecture, CUDA backend, and development docs

Key Python modules:

• python/llm.py: public offline LLM wrapper.
• python/engine/llm_engine.py: request ingestion and generate loop.
• python/engine/scheduler.py: prefill/decode batching and preemption.
• python/engine/block_manager.py: paged KV allocation and prefix cache hashes.
• python/engine/model_runner.py: model loading, KV allocation, CUDA graph capture, and batch metadata preparation.
• python/layer/attention.py: qvllm attention bridge into qvllm_ops.
• python/model/qwen3.py: Qwen3 model definition.
• python/model/qwen3_5.py: Qwen3.5 hybrid GDN + full-attention model.
• python/model/qwen3_5_vision.py: Qwen3.5 image packing, mRoPE, vision tower, and embedding merge.
• python/model/qwen3_asr.py: Qwen3-ASR audio encoder + Qwen3 decoder.
• python/model/qwen3_asr_audio.py: Qwen3-ASR audio packing, AuT encoder, and embedding merge.
• python/layer/gdn.py: Gated DeltaNet layer over the three backends.

Key CUDA modules:

• csrc/bind.cpp: pybind registration for qvllm_ops.
• csrc/ops/*/op.cpp: device dispatch and input validation.
• csrc/ops/*/cpu: CPU stubs. Performance ops should fail fast on CPU.
• csrc/ops/*/nv: handwritten CUDA kernels.

Quick Start

Install qvllm and build the CUDA extension:

bash scripts/install.sh

The install script creates .venv by default, installs a CUDA PyTorch wheel, then builds qvllm_ops with safe defaults for WSL / limited-memory hosts:

• TORCH_CUDA_ARCH_LIST=12.0 (RTX 50-series / sm_120; override for other GPUs)
• MAX_JOBS=2 (keeps parallel nvcc jobs from exhausting WSL RAM)

If you need a different PyTorch CUDA wheel, override the index:

TORCH_INDEX_URL=https://download.pytorch.org/whl/cu130 bash scripts/install.sh

If CUDA PyTorch is already installed:

SKIP_TORCH=1 bash scripts/install.sh

Override GPU arch or compile parallelism when needed:

TORCH_CUDA_ARCH_LIST="8.9" MAX_JOBS=2 bash scripts/install.sh

Checkpoints and eval sets are expected under /home/vhagor/model_repo/ (Qwen3-0.6B, Qwen3.5-0.8B, Qwen3-ASR-0.6B, plus optional GSM8K / MME / ASR clips). Put them there with ModelScope or Hugging Face; the examples take those paths as defaults.

Run a Qwen3.5 text example (after rebuilding qvllm_ops with MAX_JOBS=2):

python example/qwen35_generate.py

Image prompts use the same engine and the Qwen3-VL processor path:

llm.generate(
    ["Describe the image."],
    SamplingParams(max_tokens=64),
    images=["/path/to/image.png"],
)

Run ASR or GSM8K / MME smoke evals against local files:

python example/qwen3_asr_transcribe.py
python example/eval_asr.py --backend cuda --limit 6
python example/eval_gsm8k.py --backend cuda --limit 4

Audio prompts use Whisper features from the HF processor:

llm.generate(
    [""],
    SamplingParams(temperature=1e-5, max_tokens=64),
    audios=["/path/to/clip.wav"],
    language="English",  # optional; omit for automatic language detection
)

Run a minimal Qwen3 generation example:

from python import LLM, SamplingParams

llm = LLM(
    "/home/vhagor/model_repo/qwen3-0.6B",
    backend="cuda",  # "cuda", "torch", or "triton"
    enforce_eager=True,
    tensor_parallel_size=1,  # set >1 for single-node TP; needs that many GPUs
)

outputs = llm.generate(
    ["你好，请介绍一下你自己。"],
    SamplingParams(temperature=0.6, max_tokens=128),
)

print(outputs[0]["text"])

The extension module is normally named qvllm_ops. If you build it under a different module name, set:

export QVLLM_EXT_MODULE=qvllm_ops

Build

See docs/INSTALL.md for the full installation process. The short manual path is:

source .venv/bin/activate   # or create one first
python -m pip install --index-url https://download.pytorch.org/whl/cu130 torch
python -m pip install -r requirements.txt
python -m pip install -e . --no-build-isolation --no-deps
TORCH_CUDA_ARCH_LIST="12.0" MAX_JOBS=2 python setup.py build_ext --inplace
python scripts/check_env.py

Always set TORCH_CUDA_ARCH_LIST and MAX_JOBS when rebuilding the extension by hand; unbounded multi-arch / high-concurrency builds can OOM WSL.

The build produces a local qvllm_ops*.so extension. Python layer wrappers lazy-load this extension so pure engine modules can still be imported before the CUDA extension exists.

Architecture

The runtime mirrors nano-vLLM's compact offline design:

flowchart LR
  User["prompts"] --> LLM["LLM.generate"]
  LLM --> Scheduler["Scheduler"]
  Scheduler --> Runner["ModelRunner"]
  Runner --> Model["Qwen3ForCausalLM"]
  Model --> Attention["Attention Layer"]
  Attention --> Ops["qvllm_ops"]
  Ops --> KV["Paged KV Cache"]
  Ops --> Out["Logits / Tokens"]

Loading

The important difference is the backend boundary. qvllm keeps the same high-level engine shape while routing inference operators through one of three implementations: handwritten CUDA under csrc, PyTorch reference operations, or Triton kernels under triton_src.

See docs/ARCHITECTURE.md, docs/BACKENDS.md, and docs/CUDA_BACKEND.md for details.

Testing

Lightweight Python syntax checks:

python -m py_compile python/layer/attention.py python/engine/model_runner.py tests/test_attention_ops.py

Run tests after building the extension in a CUDA environment:

pytest tests

The attention tests compare small bf16 CUDA outputs against PyTorch reference implementations and skip automatically when CUDA or qvllm_ops is unavailable.

Development Notes

• Keep the public Python interface close to nano-vLLM's offline LLM.generate path.
• Keep custom CUDA op validation in op.cpp; keep kernels in nv/*.cu.
• Keep Triton operator signatures aligned with the public CUDA operator surface.

See docs/DEVELOPMENT.md for coding conventions.