vLLM tuning for NVIDIA CMP 170HX

Evidence-led notes for running vLLM on repurposed NVIDIA CMP 170HX cards: Ampere sm_80, 64 GiB HBM per card, PCIe Gen2 x4, and no NVLink or usable peer-to-peer copies. The reference host has seven cards and about 61 GiB of system RAM.

This is a tuning knowledge base, not a claim that every model works on stock vLLM. DeepSeek/GLM sparse-MLA paths remain model- and patch-specific.

Version anchor: vLLM v0.26.0, checked 2026-08-10. Pin a tag, commit, container digest, model revision, and patch set; do not mix main documentation with a stable image.

Start here

• PLAYBOOK.md — staged bring-up, tuning sweeps, and benchmark protocol.
• RESULTS.md — measured local/community results, failures, and unrun experiments.
• SOURCES.md — annotated official, upstream, GitHub, and community evidence ledger.

Evidence labels

Label	Meaning
LOCAL_MEASURED	Reproduced on the reference CMP host under the stated configuration.
UPSTREAM_DOC	vLLM documentation or tagged source behavior.
UPSTREAM_MERGED	Merged upstream; still verify inclusion in the pinned release.
UPSTREAM_OPEN	Open issue or pull request; requires an explicit patch/fork.
COMMUNITY_MEASURED	Reported with a usable configuration and artifacts, but not reproduced here unless also labelled local.
HYPOTHESIS	A topology- or model-derived experiment, not a result.

Conclusions that currently hold

1. Start with PP7/TP1, not TP7. vLLM recommends pipeline parallelism when GPUs lack NVLink, and current PP supports uneven layer partitions. UPSTREAM_DOC Direct CMP measurements on four cards show PP decisively beating TP for long-prefill work, but PP7 itself still needs a controlled measurement.

2. Treat TP7 as a control. TP introduces repeated collectives across a very slow, non-peer fabric. vLLM custom all-reduce only supports world sizes 2, 4, 6, and 8 and also disables itself on unsupported PCIe/P2P topologies. UPSTREAM_DOC

3. Separate weight-load OOM from KV-cache OOM. Reducing context, KV dtype, or gpu_memory_utilization cannot repair an expert-repack failure that occurs before KV allocation. The native GLM-5.2 W4A16 PP7 attempt is a measured example of this distinction.

4. Keep speculation off for the baseline. First qualify plain eager execution, then normal CUDA graphs, and only then MTP/DSpark. Speculation adds weights and capture pressure to one or more ranks. Pipeline-parallel MTP and SM80 sparse MLA are still open upstream work, not stock-v0.26 guarantees.

5. Avoid CPU/KV offload as a default. vLLM explicitly depends on a fast CPU–GPU interconnect for weight offload. PCIe Gen2 x4 plus limited host RAM makes it a fit diagnostic, not an expected performance win. UPSTREAM_DOC

6. Maximum context is empirical. It depends on the exact model, weight path, KV dtype/backend, concurrency, graph capture, and usable memory after loading. Trust the startup KV-capacity report and then qualify correctness, thermals, and preemption at increasing prompt lengths.

Conservative seven-card baseline

For a model that already fits and does not require a custom sparse-MLA path, use the following baseline. The example uses 16,384 tokens only when the model declares at least that limit; otherwise substitute its declared limit:

vllm serve MODEL \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 7 \
  --disable-custom-all-reduce \
  --attention-backend auto \
  --gpu-memory-utilization 0.85 \
  --max-model-len 16384 \
  --max-parallel-loading-workers 1 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 1024 \
  --no-enable-prefix-caching

This is a diagnostic starting point, not the final performance profile. Leave attention selection automatic, keep prefix caching off for cold benchmarks, and raise one dimension at a time. See the playbook before changing context, KV dtype, CUDA graphs, batch size, or parallelism.

For DeepSeek/GLM DSA models, use a model-specific pinned patch set and record every deviation from vLLM v0.26.0. Do not present an open pull request as a released feature.

Known no-gos

• Do not infer aggregate bandwidth from 7 × 64 GiB; each pipeline stage must fit its own weights, temporary workspaces, runtime state, and KV share.
• Do not force NVLink/P2P-oriented collectives or all-to-all backends on this topology.
• Do not use --enforce-eager as a performance recommendation; it is a bring-up and memory-isolation tool.
• Do not compare llama.cpp GGUF numbers directly with native/patched vLLM: model format, kernels, placement, batching, cache, and prompt processing all differ.
• Do not publish a context ceiling from server startup alone. Test retrieval, generation, accumulated conversation behavior, thermals, and cleanup.

Contribution rule

Every result should include model revision, vLLM/container commit, patch set, CUDA/PyTorch versions, card count, PP/TP/DP, weight/KV dtype, context, concurrency, graph/eager state, prompt/output lengths, warmup policy, and separate prefill, TTFT, ITL/TPOT, and decode measurements. Failures belong in the ledger too.