GLM-5.2 FP8 Low-Latency Deployment (8×H200)

Deployment record: serving GLM-5.2 (704 GB, FP8) with SGLang inside an Apptainer container on a Slurm HPC cluster. Hardware: single node, 8× NVIDIA H200 141 GB (TP8), driver 580.65.06, CUDA 13.0. Result: 3.41 ms median TPOT / 164 tok/s single-stream with EAGLE MTP (accept length 4.21); 9/9 API smoke tests pass.

Prerequisites

Hardware:

• 1 node × 8 NVIDIA H200 (141 GB HBM3e each)
• Intel Xeon Platinum 8480+ (112 CPU cores)
• ~2 TB RAM
• NVLink: 18 links/GPU at 26.562 GB/s
• Driver: 580.65.06, CUDA 13.0

Software:

• Apptainer/Singularity (for container)
• SGLang v0.5.14 (from lmsysorg/sglang:latest)
• PyTorch 2.11.0+cu130

Account: <project-account>

Storage Layout

All large data lives under $WORK/glm-5.2-sglang/:

$WORK/glm-5.2-sglang/
├── models/GLM-5.2-FP8/          ← 704 GB FP8 model weights
├── containers/sglang-glm52.sif   ← SGLang Apptainer image (11 GB)
├── hf-cache/                     ← Hugging Face cache
├── logs/                         ← Server & job logs
├── results/                      ← Benchmark results
├── torch-cache/                  ← Torch extensions
├── triton-cache/                 ← Triton cache
├── flashinfer-cache/             ← FlashInfer workspace
└── tmp/                          ← Temporary files

Model Download

# Direct download (no sbatch needed)
source config/runtime.env
export HF_HOME HF_HUB_CACHE
mkdir -p "$MODEL_DIR" "$HF_HUB_CACHE"
pip install huggingface_hub
python3 scripts/download_model.py

Or via sbatch:

sbatch --account=$SLURM_ACCOUNT slurm/download_model.sbatch

Container Image

Pulled from docker://lmsysorg/sglang:latest:

source config/runtime.env
export APPTAINER_TMPDIR="$TMPDIR"
export APPTAINER_CACHEDIR="$WORK_ROOT/apptainer-cache"
apptainer pull "$CONTAINER_DIR/sglang-glm52.sif" docker://lmsysorg/sglang:latest

SGLang version: 0.5.14

GPU Diagnostic

Quick single-GPU check:

sbatch --account=$SLURM_ACCOUNT slurm/gpu_diagnostic.sbatch

DeepGEMM Pre-compilation

Pre-compile Triton kernels before serving to avoid 10-20 min JIT warmup:

sbatch --account=$SLURM_ACCOUNT slurm/compile_deep_gemm.sbatch

Uses 8×H200 with the same container and model path. Triton cache saved to $TRITON_CACHE_DIR (~17 MB). Does not delete existing caches.

Low-Latency Serving

Submit 8×H200 low-latency job (1 hour):

sbatch --account=$SLURM_ACCOUNT slurm/serve_low_latency.sbatch

Launch command:

sglang serve \
  --model-path $WORK/glm-5.2-sglang/models/GLM-5.2-FP8 \
  --served-model-name glm-5.2 \
  --tp-size 8 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 6 \
  --max-running-requests 8 \
  --mem-fraction-static 0.8 \
  --host 0.0.0.0 \
  --port 30000

Querying the Server

# Check health
curl http://<node>:30000/health

# OpenAI-compatible API
curl http://<node>:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.2",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100
  }'

SSH Tunnel

ssh -L 30000:node<NNN>:30000 <cluster-login-host>
# Then access via http://127.0.0.1:30000

Check Job Status

sacct -j <JOBID> --format=JobID,JobName,Partition,State,ExitCode,Elapsed,MaxRSS,MaxVMSize
seff <JOBID>

Logs & Results

Path	Contents
$WORK_ROOT/logs/	Job stdout/stderr, server logs
$WORK_ROOT/results/<timestamp>-low-latency/	Benchmarks, smoke test, GPU metrics
$WORK_ROOT/results/<timestamp>-low-latency/gpu_metrics/	GPU dmon data
reports/	Summary reports (environment, deployment status, benchmarks)

Smoke Test Results (Round 2 — Job 162087)

9/9 tests passed.

Test	Status
Visible content without thinking	✅ "Hello, how are you today?"
Tool calling (glm47 parser)	✅ get_weather({"city": "Taipei"})
Reasoning parser (glm45)	✅ reasoning + visible content separated
Total	9 PASS, 0 FAIL

Server started in 200s. All 9 smoke tests passed on first attempt with the updated script.

Benchmark Results (Round 2 — Job 162087)

Benchmark A: Single user (10 prompts, 1 concurrent)

Metric	Round 1	Round 2
Output throughput	66.81 tok/s	164.35 tok/s
Mean TTFT	1498 ms	258 ms
Median TPOT	3.42 ms	3.41 ms
Accept length	4.14	4.21
Successful	10/10	10/10

Benchmark B: Concurrency 4 (32 prompts)

Metric	Round 1	Round 2
Output throughput	252.28 tok/s	322.00 tok/s
Mean TTFT	568 ms	344 ms
Median TPOT	10.30 ms	8.77 ms
Accept length	4.16	4.17
Successful	32/32	32/32

Round 2 improvement from DeepGEMM pre-compilation and fixed smoke test.

Known Issues

1. Tokenizer warning in bench_serving: Fail to load tokenizer config with error=glm-5.2 is not a local folder. Benign — does not affect correctness. --tokenizer "$MODEL" is already passed.
2. No HF_TOKEN set: Unauthenticated downloads are rate-limited. Set HF_TOKEN for faster HuggingFace access.
3. Peak HBM usage ~97%: With --mem-fraction-static 0.8, H200 HBM reaches ~143 GB / 141 GB. Monitor OOM risks under higher concurrency.

Next Steps

1. ✅ Run sglang.compile_deep_gemm** to pre-compile kernels — Done (job 162060)
2. ✅ Fix smoke test for visible content with thinking — Done (job 162087, 9/9 PASS)
3. Balanced deployment: Tune --mem-fraction-static, increase --max-running-requests.
4. High-throughput deployment: Increase concurrency with higher batch sizes.
5. Production: Submit to 8gpus partition for longer wall time (2 days).