tensor-parallel

A light llama-like llm inference framework based on the triton kernel.

GLM-5.3-Flash EXL3 (320B MoE) on 2x NVIDIA DGX Spark — production serving kit, 1M context, 97%+ multi-session prefix caching, DFlash2 spec decode

Reproducible recipe: serve abliterated Gemma-4-12B (gemma4_unified) at 50-118 tok/s on no-NVLink Blackwell (SM120) via vLLM nightly + ModelOpt FP8/NVFP4 + MTP spec-decode.

Reproducible kit to deploy DeepSeek-V4-Flash-DSpark on a 2× NVIDIA DGX Spark (GB10) cluster: vLLM TP=2 over QSFP 200GbE, NVFP4 KV, DSpark speculative decoding, 1M context, systemd self-heal. Apache-2.0.

Minimal SGLang patch for Qwen3.8-27B + DFlash2 on dual RTX 3080: TP-sharded Draft fc and static per-head FP8 KV.

Reproducible llama.cpp kernel and runtime optimization lab for dual NVIDIA Tesla V100 GPUs (SM70)

Patches + recipe to deploy festr2/MiMo-V2.5-Pro-NVFP4-MXFP8-attn-TP8 on 8-node DGX Spark sm_121 (Ray + vLLM, TP=8). Fixes the fused-qkv loader bug that mis-slotted Q values as K/V on 7 of 8 ranks.

Field-tested guide: multi-GPU vLLM tensor-parallel (TP=2/TP=4) on Intel Arc Pro B70 (Battlemage BMG-G31, Xe2) on Linux. Driver setup (xe force_probe=e223), bare-metal vLLM + oneAPI 2025.3, the compute-runtime multi-root USM + triton-xpu init_devices fixes, FP8/int4-AutoRound quant, root-cause error reports. AI-agent readable (AGENTS.md).

Run zai-org/GLM-5.3-Flash (NVFP4) on 3x NVIDIA DGX Spark with vLLM, TP=3 + EP, DFlash2 speculative decoding, CUDA graphs. Full recipe with rationale, benchmarks, patches, and what we tried.

GLM-5.3-Flash NVFP4 vLLM TP=2 serving cluster on 2x NVIDIA DGX Spark (GB10) — one compose, one entrypoint, stacks as pure config

A ~800-line PyTorch implementation of Megatron-LM's TP + PP + DP + AMP. 1.6-2x faster than Megatron-Core on 125M models.

Verified four-GB10 DeepSeek-V4-Flash-0731 TP4 DSpark Graph-8 deployment and benchmarks

Deploy DeepSeek-V4-Flash-0731 on dual NVIDIA RTX PRO 6000 Blackwell GPUs with vLLM PR #41834 (jasl fork) and DSpark speculative decoding, achieving ~200-227 tok/s in no-overseas-network environments.

Serve an abliterated Gemma-4-12B at high speeds on Blackwell GPUs without NVLink using vLLM, FP8 quantization, and MTP speculative decoding.

Two-node DGX Spark operator tooling and measurement harness for DeepSeek-V4-Flash

Serve DeepSeek-V4-Flash-0731 on 2x NVIDIA GB10 / DGX Spark (sm_121) with vLLM TP=2 over 200GbE RoCE — runbook, community survey, tuning, and an L1-L3 benchmark suite.

Qwen3.8-Flash-Next (NVIDIA NVFP4 weights) served as W4A16 on 2x DGX Spark GB10 with vLLM, TP=2. Pinned engine build, 4 start-time patches, measured results, and the levers that were tried and rejected.

Measured LLM benchmarks for NVIDIA DGX Spark (GB10): DeepSeek-V4-Flash 284B MoE on a TP=2 pair over 200G RoCE — tok/s by profile and concurrency, 1M-token context curve, the MoE backend flag, monitoring traps. Every number links to raw runs.

Evidence-backed GLM-5.3-Flash NVFP4 TP4 deployment validation on 4x NVIDIA GB10.

C++/SYCL local LLM inference for Intel Arc GPUs, with XMX kernels, quantized models, multi-GPU execution and GLM-5.3-Flash support.