High-performance LLM inference engine in C++/CUDA for NVIDIA Blackwell GeForce / RTX PRO (RTX 5090/5080/5070 Ti, RTX PRO 6000; sm_120). 200 tok/s decode on Qwen3.6-35B-A3B-NVFP4 MoE (RTX 5090).
cpp cuda inference quantization inference-engine mixture-of-experts blackwell llm safetensors qwen gguf rtx-5090 rtx-5080 sm120 nvfp4 rtx-pro-6000 cuda-graphs rtx-5070-ti gated-deltanet qwen3-6
-
Updated
May 14, 2026 - Cuda