A C++/SYCL local LLM inference engine for Intel Arc GPUs, built and tuned on two Arc Pro B70 cards. XMX kernels, quantized models, and multi-GPU execution.
Intel Arc is a genuinely capable AI GPU that inference tooling has mostly ignored. Mach X is built for it from the metal up — no fork of llama.cpp, no PyTorch, no vendor runtime. Hand-written SYCL kernels (XMX matrix engines, int-dot quantized GEMV, tiled FlashAttention), an OpenAI-compatible server, tensor-parallel multi-GPU, and day-one support for the newest model architectures — often running them fast on Arc before anyone else does.
Same GGUF, same GPU (1× Arc Pro B70), llama.cpp on its fastest config (FlashAttention on):
| context | Mach X prefill | llama.cpp | speedup | Mach X decode | llama.cpp | speedup |
|---|---|---|---|---|---|---|
| 512 | 1795 t/s | 927 | 1.94× | 58.3 t/s | 50.3 | 1.16× |
| 2K | 4147 t/s | 927 | 4.47× | 57.4 t/s | 49.9 | 1.15× |
| 4K | 3428 t/s | 896 | 3.83× | 55.6 t/s | 49.4 | 1.13× |
Wins both axes at every context length, and stays flat as context grows. Clean-box, reproducible (ie-bench vs llama-bench).
- 🏛 Dense, MoE and hybrid models — GLM-5.3-Flash, DeepSeek-V4-Flash, Qwen3.8-Flash-Next, Qwen3.6, Qwen3 / Coder / Tongyi, Qwen3-Next, Gemma-4, gpt-oss, and Llama-compatible dense models. GLM-5.2 and Tencent Hy4-preview have experimental standalone runners. See architecture coverage for entry points and status.
- 👁 Native vision — Qwen3.8-Flash-Next and experimental DeepSeek-V4-Flash-Vision-Exp, including image inputs through the OpenAI-compatible API. DeepSeek vision requires its native vision sidecar weights.
- 🥇 Beats llama.cpp on Arc — on prefill and decode across the models below.
- 🧠 Runs the big ones — gpt-oss-120b (117B) and Qwen3-Next-80B on 2× B70 via tensor-parallel; ~2.5× faster than LM Studio on 120b.
- 🔀 Multi-GPU built in —
ie serve --gpus 2(tensor-parallel + layer-split), no P2P required. - 🔌 OpenAI-compatible server + tool-calling (Harmony + Qwen) + image inputs (
image_urldata URIs) — point any OpenAI client (or Hermes) at:11435. - 📦 One-command Docker —
docker pull(or build) →ie-docker serve→ running on your Arc GPU in minutes. - ✅ Correctness-first — PPL-validated, per-layer cosine ≈ 1.0 vs a llama.cpp oracle, bit-exact where claimed.
Coverage below describes this source branch. A standalone runner does not imply
integration with ie serve; experimental ports have narrower qualification than
the benchmarked models. Prebuilt container images may lag these source updates.
| Family / GGUF architecture | Models | Entry point and coverage |
|---|---|---|
GLM-5.3-Flash · glm5next |
UD-Q4_K_XL GGUF | ie-glm5next-run; sparse MLA + KDA, host-resident MoE, two-GPU pipelined prefill and MTP draft; kernel and full-model validation in PERFORMANCE.md |
DeepSeek-V4 · deepseek4 |
Flash, Flash-Vision-Exp | ie serve; streaming expert caches, long-context sparse attention, prompt caching and structured tool calls; experimental native vision requires sidecar weights |
Qwen3.8-Flash-Next · qwen4exp |
Qwen4 preview | ie serve; DeltaNet + sparse QSA, hyper-connections, PLE embeddings, streamed MoE and native vision |
Qwen3.5 / Qwen3.6 hybrid · qwen35, qwen35moe |
27B dense, 35B-A3B MoE | ie serve; gated-DeltaNet + full attention, dense or MoE feed-forward paths |
Qwen3 MoE · qwen3moe |
Coder-30B-A3B, Tongyi-30B | ie serve; QK-normalized attention and routed MoE |
Qwen3-Next · qwen3next |
80B-A3B | ie serve; DeltaNet + full attention and 512-expert MoE |
gpt-oss · gpt-oss |
20b, 120b (MXFP4) | ie serve; attention sinks, sliding-window attention, Harmony chat and tool calls |
Gemma-4 · gemma4 |
31B dense, 26B-A4B MoE | ie serve; per-layer head geometry, sandwich norms, softcap and sliding-window attention |
Qwen dense · qwen2, qwen3 |
Qwen2/2.5/3, compatible Qwen distills | ie serve; shared dense transformer path with architecture-specific attention handling |
Llama-compatible dense · llama, phi3, granite |
Llama-3.x, compatible Mistral, Phi and Granite GGUFs | ie serve; shared dense path; compatibility depends on GGUF architecture and tensor layout |
GLM-5.2 · glm-dsa |
GLM-5.2 | Experimental ie-glm52-run; standalone MLA + MoE forward path |
Tencent Hy4-preview · hyv4 |
Hy4-preview | Experimental ie-hyv4-run; standalone generation/PPL, two-GPU stage split and STQ1/IQ1 quantization support; no general server qualification claimed |
Recognized without an inference runtime: Inkling-Small (inkling) and
Laguna S 2.1 (laguna). Loader recognition is not runnable model support.
Weight import: ie import converts supported AWQ, GPTQ and EXL3 safetensors
to native GGUF. Import format support does not add an unsupported architecture.
GLM-5.3-Flash now uses faster expert top-k selection, B70 KDA recurrence and shape-specialized attention indexing. In-model kernel buckets improved by 1.61×, 1.33× and 1.24×, respectively, in the measured 24K profiling run. Shared Q8 projections, FP16 XMX tiles, MLA tile reuse and convolution state handling also received performance or correctness improvements.
These are kernel gains. Repeated full-model comparisons did not establish a combined throughput improvement; slower runs also showed more expert staging and host I/O. See performance and validation for the measured results, rejected experiments, test coverage and reproduction commands.
📊 Interactive charts → · all measured on Arc Pro B70 hardware; gpt-oss rows are clean-box head-to-head with identical GGUFs.
gpt-oss-120b (117B, MXFP4) — 2× B70, tensor-parallel:
| metric | Mach X | LM Studio (same 2 cards) |
|---|---|---|
| decode | ~31 tok/s (peak 32) | ~12.4 tok/s |
| fit | full MXFP4, display-safe | — |
Coherent Harmony chat (math / poem / factual + multi-turn) and function-calling tool use. Batched-prefill PPL 15.20, bit-identical to T=1.
Qwen3.6-35B-A3B "crown" (all-Q8_0, ~36 GB) — 2× B70 vs llama.cpp SYCL layer-split:
| axis | Mach X | llama.cpp | speedup |
|---|---|---|---|
| prefill | 963 t/s | 763 | 1.26× |
| decode | 63 t/s | 42 | 1.49× |
PPL 6.36. Hybrid gated-DeltaNet + 128-expert MoE — one of the hardest architectures to run correctly, let alone fast.
Tongyi-DeepResearch-30B (qwen3moe) — 2× B70 tensor-parallel, long context (~17K):
| axis | layer-split | tensor-parallel | speedup |
|---|---|---|---|
| prefill | 124 t/s | 291 t/s | 2.35× |
| decode | 21 t/s | 27.4 t/s | 1.30× |
Gemma-4 prefill (sliding-window attention) vs llama.cpp: 2.03× @4K, 1.91× @8K, 1.58× @16K.
Qwen3.6-27B dense vs llama.cpp SYCL: prefill 1.21× (349 vs 288 t/s).
Speculative decode (self-drafting MTP head, lossless-greedy): Gemma-4 1.46×, Qwen3.6-27B 1.47×.
Qwen3.8-Flash-Next (UD-Q4_K_XL, 104 GB, host-resident experts) — 2× B70 layer-split + P2P:
| axis | figure |
|---|---|
| decode (steady, 4K ctx) | 27.4 tok/s (bit-identical to single-GPU) |
| pipelined prefill | ~540 tok/s (P2P double-bank, bit-exact vs host bounce) |
| vision: image → first token | ~1 s for a 512² image (GPU ViT, overlapped with init) |
Vision demo: a 1236×1343 terminal screenshot is read at 990 vision tokens with OCR-level detail ("VS Code terminal… session capture… segmentation fault…"), ~15 tok/s decode with the image in context.
Methodology:
ie-bench --prefill P --decode Nmirrorsllama-bench -pP -nN; runs are order-controlled and heat-soaked. A few non-gpt-oss figures predate the latest clean-box sweep and are being re-verified — the gpt-oss head-to-heads are ledger-verified.
2× Intel Arc Pro B70 — Battlemage (BMG-G31), 32 GB GDDR6 each (64 GB total), 608 GB/s bandwidth, ~183 FP16 TFLOPS via XMX. oneAPI 2026.x / SYCL. All single- and multi-GPU benchmarks above are on this hardware.
Docker (recommended) — pull the prebuilt image (or build it yourself), then serve any GGUF on your Arc GPU:
docker pull ghcr.io/red-weasel/ie-engine:latest && docker tag ghcr.io/red-weasel/ie-engine:latest ie-engine
# ── or build from source (~15 min): docker build -t ie-engine .
./scripts/ie-docker pull llama8b # or any Hugging Face GGUF
./scripts/ie-docker serve /models/…/model.gguf --gpus 1
# → OpenAI-compatible server on :11435 (point any OpenAI client at it)Full 5-minute path in QUICKSTART.md.
From source (needs oneAPI 2026.x + an Intel Arc GPU):
source scripts/env.sh
cmake -S . -B build -G Ninja && cmake --build build -j
./build/src/ie pull llama8b
./build/src/ie serve <model.gguf> --gpus 1Multi-GPU: add --gpus 2 (VRAM-aware; tensor-parallel + layer-split). Runs models bigger than one card — e.g. Qwen2.5-72B or gpt-oss-120b across 2× B70.
GLM-5.3-Flash standalone runner (two B70 cards, host-resident expert banks):
./scripts/ie-run-guarded --mem 220G ./build/tools/ie-glm5next-run \
<model-00001-of-00006.gguf> --gpus 2 --ctx 32768 \
--prompt "Your prompt" --ngen 128Prefill uses chunks of 1024 tokens and overlaps the two stages automatically,
including long --prompt input. Generation of 64 or more tokens automatically
uses MTP pipedraft when the model has an MTP head. Short generation, PPL scoring,
prefill benchmarks, plain-decode profiling/logit dumps and expert-parallel experiments do
not automatically load the draft head. Use --no-pipeline --no-pipedraft for
serial comparisons, or --pipedraft to request drafting explicitly. Drafting
uses extra model memory; its benefit depends on draft acceptance. These options
apply to the standalone runner, not the server.
Exact continuation comparisons use IE_G5_CPU_MISS=0; normal q* CPU/GPU expert
routing can produce different continuations between the two schedules.
Warm prefill batches that exceed the expert cache use waves of at most 16
experts, leaving room to retain weights needed later in the chunk. Cold
batches, batches that fit, and decode keep their existing wave width. This
adds no GPU workspace; IE_G5_PP_WAVE=0 restores the previous half-cache
wave schedule for comparisons.
For prefill kernel timings, set IE_QUEUE_PROFILING=1 and use
--ppl <corpus> --ppbench <chunks>; the runner reports instrumented kernel
time by bucket and device. Device windows cover the whole profile, including
gaps between that device's chunks.
Shared Q8 projections use shape-aware decode workgroups and combine aligned prefill batches of at least 128 tokens into one grid. Small FP16 XMX GEMMs use bounded 64-row tiles; larger or unsupported shapes retain the original 128-row path. These dispatches also apply to other models using the same operators. GLM MLA projections load ahead without changing their accumulation order, and sparse attention reuses latent tiles across heads. See performance and validation for the measured results and limits.
- Quantized GEMV — W4A8/W6A8/W8A8 int-dot kernels (dp4a) over SoA-repacked weights: read each weight once, decode in-register. Q4_K, Q6_K, Q8_0, Q5_K, MXFP4.
- FlashAttention — register-tiled SIMD inner loop (no XMX for attention, following the fastest llama-SYCL path), plus split-K decode, sliding-window, and attention-sink variants.
- MoE — expert-batched weight-stationary prefill + fused gate/up/down; oneDNN XMX GEMM for the large-M regime.
- Multi-GPU — head-sharded attention + expert-sharded MoE (tensor-parallel) with host-bounced all-reduce; layer-split for pure capacity (bit-identical to single-GPU).
- Speculative decode — self-drafting NextN/MTP head with batched int-dot verify, lossless vs greedy.
- Vision — the model's own 449M SigLIP-style ViT ported natively: XMX GEMMs + custom LN / h-w rope / bidirectional packed-attention kernels, numpy-oracle-gated to 2e-6; embeddings splice into the LLM with true 3-stream interleaved M-RoPE (bit-identical to text rope when no image is present).
- P2P pipeline — 2-GPU layer-split with device-to-device wide-state push and double-banked chunk pipelining; every transport certified bit-identical.
See MASTER_DEV_PLAN.md for the authoritative state and roadmap.
Apache License 2.0 — see LICENSE. Copyright © 2026 Red-Weasel.
Free to use, modify, and ship (including commercially). Apache-2.0's patent grant + retaliation clause protects you and downstream users.
If Mach X saved you time — or you just want to see more fast inference land on Intel Arc — you can support the work:
All donations go straight back into the project. Requests and suggestions are welcome — open an issue.