Skip to content
BranchesTags

Repository files navigation

MIInfer

MIInfer is an experimental, performance-first LLM inference runtime designed specifically for AMD gfx906, initially targeting the AMD Instinct MI50 32GB.

The project explores a simple question:

How much inference performance can be extracted from gfx906 when the runtime, memory layout, execution plan, and GPU kernels are designed specifically for the architecture instead of treating it as a legacy backend of a general-purpose inference framework?

MIInfer is intentionally specialized.

It is not intended to become another general-purpose llama.cpp, vLLM, PyTorch, or generic ROCm inference runtime.

Status

M17 local appliance quick start

From a release archive on a supported MI50/gfx906 Linux machine:

./install.sh miinfer-0.2.0-gfx906-Linux.tar.gz
~/.local/miinfer/bin/miinfer doctor --model ~/models/Qwen3.8-27B-Q4_K_M.gguf
~/.local/miinfer/bin/miinfer models ~/models
~/.local/miinfer/bin/miinfer serve --model ~/models/Qwen3.8-27B-Q4_K_M.gguf

Open http://127.0.0.1:8080/ for the bundled Web UI. It uses the same public /v1/chat/completions endpoint as external OpenAI-compatible clients.

Current phase: M18–M20 qualification closure

M11-B is frozen at the qualified 46.22 tok/s P513 packed-Q4 baseline. The opt-in layer-major prefill path reaches that rate with deferred full-attention tails, shape-specific Q5_K B=4 reuse, and direct consumption of batched workspaces. The M11-B 100 tok/s gate is not met; the production-shaped row-LDS and generic B=8 accumulator extensions were rejected in experiments/EXP-0211-m11b-row-lds-production-shape.md and experiments/EXP-0212-m11b-batched8-accumulator-gemv.md. EXP-0213 rejected dequantize-then-hipBLAS FP16 GEMM because conversion dominated the measured projection. EXP-0214's release re-evaluation measured native Q4_K split-K at 0.64× the B=4 control, and EXP-0215 rejected a larger logical chunk around B=4 microtiles as neutral at roughly 2.6 GiB extra workspace. The default token-major path remains unchanged. EXP-0216 also rejected generic Q5_K B=4 ssm_out batching after P513 fell from 39.98 to 31.61 tok/s. EXP-0217 keeps the full-attention tail and replaces that generic Q5_K mapping with a shape-specific kernel; the matched P513 control is 39.90 tok/s and the candidate is 45.56 tok/s after EXP-0221's copy elimination.

EXP-0230 measured a raw int8 GEMM ceiling at 1.387x over native Q4_K B=4 launches. EXP-0231 tested a three-plane Q4_K repack with a 64-token MMQ tile: it reached 1.125x at B=64 but was 13.8x slower at the production B=4 chunk, so it is rejected pending a causally valid larger-chunk schedule. EXP-0232 tested a 16-token version and reached only 0.351x the four-launch B=4 control; the intermediate tile is also rejected. EXP-0233 closes this projection mapping branch with an optimistic 1.387x Amdahl ceiling, or 63.19 tok/s even if applied to the whole current path; the 100 tok/s gate remains open for a materially different prefill dataflow. EXP-0234 keeps a causal recurrent-core B=4 batch inside layer-major prefill, improving P513 from 45.86 to 46.19 tok/s with approximately 12.1 MiB of bounded workspace. EXP-0235 emits the recurrent Q8_1 output in that same core, measuring 46.22 versus 45.90 tok/s in one matched P513 pair; it does not change the projection ceiling or the gate. EXP-0236 rejects direct Q8_1 emission from the existing full-attention reduction: it changed P513 by only +0.04% and was removed. EXP-0237 adds a disabled-by-default production prefill profile: at P512, recurrent layers consume 8.41 s of measured layer work, including 6.07 s in the deferred tail and 1.52 s in prepared projections. The ordered recurrent region is only 0.82 s, so the next design must raise projection token parallelism rather than start with a recurrent scan. EXP-0238 validates a causal 64-token staging schedule but rejects it because repeated B=4 launches are neutral against the qualified baseline and the fixed staging footprint is 16x larger. EXP-0239 rejects a native-layout 16-token Q4_K tile: it matches the B=4 path within 2.5e-6 but reaches only 0.459x its speed at the exact FFN-down production shape. EXP-0240 also rejects a wave-major 16-token variant: it matches within 3.7e-6 but reaches only 0.547x. EXP-0241 rejects one-row-per-workgroup token reuse at 0.591x. A materially different quantized projection dataflow or recurrent-tail fusion is still required for the 100 tok/s gate. EXP-0242 also rejects direct Q8 emission from recurrent fused normalization: P513 changed only 0.15%. EXP-0243 rejects paired SwiGLU/Q8 producer-consumer fusion: exact-shape timing fell from 372.959 us to 475.999 us (0.784x), despite removing four quantizer launches. EXP-0244 records the final measured ceiling: current P513 qualification is 46.22 tok/s, fresh P2049 is 43.31 tok/s, and the optimistic raw-int8 whole-path ceiling is about 64.1 tok/s. The 100 tok/s gate remains unmet and requires a new causal grouped-dataflow experiment. EXP-0245 rejects a native 16-token split-4 output-stationary mapping: it was correct within 1.90735e-6 but reached only 0.594x four B=4 launches. EXP-0247 rejects a two-thread-per-output-cell 64-token Q4_K decomposition at 0.116x; EXP-0248 rejects operator-major tail ordering at 0.993x end-to-end; EXP-0249 rejects native 64-row MMQ staging at 0.527x; and EXP-0250 rejects expanded decoded-metadata grouping at 0.098x. All temporary candidates were removed. The 100 tok/s gate still requires a genuinely new causal grouped-dataflow design. EXP-0252 retested the external-style repacked MMQ64 shape with exact activation-sum correctness: it was 0.978x at B64 and much slower at B4/B16, so it was also removed. EXP-0253 tested a bounded exact-sum side buffer; it fell to 0.649x at B64 and was removed as well. EXP-0254 kept the exact sum in the candidate activation footprint; it improved the grouped tile to 0.894x at B64 but remained slower and was removed. The repacked-MMQ/token-reuse family is now closed. EXP-0255 measured one Q4_K→FP16 staging pass plus hipBLAS GEMM at 1.364x for B64, then 2.724x, 4.001x, and 5.806x for B128/B256/B512. It remains a lab primitive until chunkwise recurrent execution can expose B128+ work. EXP-0256 validates the corrected 16-key-head/48-value-head chunkwise Gated DeltaNet WY oracle at chunk 64: max output error is 1.4e-8 and final state error is 1.5e-7 against the token recurrence. EXP-0257 passes the gfx906 correctness gate and reaches 3.17x over 128 token launches in isolation. EXP-0258 composes it into the opt-in runtime path; exact P512 is 47.79 tok/s in the earlier matched integration pair. The 128-token schedule is operational but neutral at 47.64 versus 47.74 tok/s, so it remains experimental. EXP-0259 adds an opt-in recurrent FFN-down dense backend: repeated exact-P512 measurements are 51.56 tok/s versus 46.60 tok/s control, a 10.6% gain. It keeps only canonical quantized sources per layer plus shared FP16 workspaces; the default and decode paths remain unchanged. B256 runtime capacity is rejected as out-of-memory. EXP-0260 composes dense B128 with chunkwise GDN at 52.99 tok/s P512; it remains opt-in because the aspirational 60 tok/s gate is still open. EXP-0261 tested resident quantized Q4_K/Q6_K multi-token projection at B64–B2048: the exact Q4 FFN-down candidate reached only 0.860–0.863x of repeated B4 and the Q6 candidate 0.695–0.717x, with exact B64 output agreement. M13 is closed without runtime integration; M14 release and packaging work is next.

M14 now provides a focused MI50 release package. It contains the miinfer runtime, the miinfer-device-info gfx906 contract probe, and operational documentation; benchmark and historical audit binaries remain build-tree only. The v0.1.0 release is tagged at c7b3737. The package gate is scripts/test-package.sh; M12 remains opt-in until its boundary, continuation, replay, and long-context qualification campaign passes. EXP-0262 records the original GDN qualification failure; EXP-0263 fixes that overflow but rejects the dense FFN-down path because its P128 greedy token differs from baseline. The default runtime remains unchanged.

M15 provides stable miinfer --version, miinfer config, and miinfer models [directory] contracts for installers, runtime configuration, and model discovery. The serving API now exposes health/readiness endpoints and reports the loaded model identity. M16 adds dependency-free Prometheus request/token counters, parsed bounded request queueing, and localhost-safe serving defaults; concurrent GPU execution and full JSON message parsing remain unqualified.

The project currently has:

  • a reproducible MI50/gfx906 development environment

  • trusted correctness references

  • reproducible performance benchmarks, including the first end-to-end M5-A baseline

  • kernel-level profiling infrastructure

  • a strong llama.cpp/gfx906 comparison baseline

  • a pinned Qwen3-8B MI50 execution path with persistent KV-cache decode

  • a model-backed tokenizer and minimal text-facing greedy CLI

  • a completed Qwen3.8-27B GGUF/architecture audit; see experiments/EXP-0042-m6a0-qwen38-27b-gguf-audit.md

  • native Qwen3.8-27B GPU generation through 128 tokens with deterministic replay and zero decode-loop allocations; see experiments/EXP-0092-m6a28-native-qwen35-generation.md

  • the first MI50 native-generation throughput baseline and fresh pinned llama.cpp comparison; see experiments/EXP-0093-m6b1-qwen35-native-generation-baseline.md

  • a production-selected Q5_K scale/min unpack-hoisting optimization improving native TG64/TG128 by about 27%; see experiments/EXP-0095-m6b2-q5k-scale-hoist.md

  • a production-selected Q4_K metadata-staging optimization improving native TG64/TG128 by about 38%; see experiments/EXP-0096-m6b3-q4k-metadata-staging.md

  • a production-selected subgroup-structured Q5_K dot loop improving the current native TG64/TG128 path by about 26%; see experiments/EXP-0099-m6b6-q5k-subgroup-layout.md

  • a production-selected paired-nibble Q5_K decoding improvement adding about 2.7% on TG64/TG128; see experiments/EXP-0100-m6b7-q5k-paired-nibbles.md

  • rejected Wave64-local cached-attention reduction and Q6_K LM-head index-hoisting candidates, retained as negative evidence; see experiments/EXP-0101-m6b8-attention-wave-local-reduction.md and experiments/EXP-0102-m6b9-q6k-lm-head-index-hoist.md

  • a rejected recurrent Q8_K input-reuse candidate; see experiments/EXP-0103-m6b10-recurrent-q8-reuse.md

  • a production-selected Q4_K packed-dot4 projection path improving native TG64/TG128 by about 9.6%/10.0%; see experiments/EXP-0104-m6b11-q4k-packed-dot4.md

  • a rejected Q6_K packed-dot4 projection experiment: functionally clean but end-to-end neutral; see experiments/EXP-0105-m6b12-q6k-packed-dot4.md

  • a rejected Q6_K×Q8_1 LM-head compatibility path: matching representation but 11.28% slower end-to-end; see experiments/EXP-0106-m6b13-lm-head-q8-1.md

  • a production-selected MMVQ-style Q6_K×Q8_1 LM-head path improving P64 by about 3.3%; the former Q6_K×Q8_K path remains available with MIINFER_LM_Q8_1_MMVQ=0 for control comparisons; see experiments/EXP-0107-m6b14-q6k-mmvq-q8-1.md

  • a production-selected recurrent state-update path that avoids the intermediate decayed-state store and improves TG64/TG128 by about 2.8%/2.6%; the former kernel remains available with MIINFER_DELTA_NO_DECAY_STORE=0; see experiments/EXP-0108-m6b15-recurrent-no-decay-store.md

  • a production-selected Q8_K projection-input reuse path that quantizes shared normalized inputs once for repeated consumers, improving TG64/TG128 by about 0.8%/0.7%; set MIINFER_REUSE_PROJECTION_Q8=0 for the separate-quantization control; see experiments/EXP-0109-m6b16-projection-input-q8-reuse.md

  • a production-selected gfx906 Q5_K×Q8_1 MMVQ-style recurrent output projection, improving native TG64/TG128 by about 10.9%/10.2%; the former Q5_K×Q8_K path remains available with MIINFER_Q5K_Q8_1_MMVQ=0; see experiments/EXP-0110-m6b17-q5k-mmvq-q8-1.md

  • a production-selected gfx906 Q4_K×Q8_1 MMVQ-style FFN Down projection, improving native TG64/TG128 by about 2.3%/2.2%; the former Q4_K×Q8_K path remains available with MIINFER_Q4K_Q8_1_MMVQ=0; see experiments/EXP-0111-m6b18-q4k-mmvq-q8-1-ffn-down.md

  • a production-selected shared Q4_K×Q8_1 MMVQ-style FFN Gate/Up path, improving native TG64/TG128 by about 5.9%/6.2%; the former Q4_K×Q8_K path remains available with MIINFER_Q4K_Q8_1_MMVQ_FFN_GATE_UP=0; see experiments/EXP-0112-m6b19-q4k-mmvq-ffn-gate-up.md

  • a rejected Q6_K×Q8_1 MMVQ recurrent QKV candidate: native replay and the external observable contract failed immediately, so the B19 Q6_K×Q8_K QKV path remains active; see experiments/EXP-0113-m6b20-q6k-mmvq-qkv.md

  • a production-selected Q4_K×Q8_1 MMVQ recurrent gate projection, improving native TG64/TG128 by about 1.8%/1.6%; the former Q4_K×Q8_K path remains available with MIINFER_Q4K_Q8_1_MMVQ_ATTN_GATE=0; see experiments/EXP-0114-m6b21-q4k-mmvq-attn-gate.md

  • a production-selected packed-dot4 Q6_K×Q8_K recurrent QKV projection, improving native TG64/TG128 by about 2.9%/2.9%; the scalar path remains available with MIINFER_Q6K_Q8K_DOT4_QKV=0; see experiments/EXP-0115-m6b22-q6k-q8k-dot4-qkv.md

  • a measurement-only full-attention stage attribution at P64; layer 3 is dominated by the combined Q/K/V preparation bucket, FFN, and cached attention, with no production behavior change; see experiments/EXP-0116-m6b24-full-attention-attribution.md

  • a measurement-only fine attribution that separates full-attention Q projection (~0.199 ms), Q head normalization (~0.085 ms), K/V projections, RoPE/KV store, and cached attention; see experiments/EXP-0117-m6b25-full-attention-fine-attribution.md

  • a rejected Q4_K×Q8_1 MMVQ full-attention Q-projection candidate: it failed the external observable contract immediately, so the Q4_K×Q8_K path remains active; see experiments/EXP-0118-m6b26-q-projection-q8-1-reject.md

  • a production-selected batched full-attention head RMS-normalization path, improving stable-peak TG64/TG128 by about 1.4%/1.4%; set MIINFER_BATCH_HEAD_RMS=0 for the separate-launch control; see experiments/EXP-0119-m6b27-batched-head-rms.md

  • a measurement-only post-B27 profile showing 48 recurrent layers consume 64.54 ms of the instrumented 88.99 ms token and remain the next measured target; see experiments/EXP-0120-m6b28-post-b27-profile.md

  • a measurement-only recurrent stage attribution across layers 0–2, showing a stable recurring cost structure and identifying state update as the next uncleared recurrent-specific target; see experiments/EXP-0121-m6b29-recurrent-stage-attribution.md

  • a production-selected transposed DeltaNet recurrent-state layout, preserving the logical state contract while improving native TG64/TG128 by about 3.4%/3.6%; set MIINFER_DELTA_TRANSPOSED_STATE=0 for the former layout; see experiments/EXP-0122-m6b30-transposed-deltanet-state.md

  • a rejected two-row recurrent FFN Gate/Up MMVQ geometry: it improved sampled TG64/TG128 by about 1.45%/1.49% but introduced an external-contract P6 decision change and was removed; see experiments/EXP-0123-m6b31-ffn-gate-up-two-row.md

  • a production-selected transposed recurrent no-decay-store kernel: it preserves the accepted external contract and improves stable-peak TG64/TG128 by about 1.71%/1.63%; set MIINFER_DELTA_TRANSPOSED_NO_DECAY_STORE=0 for the B30 control; see experiments/EXP-0124-m6b32-transposed-no-decay-store.md

  • a measurement-only post-B32 profile: total GPU time is about 84.6–84.7 ms/token, with recurrent FFN Gate/Up plus Down remaining the largest repeated family; see experiments/EXP-0125-m6b33-post-b32-profile.md

  • a rejected fused SiLU-to-Q8_1 candidate: it passed the external observable contract but improved TG64/TG128 by only 0.09%/0.08%, so the candidate was removed; see experiments/EXP-0126-m6b34-fused-silu-q8-1.md

  • a production-selected Q4_K×Q8_1 LDS activation-reuse path: it preserves the external contract and improves stable-peak TG64/TG128 by about 2.98%/3.01%; set MIINFER_Q4K_Q8_1_LDS_INPUT=0 for the B32 control; see experiments/EXP-0127-m6b35-q4k-q8-1-lds-input.md

  • a measurement-only post-B35 profile: total GPU work is 82.12 ms/token and long-K Q4_K×Q8_1 FFN Down remains the largest repeated projection; see experiments/EXP-0128-m6b36-post-b35-profile.md

  • a reproducible llama.cpp-backed Qwen3.8-27B hybrid tensor/state fixture; see experiments/EXP-0043-m6a1-qwen38-reference-fixture.md

  • a read-only Qwen3.8 projection/kernel compatibility map; see experiments/EXP-0044-m6a2-qwen38-projection-compatibility.md

  • an explicit M6-B1 readiness audit showing that the current qwen3 HIP path cannot profile qwen35; see experiments/EXP-0051-m6b1-qwen38-miinfer-gpu-readiness.md

  • L29 recurrent/gated provenance diagnostics through A26.8; see experiments/EXP-0076-m6a266-qwen35-l29-output-provenance.md, experiments/EXP-0077-m6a267-qwen35-l29-gated-output-provenance.md, and experiments/EXP-0078-m6a268-qwen35-l29-gate-input-provenance.md

  • a qwen35 model boundary and real MI50 RMSNorm fixture; see experiments/EXP-0052-m6a8-qwen35-gpu-foundation.md

  • a qwen35 Q6_K×Q8_K LM-head projection validated against external logits; see experiments/EXP-0053-m6a9-qwen35-lm-head.md

  • a qwen35 Q4_K×Q8_K attention projection validated against an external checkpoint; see experiments/EXP-0054-m6a10-qwen35-q4k-projection.md

  • a composed qwen35 layer-3 RMSNorm→Q8_K→Q4_K×Q8_K attention prefix; see experiments/EXP-0055-m6a11-qwen35-attention-prefix.md

  • qwen35 layer-3 Q/K/V projections with K normalization validated on MI50; see experiments/EXP-0056-m6a12-qwen35-attention-projections.md

  • a complete qwen35 layer-3 full-attention path through FFN and residual, checked at positions 0–8; see experiments/EXP-0057-m6a13-qwen35-full-attention-layer.md

  • qwen35 state fingerprints, poisoned reset, and replay checks for the recurrent layers and layer-3 KV cache; see experiments/EXP-0058-m6a14-qwen35-state-audit.md

  • qwen35 layers 0–3 stateful hybrid-block composition through the complete layer-3 output boundary at positions 0–16; see experiments/EXP-0059-m6a15-qwen35-hybrid-block-audit.md

  • qwen35 layers 4–7 independently composed after layers 0–3 through position 16 with recurrent/KV fingerprints; see experiments/EXP-0060-m6a16-qwen35-hybrid-block-4-7-audit.md

  • qwen35 composition ladder through 8, 16, 32, and 64 layers with linear one-token scaling and final-logit validation; see experiments/EXP-0061-m6a17-qwen35-composition-ladder.md

  • a real MI50 qwen35 DeltaNet GPU state-update core with persistent [48,128,128] state validated across positions 0→1; see experiments/EXP-0062-m6a18-qwen35-deltanet-state-gpu.md

  • a real gfx906 qwen35 four-tap convolution, SiLU, Q/K/V split, and Q/K normalization path with persistent circular history; see experiments/EXP-0063-m6a19-qwen35-conv-gpu.md

  • a complete layer-0 qwen35 recurrent GPU path through FFN and residual, including recurrent projections and beta/alpha preparation; see experiments/EXP-0064-m6a20-qwen35-recurrent-layer-gpu.md

  • qwen35 GPU layers 0–3 composed through the complete full-attention layer-3 boundary at positions 0→1; see experiments/EXP-0065-m6a21-qwen35-gpu-hybrid-block.md

  • qwen35 GPU layers 0–3 audited through positions 0, 1, 2, 4, 8, 16, 32, and 64 with bounded external-reference error and state fingerprints; see experiments/EXP-0066-m6a22-qwen35-gpu-hybrid-position-audit.md

  • qwen35 GPU layers 4–7 composed from the actual L0–L3 output through P64 using independent recurrent/KV state; see experiments/EXP-0067-m6a23-qwen35-gpu-hybrid-block-4-7.md

  • qwen35 GPU layers 0–7 composed through one common stateful executor through P64 with poisoned reset/replay and cached-attention determinism coverage; see experiments/EXP-0068-m6a24-qwen35-eight-layer-gpu-prefix.md

  • qwen35 GPU layers 0–15 composed through the same executor through P64 with later recurrent/KV state validation; see experiments/EXP-0069-m6a25-qwen35-sixteen-layer-gpu-prefix.md

  • qwen35 GPU layers 0–31 composed through P64; all layer outputs pass, while L30 recurrent-state checkpoints remain in retest; see experiments/EXP-0070-m6a26-qwen35-thirty-two-layer-gpu-prefix.md and experiments/EXP-0071-m6a261-qwen35-l30-state-localization.md

  • L30 recurrent-update provenance localizes the tracked P63→P64 discrepancy to the external-versus-GPU recurrence result; a full P1–P64 scan also finds a larger P20 outlier; see experiments/EXP-0072-m6a262-qwen35-l30-update-provenance.md

  • external-input replay clears the MIInfer L30 recurrence itself at the P19→P20 worst-case transition; A26 still awaits an explicit state contract; see experiments/EXP-0073-m6a263-qwen35-recurrent-contract-adjudication.md

  • production operand substitution identifies L30 k_in as the dominant upstream discrepancy at P19→P20; see experiments/EXP-0074-m6a264-qwen35-l30-production-operand-attribution.md

  • L30 K-path provenance finds the first mismatch at its input (l_out-29), while the K path remains bounded through convolution/SiLU and normalization; see experiments/EXP-0075-m6a265-qwen35-l30-k-path-provenance.md

Sampling and serving remain out of scope. M5 is closed as a measured local optimization campaign. M6-A bring-up is complete for the native qwen35 GPU generation path. M6-B1 now has a first performance baseline; the current benchmark harness is functional but not yet workload-equivalent enough for a final parity claim.

The previous Qwen3-8B production path is approximately 55 tok/s at stable peak. The current Qwen3.8-27B native GPU path measures approximately 11.72 tok/s for TG128 at stable peak after the B30 recurrent-state layout change; the pinned llama.cpp parity target remains higher.

Initial target

MIInfer deliberately begins with a narrow target.

Area Initial target
GPU AMD Instinct MI50 32GB
Architecture Vega 20 / gfx906
Execution Single GPU
Platform Linux
Workload LLM inference
Batch Batch 1 / low batch
Models Explicitly selected model family
Quantization Explicitly selected format
Primary language C++20
GPU programming HIP
Portability Not an initial goal

Support for other GPUs, operating systems, model architectures, or execution modes must be justified by project goals and benchmark evidence.

Why MIInfer?

Modern inference frameworks need to support:

  • many GPU generations
  • many model architectures
  • many quantization formats
  • dynamic execution graphs
  • different batch sizes
  • different serving workloads
  • multiple accelerator vendors
  • distributed execution

Those capabilities are valuable, but they also impose architectural constraints.

gfx906 is now an older and poorly supported architecture in modern ROCm software, yet hardware such as the MI50 still provides:

  • 32 GB HBM2
  • approximately 1 TB/s memory bandwidth
  • Wave64 execution
  • useful packed integer instructions
  • substantial FP16/FP32 compute capability

MIInfer investigates what becomes possible when we remove most generic-runtime requirements and optimize for the hardware directly.

Core hypothesis

MIInfer is an engineering experiment built around two competing hypotheses.

H0

A purpose-built gfx906 runtime cannot materially outperform an already well-optimized llama.cpp/gfx906 implementation.

H1

Removing general-purpose runtime constraints enables meaningful inference improvements on MI50/gfx906.

The project must remain structured so this hypothesis can be tested objectively.

If specialization does not produce meaningful benefits, that is a valid project result.

Design principles

Specialization is intentional

MIInfer may rely on known properties of:

  • gfx906
  • Wave64
  • MI50 memory characteristics
  • supported model dimensions
  • supported quantization formats
  • fixed tensor layouts

Genericity is not automatically desirable.

Measure before optimizing

No optimization is accepted because it:

  • looks theoretically faster
  • worked on another GPU
  • worked in another gfx906 project
  • uses a lower-level ISA instruction
  • reduces instruction count
  • increases theoretical occupancy

Every meaningful optimization must be measured against a reproducible baseline.

Correctness before performance

A faster kernel that changes model behavior incorrectly is a regression.

Kernel work is validated progressively through:

  1. numerical reference comparisons
  2. tensor or logits comparisons where practical
  3. short generation tests
  4. long generation and long-context tests

Negative results are retained

Failed performance experiments are part of the project knowledge base.

An optimization that loses performance should normally be documented rather than silently discarded.

This helps prevent repeated work and makes architectural decisions evidence-based.

Hot paths should be static

Where possible, decisions should happen during model loading rather than during every generated token.

The intended direction is:

model
  ↓
validate supported configuration
  ↓
select kernels
  ↓
construct static execution plan
  ↓
allocate memory
  ↓
load / repack weights
  ↓
execute

rather than:

model
  ↓
generic execution graph
  ↓
dynamic operator selection
  ↓
generic scheduler
  ↓
generic accelerator backend

What MIInfer intends to own

The project intends to directly control performance-critical architecture such as:

  • gfx906 GPU primitives
  • quantized matrix/vector kernels
  • matrix multiplication kernels
  • model-specific kernel selection
  • tensor packing and memory layout
  • activation reuse
  • GPU buffer lifetime planning
  • static execution planning
  • attention implementation where justified
  • MoE execution where relevant
  • HIP graph capture strategy
  • benchmark methodology

Commodity infrastructure may be reused when doing so does not compromise the experiment.

What MIInfer does not initially aim to build

Initial non-goals include:

  • NVIDIA CUDA support
  • Intel GPU support
  • generic AMD GPU support
  • RDNA support
  • MI200 / MI300 support
  • CPU inference optimization
  • Windows support
  • macOS support
  • training
  • fine-tuning
  • distributed inference
  • tensor parallelism
  • pipeline parallelism
  • high-concurrency serving
  • arbitrary model support
  • arbitrary quantization support
  • OpenAI-compatible serving
  • multimodal inference

These may be reconsidered only after the initial gfx906 hypothesis has been evaluated.

Architecture direction

The planned architecture is deliberately small.

                    MIInfer

                     Model
                       │
                       ▼
                 Model Loader
                       │
                       ▼
              Supported-Model Check
                       │
                       ▼
             Static Execution Planner
                       │
          ┌────────────┴────────────┐
          ▼                         ▼
     Memory Plan               Kernel Plan
          │                         │
          └────────────┬────────────┘
                       ▼
                 gfx906 Runtime
                       │
                       ▼
               gfx906 Kernels
                       │
                       ▼
                AMD Instinct MI50

MIInfer does not intend to adopt GGML's generic execution graph or scheduler as its runtime architecture.

Existing frameworks remain important reference implementations and benchmark competitors.

Research influences

MIInfer is informed by several existing projects.

llama.cpp

Used as:

  • correctness reference
  • model-format reference
  • benchmark baseline
  • source of implementation knowledge

MIInfer is not intended to become a llama.cpp fork.

gfx906 llama.cpp forks

Projects specializing llama.cpp for Vega20/gfx906 provide valuable evidence around:

  • Wave64-specific execution
  • DPP and swizzle operations
  • quantized GEMV
  • weight repacking
  • attention
  • MoE
  • HIP graph behavior
  • architecture-specific tuning
  • failed optimization paths

Ideas from these projects must still be independently benchmarked against MIInfer workloads.

NInfer

Provides an important architectural lesson:

Selected hardware and selected models allow much deeper specialization than generic inference frameworks.

MIInfer applies a similar philosophy to gfx906 rather than NVIDIA hardware.

gfx906 vLLM work

Provides useful information about:

  • modern-model compatibility on gfx906
  • numerical precision problems
  • Triton experimentation
  • attention behavior
  • MoE bottlenecks
  • unsupported ROCm paths

See docs/references.md for the research inventory.

Development roadmap

M0 — Baseline

Establish:

  • reproducible MI50 environment
  • strongest practical gfx906 reference implementation
  • target model
  • benchmark methodology
  • hardware-state capture
  • correctness reference

M1 — Kernel laboratory

Build:

  • HIP benchmark infrastructure
  • CPU reference implementations
  • numerical validation
  • initial GEMV/GEMM experiments

M2 — Prove specialization

Investigate:

  • Q4/Q8 and related packed dot products
  • Wave64 and half-wave execution
  • DPP/swizzle operations
  • architecture-specific memory layouts
  • register/LDS trade-offs
  • static kernel configuration

This is the first major go/no-go milestone.

M3 — Minimal runtime

Only after sufficient kernel evidence:

  • model metadata
  • tensor loading
  • GPU memory planning
  • supported-model validation
  • static execution planning

M4 — First correct generation

Execute one supported model end-to-end.

M5 — Beat the reference

Compare MIInfer against the strongest reproducible gfx906 baseline.

Primary metrics include:

  • prompt processing throughput
  • token generation throughput
  • time to first token
  • VRAM
  • context capacity
  • power
  • tokens per joule

M6 — Runtime specialization

Potential work:

  • native packed model artifacts
  • activation reuse
  • operation fusion
  • fixed buffer reuse
  • HIP graph capture
  • static decode replay

M7 — Expansion

Only after the original hypothesis is demonstrated:

  • additional quantization formats
  • second model
  • deeper long-context optimization
  • speculative decoding / MTP
  • optional serving layer

See docs/roadmap.md for the current roadmap.

Benchmark philosophy

Performance measurements are treated as engineering evidence.

Important comparisons should include:

  • exact baseline commit
  • exact candidate commit
  • GPU state
  • ROCm version
  • compiler version
  • model and quantization
  • context length
  • workload shape
  • repeated runs
  • correctness verification

For small performance differences, interleaved testing is preferred:

A
B
A
B
A
B

rather than running all baseline tests followed by all candidate tests.

Hardware clocks, temperature, and power state must be considered part of benchmark validity.

See docs/benchmarking.md.

Experiments

Performance investigations are recorded under:

experiments/

Each significant experiment receives an identifier:

EXP-0001
EXP-0002
EXP-0003
...

Experiments should document:

  • hypothesis
  • bottleneck
  • baseline
  • candidate
  • environment
  • correctness
  • raw measurements
  • aggregated measurements
  • profiling evidence
  • interpretation
  • KEEP / REJECT / RETEST decision

Negative experiments are intentionally retained.

Building

The build system is CMake.

The intended canonical development presets are:

cmake --preset mi50-debug
cmake --build --preset mi50-debug

and:

cmake --preset mi50-release
cmake --build --preset mi50-release

Tests:

ctest --preset mi50-debug

The initial benchmark validates HIP execution and timing infrastructure:

./build/mi50-release/miinfer-bench --warmup 5 --iterations 100
scripts/run-bench.sh ./build/mi50-release/miinfer-bench \
  --warmup 5 --iterations 100

The benchmark emits JSON and uses HIP events. The runner stores before/after environment captures, active-run rocm-smi telemetry, and the JSON result under bench/results/<run-id>/.

The project is currently being bootstrapped. The initial benchmark validates infrastructure rather than MIInfer inference performance.

Hardware requirements

The initial supported target is:

AMD Instinct MI50 32GB
gfx906
Linux
ROCm/HIP toolchain capable of producing gfx906 code

Current ROCm releases may not provide complete official gfx906 support.

The exact validated development stack will be documented in:

docs/hardware.md

Do not assume that every modern ROCm library or prebuilt binary contains working gfx906 support.

Repository structure

The intended repository layout is:

MIInfer/
├── AGENTS.md
├── README.md
├── CMakeLists.txt
├── CMakePresets.json
│
├── include/
│   └── miinfer/
│
├── src/
│
├── gfx906/
│   ├── primitives/
│   └── kernels/
│
├── bench/
├── tests/
├── experiments/
├── scripts/
│
└── docs/
    ├── architecture.md
    ├── benchmarking.md
    ├── current-state.md
    ├── decisions.md
    ├── hardware.md
    ├── references.md
    └── roadmap.md

The structure will evolve only when implementation evidence requires it.

Contributing

MIInfer is currently in an early research phase.

Before modifying performance-critical code, read:

Performance changes should remain:

  • focused
  • measurable
  • reproducible
  • correctness-verified
  • easy to compare against a baseline

Do not broaden supported hardware or model scope without an explicit project decision.

License

MIInfer is licensed under the MIT License.

See LICENSE.

External code incorporated into the project must retain all attribution and licensing required by its original license.

Guiding principle

MIInfer favors:

simple, explicit, architecture-aware, measured code

over unnecessary generality.

But specialization without measurement is not optimization.

The project follows this loop:

UNDERSTAND
    ↓
MEASURE
    ↓
HYPOTHESIZE
    ↓
IMPLEMENT
    ↓
VERIFY CORRECTNESS
    ↓
BENCHMARK
    ↓
KEEP OR REJECT

The goal is not merely to make gfx906 run modern LLMs.

The goal is to determine how fast gfx906 can run them when the software is designed around the hardware.

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages