A fork of Neroued/ninfer adding rk4v4-e8, a packed
4-bit KV cache using an E8 Conway–Sloane lattice codec, ported to SM120 / RTX 5090.
It roughly doubles the usable context of Qwen3.8-27B on a single 32 GB card, at identical decode
speed. Upstream's own documentation is preserved at README-upstream.md.
Validated: 649,802 tokens, 5/5 needle retrieval, on one RTX 5090.
The KV cache dominates memory at long context. At 8-bit it costs ~35.2 KB per token, so a 32 GB card holding 19 GB of weights runs out at roughly 330K tokens.
rk4v4-e8 stores K and V as packed 4-bit codes — two codes per byte, so each plane is half
width (head_dim 256 → extent 128, U8). Codes are assigned by projecting each rotated 8-dimensional
sub-vector onto the E8 lattice, the densest known 8-dimensional sphere packing, which gives
lower quantization error for a given bit budget than independent per-scalar rounding.
The codec and the packed attention kernels come from the
sergiuszm/ninfer-4090 fork, which implemented them for
sm_89. This repository ports that work to sm_120 and validates it end to end.
Two E8 modes exist upstream. Only one is shipped here:
| mode | K | V | codec | status |
|---|---|---|---|---|
rk4v4-e8 |
4-bit | 4-bit | E8 lattice (half-coset) | released and validated |
rk2v4-e8 |
2-bit | 4-bit | E8 root (exact) | default-deny, not ported |
rk2v4-e8 refuses to start by name. See Why rk2 was not shipped.
All figures measured on one RTX 5090 (32,607 MiB), Qwen3.8-27B NVFP4, weights 18.98 GiB.
Retrieval is 5 needles at 10/25/50/75/90% depth, each scored individually, finish=stop required.
| configuration | max validated context | retrieval | KV bytes/token |
|---|---|---|---|
| int8 | 329,804 | 5/5 | 35,219.6 |
rk4v4-e8 factor 2 |
499,828 | 5/5 | 17,979.7 |
rk4v4-e8 factor 4 |
600,078 | 5/5 | 17,979.7 |
rk4v4-e8 factor 4, graphs off |
649,802 | 5/5 | 17,979.7 |
1.97× the validated context of int8, at 51% of the KV cost per token.
| context | int8 | rk4v4-e8 |
saving |
|---|---|---|---|
| 262,144 | 30,331 MiB | 26,262 MiB | 4,069 MiB |
| 329,804 (int8 ceiling) | 31,776 MiB | — | — |
| 649,802 (rk4 ceiling) | not reachable | 32,025 MiB | — |
Taken from the engine's own KV capacity line, not inferred from nvidia-smi:
524,288 tokens runtime=8.98 GiB free-after-weights=11.23 GiB
610,048 tokens runtime=10.37 GiB free-after-startup=754.64 MiB
660,032 tokens runtime=11.17 GiB free-after-startup=0.00 MiB, slack=62.28 MiB
Predicted 8,991 MiB at 524,288 against 9,195 measured — 2.3% agreement between the reservation arithmetic and the actual allocation.
| int8 | rk4v4-e8 (factor 4) |
|
|---|---|---|
| largest context that starts | 340,000 | 650,048 |
| smallest that is refused | 350,000 | 700,000 |
Matched benchmark. max_context=98304, prompt 89,880 tokens, 300 generated, greedy, thinking off,
prefix reuse disabled, CUDA graphs on for both, 3 repeats, prompt head varied per run so nothing
is served from cache. Only --kv-dtype differs. No cross-tier comparison.
| int8 | rk4v4-e8 |
|
|---|---|---|
| decode tok/s (median of 3) | 59.68 | 60.03 |
| prefill tok/s (median of 3) | 4,908.3 | 3,934.0 |
Decode is at parity — 0.6% apart, well inside run-to-run spread. Prefill costs about 20%, which is the packing and unpacking work on the write path.
Decode is where the tradeoff lands well: long-context work is dominated by how long you wait for the first token and then by tokens per second, and rk4 gives up nothing on the latter.
One caution on reading these numbers: int8's decode series contains occasional outliers (a 101.49
against its own 59.68 and 59.62 in the same run). rk4v4-e8 was in fact the more stable mode, 1.8%
spread against int8's 70%. Single-sample comparisons of these two modes are unreliable; use medians.
Measured, factor 2 unless noted. Prefill degrades superlinearly, and this — not memory — is what limits practical use at the top of the range.
| context | prefill tok/s | wall time | decode tok/s |
|---|---|---|---|
| 254,817 | 1,794 | 2.4 min | 61.7 |
| 499,828 | 995 | 8.4 min | 39.9 |
| 600,078 (f4) | 839 | 11.9 min | 33.4 |
| 649,802 (f4, no graphs) | 611 | 17.7 min | 29.4 |
Requires a 64-bit Linux host, an RTX 5090, and a CUDA 13.1-compatible driver. The build is
hard-gated to sm_120a and will refuse any other architecture.
The e8-sm120 branch lives in this repository, not upstream. Cloning
Neroued/ninfer and checking out e8-sm120 will fail with
pathspec 'e8-sm120' did not match any file(s) known to git — upstream does not carry it.
git clone -b e8-sm120 https://github.com/splickz/ninfer-rk4v4-e8.git
cd ninferSubstitute the URL of the repository you are reading this in.
nvcc must be on PATH before configuring. Without it CMake fails with a message naming
CUDACXX/CMAKE_CUDA_COMPILER rather than saying CUDA is missing:
export PATH=/usr/local/cuda/bin:$PATH
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallelProduces build/apps/ninfer and build/apps/ninfer-serve.
./build/apps/ninfer-serve /path/to/qwen3_8_27b_nvfp4.ninfer \
--kv-dtype rk4v4-e8 --rope-yarn-factor 2 --rope-original-max-position 262144 \
--max-context 262144Validated exactly as written: 254,817 tokens, 5/5 retrieval, decode 61.7 tok/s, prefill 1,794 tok/s, 25,765 / 32,607 MiB leaving 6.8 GB headroom. Decode matches the 98K matched benchmark, so this context costs nothing in decode speed. Factor 2 is preferred over 4 because it is the least aggressive RoPE scaling that covers the range.
./build/apps/ninfer-serve /path/to/qwen3_8_27b_nvfp4.ninfer \
--kv-dtype rk4v4-e8 --rope-yarn-factor 4 --rope-original-max-position 262144 \
--max-context 610000600,078 tokens validated, 5/5, CUDA graphs on, ~12 min prefill.
This configuration fails to start roughly one attempt in three — measured 2 successful starts in 3 attempts — because CUDA graph preparation exceeds its fixed 12 MiB allowance. It is not a memory problem, and a retry usually succeeds. For deterministic startup add
--no-cuda-graph, at some cost to decode speed.
./build/apps/ninfer-serve /path/to/qwen3_8_27b_nvfp4.ninfer \
--kv-dtype rk4v4-e8 --rope-yarn-factor 4 --rope-original-max-position 262144 \
--max-context 660000 --no-cuda-graph649,802 tokens validated, 5/5. Leaves 582 MiB spare and 62 MiB of planner slack, and prefill is ~18 minutes. It works, but it is fragile and not recommended for routine use.
int8 remains slightly better for prefill-latency-bound work below 330K: 20% faster prefill, identical decode. Above 330K it cannot run at all.
Nothing below counts a configuration as validated merely because the server started.
Needle retrieval. Five needles at 10/25/50/75/90% depth in the prompt, with the question asked
from the very end, so every answer requires a genuine long-range lookup. A needle at the end of a
600K prompt would be a 5K retrieval test in a 600K costume. Each needle is scored individually and
finish=stop is required — a truncated answer is not a pass.
No prefix-reuse contamination. Prefix caching is disabled for benchmarking, and prompt heads are
varied per repeat. An earlier version of the performance run was discarded when runs 2 and 3 showed
ttft=0.13s: prefill had been skipped entirely and the resulting decode figure was a cache artifact.
Generation sanity. Factual recall, exact arithmetic (17 × 23 = 391), code generation, and an explanation requiring correct physics, all against an int8 control run through the identical harness. The control matters: the first generation run showed empty responses that looked like a codec defect and were in fact thinking-tokens consuming the budget — int8 did the same thing.
Memory validation. Three independent layers, required to agree: the reservation arithmetic, the
engine's own reported KV allocation, and nvidia-smi peak. Reservation and observation agree within
2.3% at 524,288. Passing the reservation check alone was explicitly not accepted as proof.
Packed-path verification. The allocator's selected plane geometry is asserted, not assumed:
packed_k=1 packed_v=1 e8_lattice=1 e8_root=0, extents 256 → 128, dtype U8. This matters because
an earlier bug allocated full-width int8 planes while every mode flag looked correct, and the
configuration would have passed retrieval perfectly while measuring nothing.
CUDA graph preparation draws on a fixed 12 MiB allowance with no flag to raise it, and its demand is not monotonic in context:
int8 300,000 fails (92.5 MB) 320,000 / 330,000 / 340,000 start
rk4 f4 550,000 fails (94.0 MB) 600,000 starts
650,000 fails (18.3 MB) 650,048 starts
Measured retry behaviour: --max-context 610000 with graphs on started on 2 of 3 attempts.
A context that fails this way is not at a memory limit. Shift the context slightly or pass
--no-cuda-graph. The 649,802 result required graphs off; graphs-on and graphs-off are recorded as
separate measurements throughout, never merged.
rk2v4-e8 reduces K to 2 bits, projecting ~13,883.6 bytes/token and ~841K context. It was declined
on evidence:
rk4v4-e8is memory-limited, not quality-limited — 5/5 at every tier from 32K to 649,802, with no quality wall found below the memory frontier.- Prefill degrades superlinearly: 2.4 min at 254K, 17.7 min at 649K. rk2's additional ~190K tokens land where a single request would cost 25–30 minutes.
- More KV compression buys memory, and memory is no longer the binding constraint. Additional context makes the actual constraint — prefill time — worse.
- It would cost a 733-line codec port plus an unmeasured quality risk from 2-bit K.
The mode remains wired through the allocator and refuses at startup by name, so the work is recoverable if the tradeoff ever changes.
| document | contents |
|---|---|
docs/e8/DESIGN.md |
packed planes, bytes/token derivation, the allocator chain |
docs/e8/BENCHMARKS.md |
full benchmark tables and methodology |
docs/e8/FINAL-EVIDENCE.md |
every measurement, measured vs projected |
docs/e8/DEBUGGING-STORY.md |
the four root causes and how each was found |
docs/e8/artifacts/ |
raw logs for every result quoted here |
docs/e8/harness/ |
the harness scripts, runnable as-is |
- Neroued/ninfer — the engine.
- sergiuszm/ninfer-4090 — the rk4v4 packed KV path and
the E8 lattice codec, implemented for sm_89. This fork ports commit
5df406b0to sm_120 and contributes the SM120 validation, the YaRN long-context path, and a fix to a shared-memory overrun that limited context in the decode kernels.