Durable local inference for coding agents — the qualified appliance for Oh My Pi. It runs Qwen3.8 27B through the NInfer engine on one NVIDIA RTX 5090, 4090, or 3090 and preserves explicit OpenAI Responses continuation state across process restarts — a durable primitive, not a lucky prefix-cache hit. Qualified on all three lanes: measured, hash-pinned, fail-closed.

Get started → · Download v0.6.9

Lanes · Facts · Compare · Benchmarks · Architecture · Performance · Security · Roadmap · Changelog

_{Private by design: loopback-only endpoints · bearer-authenticated · fail-closed instead of cloud fallback · every byte hash-pinned}

_{A real recorded session on the released RTX 5090 runtime — bug found and fixed, tests rerun to green, then a follow-up turn continues from retained GPU state instead of re-sending the transcript.}

_{MP4 · poster · provenance and checksums}

What changes for you	Released evidence
Retained state outlives the turn — and the process	Warm follow-up 1.790 s vs 47.920 s cold at a 109,594-token session, and 0.778 s first token after a docker restart from the durable checkpoint — the current measurement receipts
Agent branches share the base, not re-prefill it	Four subagent branches from one 67.7K-token base: 148.7 s → 3.84 s on v0.4.4, 0.40 s to first token when the anchor is device-resident
Interactive output is fast	136.03 tok/s decode on the qualified RTX 5090 profile (41.20% MTP acceptance at temperature 0 on the technical-writing gate)
Long coding sessions fit	Exact retrieval at a 130,048-token prompt; 131,072-token ceiling
The route does not escape to cloud	Loopback-only, bearer-authenticated, fail-closed; acceptance-tested
Checkpoints don't stall the session	Export runs off the engine lock on v0.4.4: warm follow-up during checkpoint traffic 15.26 s → 0.91 s, explicit 5.19 GB save 31.6 s → 13.8 s

The warm/cold pair is server-side, one sample per point, bound into the released qualification chain on the exact published bytes — not a universal latency claim. Method and receipts.

Use OMP NInfer when: you use OMP, own a qualified card, want Qwen3.8, and care about private, long-lived coding sessions.

Use something else when: you want a broad model catalog, an unsupported GPU, multi-user serving, or generic OpenAI-compatible inference.

Important

v0.6.9 is the current public release. If you own a qualified card, the v0.6.9 quickstart is the supported onboarding: three GPU lanes with public install authority — the RTX 5090 durable container plus native Windows RTX 4090 and RTX 3090 — each bound to exact bytes and a qualification receipt. The 0.x series carries an explicit support boundary: the latest published release and its exact manifest/profile. Details: release status · compatibility matrix. Both parser runtime components passed lane qualification and published-component acceptance: RTX 5090 documented host 2/2 and macOS 10/10 blocks; RTX 4090 public-URL already-installed acceptance. Composed receipt.

What this is — and isn't

A live ninfer or llama.cpp process with prefix caching already avoids recomputing an append-only prefix — the engine this project ships does it too (prefix_reuse is enabled in every shipped profile). If a warm in-process cache on a live server is all you need, that works today without OMP NInfer.

What this project adds is continuation as an explicit, transactional, durable primitive rather than an implicit longest-prefix match. OMP tracks the Responses lineage end to end — previous_response_id, forks, rollback — and as of v0.4.0 the continuation (session state plus its required KV) is checkpointed to disk and restored after process death on all three lanes: 109,589 tokens restored across a docker restart on the 5090, 102,075 tokens restored on the 4090, 310 MB checkpoint restore on the 3090. A prefix cache cannot outlive its process; a checkpoint can.

Checkpoints already leave the machine. The shipped sync tool exports verified generations, copies them to another host or NAS, and imports them back to local storage for restore. Recovery after loss of local state was exercised on all three lanes; host-to-host transport and NAS replication have separate receipts. Restore requires the same runtime binary, model, profile, and session credentials: copying a checkpoint to a 4090 does not make a 5090 session runnable there. Network shares are replica storage, never the live checkpoint root. Scope, usage, and evidence.

Why this exists

Serious OMP coding sessions run long: 100K-token transcripts, thinking, tool calls, images. Routed to a cloud provider, every one of those tokens is metered and every file leaves your machine. Routed to a typical local OpenAI-compatible server, the API is stateless — each turn re-sends the whole transcript, and while a live server's prefix cache usually avoids recomputing an append-only prefix, that reuse is an implicit longest-prefix guess that dies with the process.

OMP NInfer ships the third option as a small set of qualified lanes — OMP, the NInfer engine, and one pinned Qwen3.8 27B artifact on an RTX 5090, RTX 4090, or RTX 3090 — with three properties the exact qualified releases do not give you together elsewhere:

1. Continuation is explicit and durable, not guessed. OMP drives NInfer through stateful OpenAI Responses (previous_response_id): continuation is addressed by transactional lineage — forks and rollback qualified — instead of inferred by longest-prefix matching, and on all three lanes the continuation is checkpointed and survives process restarts. OMP commits its transcript before advancing provider state, so losing retained state degrades to a replay, never a broken session.
2. Private and fail-closed. Both endpoints bind loopback only; the route is bearer-authenticated; the shipped OMP configuration disables model fallback. When your GPU is unreachable, the turn fails with an error — it is never silently answered by a cloud model. That behavior is part of the acceptance suite, not a promise.
3. Exact and verifiable. The model is pinned by SHA-256, the runtime image by OCI digest with an SPDX SBOM, the client by checksum, and one release manifest binds them all. python3 scripts/verify_release.py --require-ready proves your clone is the qualified release.

Measured, not estimated

Historical v0.6.8 profiles and receipts in qualification.json:

Gate	Result
RTX 5090 decode	139.78 tok/s server-side over 2,048 tokens at temperature 0 on the lifecycle-started v0.6.4 candidate (134.80 tok/s wall); MTP3, 41.20% acceptance, 2.24 tokens per round
RTX 5090 prefill	2,178.80 tok/s at 130,048 tokens, exact retrieval, cold process, on the v0.6.4 candidate (2,177.70 tok/s again on the published image through the documented tunnel)
RTX 5090 fanout	4/4 sibling forks on the base anchor at 57,853 and 67,681 tokens (medians 1.41 s and 1.82 s), and 4/4 again after a verified restart (resume 3.45 / 3.65 s, forks ~1.4 s) — a 5.2 GB session restores in 3.6-3.8 s and a flipped payload byte is refused
Warm vs cold follow-up	1.790 s vs 47.920 s at a 109,594-token session, and 0.778 s first token after a process restart — v0.4.0 qualification, server-side, one sample per point
RTX 3090 native	90.66 tok/s decode, 93.43% MTP3 acceptance, exact 130,048-token retrieval at the 131,072 ceiling, 310 MB durable restart, durable v0.2.5-beta.1 train with origin-authenticated checkpoints, 300.2 W observed peak
RTX 4090 native	exact 130,048-token retrieval in 91.5 s; 153.4 tok/s decode at 87.6% MTP3 acceptance and 2,114.1 tok/s prefill on the C1 gate (a trajectory-sensitive fixture, EXP-037); 15/15 protocol checks at the shipped pool and again at a third of it; a never-published 45-token session and an explicitly saved one both restored across a graceful managed restart; exact OMP Golden-equivalent (mainline runtime v0.6.2-beta.1, sm_89, the same source as the 5090's v0.6.4)
Serving contract	OpenAI, Anthropic, and Responses protocols; tools; authenticated identity

The v0.6.9 release moves both mainline lanes to source 696e78c7 for the independently implemented Qwen tool-parser semantic port, without rebasing the serve adapters. The RTX 5090 lifecycle candidate measured exact 130,048-token retrieval at 2,193.3 tok/s and 2,048-token decode at 134.87 tok/s wall; the RTX 4090 candidate retrieved exactly in 91.2377 s and decoded at 153.464 tok/s on C1. These are candidate measurements, not public-route acceptance. Parser changes · Measurement scope and receipts.

Durable session checkpoints ship on both native Windows lanes — DirectStorage-backed — so on the RTX 4090 and RTX 3090 a follow-up continues from restored state even across a process restart, each bound by its own receipt. The RTX 5090 container keeps live-process warm continuation; as of v0.4.0 a process restart restores the session from its durable checkpoint (109,589 tokens hot in the qualification), with OMP transcript replay as the fallback when no checkpoint exists.

The shipped artifact holds its capability through quantization — 96.67% AIME 2025/2026 and 87.37% GPQA-Diamond in the upstream single-sample evaluation campaign. Numbers, methodology, caveats, and the community leaderboard: Benchmarks. These are measurements of one recorded machine and profile, not universal GPU claims.

What you get

• A real coding model, resident. Qwen3.8 27B — a hybrid Gated DeltaNet + attention architecture — as one 18.2 GB hash-pinned artifact, resident on your GPU with a 131,072-token context ceiling.
• The full OMP agent surface. Tools, Vision, stateful follow-ups, session forks, and preserved thinking, qualified together in one profile rather than advertised separately.
• Speculative decoding that pays for itself. The primary MTP3 profile measured 152.2 tok/s; each native GPU variant retains its own profile and receipt rather than inheriting that number.
• An operable runtime. Digest-pinned container, authenticated status identity, observable restart policy, owned stop path, and a launcher that refuses identity mismatches.
• A support boundary you can read. One compatibility authority, explicit non-claims, and issue forms that never ask for your prompts or logs.

Get started

Pick your lane: the RTX 5090 container route needs Docker with the NVIDIA runtime on Windows 11 + WSL2; the RTX 4090 and RTX 3090 native Windows routes install their exact pinned packages. Every route needs one published OMP client and about 40 GiB free disk. Use the exact v0.6.9 tagged guide and manifest together; do not mix releases.

git clone --branch v0.6.9 --depth 1 https://github.com/alphastorm/omp-ninfer.git
Set-Location omp-ninfer
python3 scripts/verify_release.py --require-ready

Then follow the quickstart: install the checksummed OMP client, fetch the hash-pinned model, start the digest-pinned NInfer container, add the provider fragment, and run the documented acceptance checks. The same document contains managed macOS SSH, native Linux, and native Windows 3090/4090 paths.

How it works

• OMP owns the truth. Transcript, tools, branches, and replay live in OMP. A turn advances provider state only after a complete valid stream and durable transcript publication.
• NInfer owns inference and retained state. The hardware-tuned C++/CUDA engine comes from Neroued/ninfer and its GPU ports; this project's runtime adds explicit Responses state and durable checkpoints, scoped by authenticated client and session identity. Retained state is an acceleration, never the source of truth.
• The manifest owns identity. Exact client, image, model, configuration, and qualification bytes; ready status requires the composed external acceptance from public URLs.

Deep dive: Architecture · Security model · Release lifecycle.

Engine throughput and avoiding repeated prefill are separate benefits. The benchmarks measure specific profiles and workloads, not a matched head-to-head speed ranking against vLLM or llama.cpp.

How it compares

Source-verified against public documentation, 2026-08. These projects move quickly; check their current docs. Fuller analysis including LM Studio, vLLM's Agentic API, LMCache, SGLang HiCache, and why no second gateway sits between OMP and NInfer: Related work.

	OMP NInfer	Ollama	LM Studio	llama.cpp server	vLLM
What it is	A small closed set of qualified OMP + runtime + model + GPU combinations with receipts	General local runtime with a large model library	Desktop app plus headless daemon with a large model catalog	General GGUF serving with the broadest hardware reach	High-throughput general serving engine
Session state across OMP turns	Stateful Responses owned end to end: transcript commits first, GPU-resident baseline advances second; survives OMP exit/resume; forks qualified	Stateless per request; transcript re-sent; in-process prefix reuse avoids recomputing matching prefixes	Stateless per request; chat state lives in the client	Stateless per request; per-slot prefix cache reuses matching prefixes	Stateless core with automatic prefix caching; separate Agentic API gateway adds server-side state
Restart and off-machine checkpoints	Durable restore on all three lanes; verified host/NAS replicas; same-runtime/profile/credentials restore only (scope)	Different mechanism/contract; verify current support	Different mechanism/contract; verify current support	Different mechanism/contract; verify current support	Different architecture, including external KV systems
Speculative decoding on the shipped model	Profile-specific: MTP3 on all three shipped lanes	Model/config dependent	Desktop app plus headless daemon with a large model catalog	Optional draft/ngram setups	Optional
Vision, tools, thinking	Qualified together in one profile	Varies by model	Varies by model; tools and structured output documented	Varies by model and build	Varies by model
Release discipline	Model SHA-256, image OCI digest, SBOM, client checksums, one ready manifest	Rolling releases, mutable tags	Rolling desktop releases	Rolling builds	Rolling releases
Fail-closed OMP route	Shipped and acceptance-tested	Depends on your client config	Depends on your client config	Depends on your client config	Depends on your client config
Breadth	One pinned artifact and three qualified GPU lanes bound by one ready manifest	Thousands of models, broad hardware	Large catalog, desktop UX, llama.cpp/MLX backends	Any GGUF, broad hardware	Broad models, datacenter and consumer GPUs

Where each shines: Ollama is the easiest way to run many models locally. LM Studio is the most polished desktop experience for browsing and running them. llama.cpp has the broadest hardware and quant ecosystem. vLLM is the throughput and multi-tenant serving reference. OMP NInfer is for one specific job — OMP plus Qwen on your own RTX card, long stateful coding sessions, privacy as a tested invariant rather than a configuration hope.

The NInfer family

This product rides an ecosystem of single-GPU NInfer ports, each specializing the engine for one architecture. Numbers below are published by each repository's maintainers on their own profiles and quantization schemes; they are not cross-comparable and are not claims of this product.

Repository	GPU	Published highlights	Relationship
Neroued/ninfer	RTX 5090 (sm_120a)	1,313.8 aggregate tok/s at C=8 (35B-A3B); 15,544 tok/s prefill at 7,680 tokens	The original engine; everything below forks it
alphastorm/ninfer	RTX 5090, RTX 4090, RTX 3090	152.2 tok/s on the primary 5090 profile; 90.17 C1 decode tok/s on the qualified native 3090 lane	This product's public runtime source and component releases
UDPSendToFailed/ninfer-4090	RTX 4090 (sm_89)	229.9 tok/s MTP7 deep-context decode; 10.1 GB/s DirectStorage cold weight DMA; E8-lattice KV to 567K-token ceilings	Upstream of the qualified native 4090 beta branch
Don-Chad/ninfer-3090	RTX 3090 (sm_86)	165.3 tok/s decode at C=8; RotorQuant KV to 247,872-token contexts; ReplaySSM	Upstream of the released preview and fresh parity candidate

All three lanes are qualified releases in the v0.6.9 manifest, each bound to its exact package, receipt, and profile. What comes next: ROADMAP.md.

Benchmarks and leaderboard

Benchmarks holds the qualified results, the warm-vs-cold and per-lane charts, the upstream campaign highlights, the model-quality table, and a community results table seeded with the maintainer entries. Submit your environment's numbers with the performance result form after the documented acceptance checks pass. Planned measurements we want next are listed there too.

Help make it better

• Ran a clean install? Report time-to-first-turn and every manual step — install friction is a bug.
• Measured your card? Submit numbers with the performance result form after the acceptance checks pass; verified rows join the community leaderboard.
• Work on CUDA kernels? Pick a measured bottleneck from the performance program and submit before/after receipts with the same form.
• Need a different model or profile? File a model/profile request so artifact identity and qualification scope stay explicit.

Docs, release tooling, and profile contracts belong here; engine work belongs in the runtime repositories. The complete routing and evidence rules are in CONTRIBUTING.md.

The beta OMP client carries the NInfer stateful-Responses provider integration. Its exact accepted source is public at alphastorm/oh-my-pi; the standing intent remains to upstream reusable provider and lifecycle pieces to can1357/oh-my-pi.

Roadmap

v0.6.9 ships an independently implemented semantic port of upstream Qwen tool-parser fixes 3b50962b and 0c5d570c: supported scalar unions, case-insensitive booleans, precise numeric lexemes and mathematically integral values, duplicate parameters, and balanced embedded markup. It preserves custom raw input, history, opaque IDs, and stream ownership; review remediation prevents malformed-region rescans and recursive union traversal. Both mainline components are published and lane-qualified; documented host/macOS and native public-install acceptance passed against their published bytes. v0.6.8 puts both mainline lanes on one runtime source (68a0722f): the RTX 4090 native lane takes the upstream engine work and the GDN capacity fix, and a fork continued while its sibling is alive no longer answers HTTP 500 on any lane (ninfer#43, EXP-036). v0.6.7 moves the RTX 5090 runtime onto a selective backport of the upstream engine work - 18 commits taken with reasons, the rest deferred with reasons - requalified on every lane gate within noise of the shipped runtime (EXP-035). v0.6.6 keeps the pinned client on its channel: the config every route installs turns the client's startup update check off, so nobody is advised to omp update away from the hash-pinned release bytes (#18). v0.6.5 closes the last uncovered route: the quickstart's primary macOS row runs end to end from its own blocks with every outcome decided by the shell, including a session that survives the server process - two blocks that were wrong for a Windows destination fixed at source (EXP-033). v0.6.4 makes the documentation itself the accepted route: every Windows route runs end to end from its own quickstart blocks on a stock host, and a clone yields the recorded bytes on every platform - six defects a reader would have hit, and no acceptance script ever did, fixed at source (EXP-032). v0.6.3 makes the route the documentation tells you to run durable: the published RTX 5090 container launcher now mounts a session store, so a session saved on it survives the server process and continues exactly - which that route could not do at all before, while its server reported the identity of a configuration qualified elsewhere (EXP-031). v0.6.2 puts all three lanes on one runtime tree: the RTX 5090 container lane moves off the branch head it had served from since v0.4.4 onto the mainline commit the native lanes build from, requalified 7/7 on the owner appliance and re-verified against the anonymously pulled image, with throughput and durability within run-to-run noise of v0.5.1 (EXP-030). v0.6.1 makes a managed stop of the RTX 4090 lane save every live session: the manager signals the server through a per-launch named kernel event instead of terminating it, and a session that was never published survives a deliberate restart (EXP-028/EXP-029). v0.6.0 brought the RTX 4090 lane onto the mainline runtime: the same context-cache architecture the RTX 5090 container ships - sibling forks on a shared long anchor, warm arrival across a restart, streamed SHA-verified restore - now serves on Ada from one source tree instead of a divergent lane branch, requalified 15/15 on its own lifecycle tool (EXP-025 through EXP-027). v0.5.1 closed the second v0.5 promise on the RTX 5090: a checkpointed template arrives warm across a restart, and a 5 GB restore takes about 4 s instead of 24 s (EXP-022/EXP-023). v0.5.0 made sessions leave the machine: origin-authenticated checkpoint manifests hold on all three lanes and scripts/checkpoint_sync.py replicates verified generations out and back (EXP-018). Next: the RTX 3090 lane follows onto mainline when its host returns, and a same-profile machine-pair resume when a second card of one lane exists. Signing/notarization, a shared public client acceptance runner, and multi-owner clean-install evidence remain on the path to v1.0. No item becomes a support claim before an exact package, receipt, and product manifest bind it.

Scope boundaries and explicit non-claims: ROADMAP.md.

Release integrity

The release manifest is the authority for component identity. A release is ready only when:

python3 scripts/verify_release.py --require-ready
python3 -m unittest discover -s tests -v

Both pass on the tagged release. The ready manifest binds the accepted Windows client archive, compatibility authority, NInfer image and SBOM, model artifact, qualification summary, and an owner-operated tester-equivalent external acceptance composed from the immutable client platform receipts, the public-URL byte-identity and smoke acceptance of the new RTX 3090 lane, and the fresh RTX 5090 requalification. Published tags and release notes must use those exact bytes. Lifecycle details: Releases.

Feedback and support boundary

Use the hardware report, installation failure, or benchmark forms. Remove API keys, hostnames, usernames, private prompts, model outputs, and raw request logs before attaching anything; the forms only ask for content-safe facts. The support boundary assumes a single trusted owner on both machines; this release is not a multi-tenant service. Security reports go through private vulnerability reporting, never a public issue.

Credits

Ordered by how much this product owes them:

1. Oh My Pi by can1357 — the coding agent this appliance exists to serve. OMP's provider architecture, transcript ownership, and session semantics are what make a stateful local backend worth building. Oh My Pi itself builds on Pi by Mario Zechner. MIT.
2. NInfer by Neroued — the from-scratch C++/CUDA engine this whole family rides: the .ninfer artifact format, MTP speculative decoding, the hybrid GDN runtime, and the published performance and evaluation campaigns cited throughout these docs. Apache-2.0.
3. The Qwen team — the Qwen3.8 model family. The shipped artifact is the registered NInfer conversion published at neroued/Qwen3.8-27B-NInfer. Apache-2.0.
4. UDPSendToFailed/ninfer-4090 — the RTX 4090 port (E8-lattice KV quantization, DirectStorage weight DMA) the qualified 4090 lane builds on. Apache-2.0.
5. Don-Chad/ninfer-3090 — the RTX 3090 port (ReplaySSM, RotorQuant) underlying both the released preview and fresh parity candidate. Apache-2.0.
6. Algorithm and library lineage — Gated DeltaNet (arXiv:2412.06464), Tri Dao's ReplaySSM note, Z-Lab's DFlash, Unsloth's NVFP4 weights, and vendored utf8proc, nlohmann/json, and cpp-httplib — credited in full in the runtime repositories.

How each upstream is tracked, with fork points and the current pull-in position: Upstream watch.

OMP NInfer is a community project; it is not affiliated with or endorsed by Oh My Pi, Qwen, or NVIDIA.

Repositories

Repository	Owns
alphastorm/omp-ninfer	Product front door: release manifests, profiles, quickstart, qualification composition, support boundary
alphastorm/ninfer	Public tagged RTX 5090, RTX 4090, and RTX 3090 component source
alphastorm/homebrew-omp	Client distribution: release archives plus stable omp and prerelease omp-beta casks

The OMP client source is a public fork of can1357/oh-my-pi at alphastorm/oh-my-pi; upstreaming remains a roadmap goal. The user-facing command remains omp; "appliance" names the operating concept, and OMP NInfer names this integration and repository.

License

MIT. NInfer and the Qwen artifact retain their own licenses and notices.