memra

Running in production: memra serves the public API at inference.tiyuvta.ai — prepaid, OpenAI-compatible, one model live. If you want the engine's behaviour without owning the card, that is the shortest path to it, and the endpoint's model card publishes the same measurements this README does, with their conditions. The lab behind both is tiyuvta.ai.

What	Inference engine, Rust + CUDA, OpenAI-compatible serving
Tuned for	RTX PRO 6000 Blackwell (sm_120a) and RTX 5090, with a compile-gated H100 (sm_90a) lane
Format	GGUF primary; safetensors/NVFP4 paths under active work
Shape	One model per GPU, PP-2 across two when the weights do not fit
Hosted	inference.tiyuvta.ai · lab: tiyuvta.ai
Licence	MIT

Contents — Install · Quick start · Performance samples · Models and hardware · Serving · Design boundaries · Docs · Contributing

---

memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. It serves supported Hugging Face safetensors checkpoints and GGUF artifacts over an OpenAI-compatible API, and every default it ships was measured on those two cards — both Blackwell sm_120a, with a separately compile-gated Hopper/H100 (sm_90a) lane alongside.

That focus is the point, not a limitation. The mainstream serving stacks are tuned for datacenter Hopper and Blackwell; memra is tuned for the Blackwell cards people actually own — the 96 GB PRO 6000 Workstation/Server Edition and the 5090 — on the assumption that you want yours saturated rather than merely supported. Concretely, on ONE PRO 6000 it serves a 35B MoE at a 262,144-token context, holds a ~48 GiB in-VRAM prefix cache, and sheds excess load with clean 429s instead of degrading everyone's latency; the 5090 carries the same engine at its own measured settings.

Both cards are first-class, and neither is tuned at the other's expense. Arms are selected per device: when a mechanism wins on the 5090 and loses on the PRO 6000, or the reverse, it becomes a per-hardware default keyed on the device rather than a compromise applied to both. Detection drives that choice, so a naked command runs at full speed on whichever card it lands on.

Its center of gravity is OpenAI-compatible serving: streaming chat and text completions, prefix and session reuse, VRAM-aware admission, MTP speculative decoding, and PP-2 for models that need two cards.

The deployment philosophy is specialization: one model on one GPU, with tuned paths selected by default. PP-2 extends that shape to one model across two GPUs when the weights do not fit on one card. For new upstream models, the official safetensors checkpoint plus its config, tokenizer/template, quantization metadata, and auxiliary tensor files is the preferred semantic source. Memra repacks those source tensors once into measured rig-native layouts; safetensors is not the internal compute format. GGUF remains a fully supported, self-contained portable import and distribution format.

Use memra if you are serving on RTX PRO 6000 Blackwell (or a 50-series card) and want defaults that were measured on your hardware. Look elsewhere if you need broad model coverage, tensor parallelism, or a datacenter-H100 fleet — those are other projects' strengths, and Design boundaries says so plainly.

Measured speed is summarized in three generated same-rig samples; the complete boards and conditions live in docs/PERFORMANCE.md.

Releases are listed at releases/latest, which is where the current tag lives — this README deliberately does not repeat the number, because a version in prose is stale the day after it is written. main runs ahead of the latest tag. Recent waves: security hardening (serving refuses unauthenticated non-loopback binds at boot, metrics endpoints require bearer auth whenever keys are configured, API-key comparisons are constant-time, keyring rewrites are atomic, and the GGUF parser returns contextual errors instead of panicking on truncated or malformed files — a truncated multi-gigabyte download is an ordinary event, not a crash), then a correctness wave on the serving path: a request may no longer produce tokens under two different numerical programs, which is now a documented rule rather than a series of separate fixes. Details under Recent fixes.

Serving Qwen3.8-27B in production

memra serves Qwen3.8-27B at its full native 262,144-token context on RTX PRO 6000 Blackwell, in production at tiyuvta.ai — live the day after the checkpoint's release, gated by the full exactness battery before a single public token.

Measured on the serving hardware (one PRO 6000, NVFP4+Q5_K artifact, real agentic prompts, 3-rep medians, zero sheds or errors across every cell — 2026-08-15):

Metric	Measured
TTFT p50, cold	0.156 s (c=1) — ≤0.32 s through c=4
TTFT, cached conversation turn	0.130 s on a 5.7k-token context (full prefix restore)
Decode p50, single stream	RTX PRO 6000: 140 tok/s (rep medians 138–141, v0.86.1 fused4; plain 75) · RTX 5090 Laptop: 75 tok/s (range 71–80, v0.85.1; plain 44)
Sampled-config throughput	top-p/top-k/min-p requests sample on-device — sampled aggregate equals greedy (240–245 tok/s at c=16–32)
Aggregate completion	238–245 tok/s at c=16, flat to c=32, zero sheds across capacity mixes
Sustained soak	576/576 requests, 0 errors, 0 sheds, −0.27% drift
Spec ON/OFF exactness	8/8 byte-identical; verify gate: zero differing logits at T=1..4, K=1/3/8

Since v0.86: image and video input on the same endpoint (OpenAI image_url / video_url content parts, base64 data URIs; videos as animated GIF, decoded in-process) — the checkpoint's native ViT tower runs in-engine, gated by a per-token cosine parity oracle against the HF reference before serving (images min-cos 0.9997, video 0.99999); vision tokens bill as ordinary prompt tokens.

Day-one extras: the checkpoint's MTP head ships as a published own-generation-trimmed FR-Spec masked draft (Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF, vocab 248,320 → 32,768, re-measured on v0.86.2 at +5.1% e2e on the PRO 6000 and +6.4% on the 5090 Laptop vs the full head, interleaved A/B ×5 — zero correctness cost, verification is the target's job either way). Two honest architecture notes, published because they were measured: on this GDN-hybrid class the prefix cache serves extension shapes (conversation continuation, session affinity) rather than fan-out shapes, because the recurrent state exists only at an entry's end boundary.

Tuning continues on the same hardware for Step-3.7-Flash and the Qwen3.6 family; numbers land in docs/PERFORMANCE.md as they are measured; a default only ships for a card class it was measured on.

Two results from that tuning are worth stating up front, because both are counter-intuitive and both were measured on one RTX PRO 6000 Blackwell Workstation serving a 35B MoE at a 4,860-token prompt shape with a shared prefix:

• Prefix-cache depth dominates. With the cache holding the full shared prefix, concurrency 16 ran at 8.50 req/s and a 1.87 s median. With a shallower entry covering the same prefix class, the same build on the same card ran 2.72 req/s at 5.84 s — a 3.1x swing in both throughput and latency from cache depth alone.
• Speculative decoding is not free on cache-carried shapes. It is numerically exact, and offline it looks like a 1.5-1.7x win, but on this serving shape it cost 4x because a speculative session gives up the cross-request prefix cache. MEMRA_SERVE_SPEC=0 is the measured operational posture for that shared-prefix shape; the binary default remains placement-aware and spec-capable for cache-poor requests. See docs/FLAGS.md.

Raw per-run receipts for both are under research/, and the second is the kind of result this project publishes either way: the arm expected to win lost.

Install

The release installer is the shortest path. It selects the published sm_120a, sm_90a, or sm_89 prebuilt, verifies the release checksum, and installs memra-server, run-gen, run-spec, and kernel-check.

curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

Prebuilt binaries require Linux x86_64, glibc 2.35 or newer, NVIDIA driver 580 or newer, and the CUDA runtime libraries. They do not require nvcc. Set MEMRA_INSTALL_DIR to override ~/.local/bin.

To build all workspace binaries from source:

git clone https://github.com/avifenesh/memra.git
cd memra
cargo build --release
export PATH="$PWD/target/release:$PATH"

Source builds require Rust 1.85 or newer and the CUDA 13.1 toolkit. Architecture is detected at build time; MEMRA_CUDA_ARCH is the documented override.

Quick start

Point the command at a supported Hugging Face checkpoint directory or GGUF artifact to run one chat-templated generation. New upstream model onboarding starts from the official checkpoint:

MEMRA_CHAT=1 run-gen /absolute/path/to/hf-checkpoint \
  --prompt "Explain KV caches in one sentence."

run-gen also accepts a supported GGUF or an hf:owner/repo[:file] spec. The hf: form downloads and caches the selected artifact on first use.

Start the OpenAI-compatible server in one terminal:

MODEL=/absolute/path/to/hf-checkpoint
MEMRA_MODELS="qwen=$MODEL" memra-server

The default bind address is 127.0.0.1:8080. From another terminal, stream a chat completion:

curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Explain KV caches in one sentence."}],
    "max_tokens": 128,
    "stream": true
  }'

The qwen value is the alias from MEMRA_MODELS. Bearer authentication, multiple model aliases, MTP drafter attachment, and deployment probes are covered in docs/SERVING.md.

Performance samples

The README carries only representative regression samples. The full boards, methodology, thermal regime, N, and open cells live in docs/PERFORMANCE.md.

Model / scenario	memra tok/s	llama.cpp tok/s	Ratio
Qwen3.6-35B-A3B MoE — plain decode (the flagship serving model, tg128 at 512-token context)	187.0	164.9	1.13x
Qwen3.5-9B — MTP speculative, short / medium / long-agentic prompt classes (both engines at their measured best)	281.0 / 211.7 / 187.1	122.2 / 121.5 / 117.7	2.30x / 1.74x / 1.59x
Qwen3.6-35B-A3B MoE — plain decode at 6.3k-token context (the short-context lead holds at depth)	177.1	156.3	1.13x

Measured 2026-08-02 on the RTX 5090 Laptop — same-session interleaved medians, same exact prompts; memra at its naked defaults, llama.cpp at its swept best (docs/COMPETITOR-SETUP.md). The llama.cpp column is a frozen reference recorded through 2026-08-03 (benching stopped that day). N, thermal regime, and the full boards: docs/PERFORMANCE.md; raw per-run logs: research/tune-data/.

Models and hardware

Support is specific to a model, quantization, and drafter combination. The generated table is numbers-free; per-model measurements stay in docs/PERFORMANCE.md.

• RTX PRO 6000 Blackwell (sm_120a) — co-primary target. Workstation and Server Edition, 96 GB. Carries verification, final tuning, and serving, single-card and as PP-2 pairs.
• RTX 5090 / 50-series (sm_120a) — co-primary target. Same architecture, its own measured settings, and its own defaults where they differ. Local 5090 performance is never traded away to simplify a remote default; a perf claim needs numbers from both cards before it sets a global one, and one-card evidence sets a one-card default at most.
• Hopper sm_90a — separately compile-gated H100 lane with its own validation battery and evidence ledger in ARCHITECTURE-H100.md. Secondary: it does not change the naked sm_120a build or its defaults.
• Ada sm_89 — portable prebuilt target, not a tuned performance target.
• B200 sm_100a — source-only, compile-gated backend. Other architectures are not a tuned support promise.

Model	Class	Quant	Drafter	Supported since
Qwen3.5-9B	dense	NVFP4 (5090), Q8_0 (H100)	MTP + own-gen trimmed draft	v0.1.0
Qwen3.8-27B	dense hybrid (GDN + gated attention)	NVFP4+Q5_K GGUF (from the unsloth BF16 re-export)	MTP + own-gen FR-Spec masked head (HF: Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF)	v0.82.2
Qwen3.6-27B	dense	NVFP4, Q4_K_M MTP-baked	MTP + own-gen trimmed draft	v0.1.0
Qwen3.6-35B-A3B	MoE	IQ4_XS	MTP + own-gen trimmed draft	v0.1.0
Gemma-4 26B-A4B	MoE	QAT Q4_0	Gemma assistant draft (CLI only)	v0.23.0
Gemma-4 31B	dense	QAT Q4_0	Gemma assistant draft (CLI only)	v0.35.0
Gemma-4 E4B	dense	QAT Q4_0	Gemma assistant draft (CLI only)	v0.35.0
Gemma-4 12B	dense	QAT Q4_0	Gemma assistant draft (CLI only)	v0.40.0
Ornith-1.0-9B	dense	Q8_0	own-gen donor-block draft	v0.63.0
Ornith-1.0-35B	MoE	Q4_K_M	own-gen donor-block draft	v0.64.0
Qwen-AgentWorld-35B-A3B	MoE	UD-IQ4_XS (avoid UD-Q4_K_M — its Q5_K expert mix sits outside fast-path coverage)	own-gen drafter	v0.66.0
Step-3.7-Flash 196B-A11B	MoE	IQ4_XS + Q8_0 MTP head (two-card PP-2)	MTP (single-card); plain batched decode on PP-2	v0.73.1

Step-3.7-Flash serves on a two-card PP-2 pair (MEMRA_PP_STAGES=2); its explicit configuration and qualification boundaries are recorded under bring-up notes. On PP-2 it decodes in a single numeric class at every batch width — served bytes do not depend on load history.

The Gemma-4 drafter is a separate-assistant format, not a NextN/MTP head, and it is reachable only from the gemma-gate CLI path — memra-server attaches drafters exclusively through the generic NextN loader, which refuses this format (draft n_embd != model n_embd). Gemma-4 serves with speculative decoding off; speculative numbers for these SKUs are CLI measurements, not serving ones.

Serving

memra-server is the center of the current engine. The public surface includes:

• OpenAI-compatible HTTP — /v1/chat/completions and /v1/completions, blocking or SSE streaming; /v1/models reports loaded-model capabilities through supported_parameters.
• Request reuse and admission — tenant-scoped prefix caching, same-window prefix dedup, rewritten-history session affinity, request-sized KV allocation, and VRAM-aware admission.
• Speculative decoding — embedded or attached MTP/NextN drafters, with request-conditioned depth and automatic plain-decode fallback where batching or PP-2 wins.
• Agent surfaces — tool schemas and streaming tool_calls, separated reasoning output, reasoning_effort mapped through each model's chat template, and constrained decoding via response_format json_object or json_schema.
• Operations — optional keyed bearer auth, per-tenant cache boundaries and request caps, health/readiness probes, metrics, and graceful drain.
• Long context and larger models — model-capped 262,144-token serving, plus PP-2 to split one model across two cards. Step-3.7-Flash is the current large-MoE PP-2 trial.

See docs/SERVING.md for request fields, response shapes, capability gates, auth configuration, cache semantics, admission behavior, and deployment runbooks.

Recent fixes

The latest serving wave closes several serving-contract failures:

• Serving fails closed on exposure: a non-loopback bind without a configured key source refuses to boot, /metrics and /yield/metrics require bearer auth whenever keys are configured or the bind is public, API-key comparisons are constant-time, and keyring rewrites are atomic (receipt).
• The GGUF parser rejects truncated and malformed files with contextual errors instead of panicking the GPU worker — every byte-prefix of a valid file is covered by fixture tests (receipt).
• The GPU Gumbel sampler now keeps uniforms below 1.0, removing rare +inf winners that injected arbitrary tokens during long sampled generations. The 262,144-token serving target remains in place (receipt).
• Explicit max_tokens is enforced exactly. Token events, usage, and scheduler accounting now follow visible token IDs rather than speculative rounds (receipt).
• Plain-decode session affinity now checkpoints and resumes rewritten conversation histories, so later turns prime the new suffix instead of an ever-growing history (receipt).
• New admissions can end an in-flight speculative burst at its next round boundary instead of waiting for the whole burst; request-sized KV allocation remains VRAM-aware (receipt).
• Cache boundaries were hardened: release builds retire prefix pins and client cache salts are bounded and validated (receipt).
• /v1/models now advertises the parameters each loaded model actually supports (gateway contract), and as of v0.86.3 renders pricing from the same MEMRA_MODEL_METADATA entry the request ledger bills from (cache prices under input_cache_read/input_cache_write) plus the declared input modalities in the modality string (text+image+video->text) — previously a priced, vision-serving endpoint reported a hardcoded "0" text-only stub.
• v0.87.0 adds the router-marketplace provider contract (v2) to the same catalog: contract_version, a [provider] metadata block (status URL, contacts, regions) with the server-truth error contract (429/503 + Retry-After, insufficient_balance quota code, x-request-id echo), per-model lifecycle and reliability-timeout blocks, capability booleans, and per-1M-token decimal-string prices computed by exact decimal shift from the billed per-token metadata.
• Step-3.7-Flash PP-2 decodes in one numeric class at every batch width: a greedy request returns the same bytes whether it decodes solo, joins a batch mid-generation, or starts batched. The transition matrix that proved and closed the load-history divergence is the receipt (isolation, fix).
• Grouped expert prefill (MEMRA_MOE_GROUPED=1) looked promotable on two-card Step serving in August 2026 — 4k streaming TTFT 10.96 s -> 7.26 s, N=5 per arm (receipt) — but a re-sweep on an RTX PRO 6000 pair withdrew it: the arm is slower there (2,687.5 vs 8,193.7 tok/s resident-KAT transfer, 0/5 paired wins) and it fails the Q35 mixed c=4 exact-token gate, truncating every request in the cell (receipt). It stays off by default and should not be enabled; see MEMRA_MOE_GROUPED in docs/FLAGS.md.
• An opt-in SWA ring (MEMRA_SWA_RING=1) right-sizes sliding-window KV for Step sessions: the 262k session KV component drops 3.6x — measured 2 -> 12 concurrent 262k sessions before the first defer; lapped checkpoints decline safely (receipt, flag-on validation).
• A B=1 decode specialization for Step models recovers most of the one-class contract's cost: +4.4% sustained c=1 decode with byte-identical output (the batched walk was issuing 90 arithmetic-free device copies per token at B=1) (receipt).
• A host-staged PP boundary fallback (MEMRA_PP_HOST_BOUNCE=1, default off) serves byte-correct on hosts whose GPU peer-copy path reports success but does not preserve bytes — a failure mode a capability flag or bandwidth test does not reveal. Peer paths should be byte-probed at provisioning (receipt).
• Graceful shutdown joins the GPU worker thread before exit, closing a restart race where a new server could boot while the old worker still held device state.

Design boundaries

• Tuned paths are defaults. Environment variables are runtime parameters, machine configuration, diagnostics, or rollback seams; the catalog is docs/FLAGS.md.
• The standing correctness battery is kernel-check, the run-gen argmax gate, and run-spec K=1..8 self-consistency. GitHub CI is compile-only; GPU evidence runs on the target rig.
• memra is not a broad model-compatibility or multi-vendor runtime. Unsupported architectures, model families, and quantizations may compile or load without becoming supported configurations.
• Multi-GPU support is replica fleet or pipeline-parallel PP-2. Tensor parallelism is not implemented.
• This is a moving research engine. Public APIs and flags can change between releases.

Docs

Read this	For
docs/SERVING.md	API contract, caching, auth, admission, PP-2, and operations
docs/PERFORMANCE.md	Full boards, methodology, rigs, gaps, and model qualification
docs/FLAGS.md	Audited environment-variable catalog
ARCHITECTURE.md	Engine structure and the sm_120a implementation ledger
ARCHITECTURE-H100.md	H100-specific mechanisms and evidence
CONTRIBUTING.md	Development workflow and GPU correctness gates
docs/RELEASING.md	Versioning, target-rig battery, tags, and publication

Contributing

Issues and PRs are welcome. Start with CONTRIBUTING.md; validation on a new GPU should use the hardware report template.

License

MIT — see LICENSE.