blazr

Production-grade inference server for LLMs. Supports standard HuggingFace models (Llama, Mistral, Qwen, Phi, Gemma, DeepSeek) and custom hybrid architectures (Mamba2, MLA, MoE). Loads SafeTensors, AWQ, GPTQ, and GGUF formats.

Features

• Multi-Architecture - Llama, Mistral, Mamba2, MLA+MoE, hybrid models, and more
• Multi-Format - SafeTensors (F16/BF16), AWQ INT4, GPTQ INT4, GGUF (23 quantization levels)
• Auto-Detection - Detects architecture, format, and tokenizer from checkpoint tensor names
• OpenAI-Compatible API - Drop-in replacement with /v1/completions and /v1/chat/completions
• Streaming - Server-Sent Events (SSE) for real-time token generation
• HuggingFace Hub - Pull models directly from HuggingFace
• Production Features - Rate limiting, request timeouts, graceful shutdown, CORS, error handling
• CUDA Acceleration - Optional GPU inference with optimized quantization kernels

Quick Start

Installation

# Build (CPU-only)
cargo build --release

# Build with CUDA support (requires CUDA 12.x)
cargo build --release --features cuda

Generate Text

blazr run \
  --model meta-llama/Llama-3.2-1B \
  --prompt "Once upon a time" \
  --max-tokens 100

Start Server

blazr serve --model meta-llama/Llama-3.2-1B --port 8080

# Text completion
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The capital of France is",
    "max_tokens": 50,
    "temperature": 0.7
  }'

# Chat completion
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100,
    "stream": true
  }'

Other Commands

blazr info --model ./models/mistral-7b     # Show model architecture and config
blazr list                                  # List available local models
blazr pull Qwen/Qwen2.5-0.5B               # Download from HuggingFace

Supported Models

Tested

Model	SafeTensors	AWQ	GPTQ	GGUF
Llama 3.2 1B	x	x	x	x
Mistral 7B	x	x	x	x
Mamba2 (oxidizr)	x	-	-
DeepSeek-V2/V3 (MLA+MoE)	x
Hybrid Mamba+Attn (oxidizr)	x	-	-

Expected Compatible (route to Llama)

Qwen2/2.5, Phi-3/3.5/4, Gemma/Gemma2, StarCoder2, Yi, InternLM2, CodeLlama, Solar

Planned

Mixtral (MoE), Falcon (ALiBi), Command-R, GPT-NeoX/Pythia, DBRX

Model Formats

blazr auto-detects the format from checkpoint contents:

SafeTensors (HuggingFace)

model_dir/
├── model.safetensors          # or sharded: model-00001-of-00002.safetensors
├── config.json                # HuggingFace model config
└── tokenizer_config.json      # Chat template (optional)

SafeTensors (oxidizr)

checkpoint_dir/
├── model.safetensors
└── config.json                # oxidizr config (optional, inferred from tensors)

GGUF

model.gguf                     # Single file: weights + tokenizer + config

Supports all 23 GGUF quantization levels (Q2_K through Q8_0, IQ series, TQ series). CPU has dedicated kernels for all formats. CUDA has optimized dp4a kernels for Q4_K, Q6_K, Q8_0 with generic fallback for the rest.

API Reference

Endpoints

Method	Path	Description
GET	/health	Health check
GET	/v1/models	List loaded models
POST	/v1/completions	Text completion
POST	/v1/chat/completions	Chat completion

Request Parameters

{
  "prompt": "text",
  "messages": [{ "role": "user", "content": "text" }],
  "max_tokens": 100,
  "temperature": 0.7,
  "top_p": 0.9,
  "top_k": 40,
  "stop": ["\n\n"],
  "stream": false
}

Response Format

Non-streaming responses follow the OpenAI format:

{
  "id": "cmpl-...",
  "object": "text_completion",
  "created": 1234567890,
  "choices": [
    {
      "text": "generated text",
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 20,
    "total_tokens": 25
  }
}

Streaming responses use SSE with data: {...} chunks and data: [DONE] sentinel.

CLI Reference

blazr run       Generate text from a prompt
blazr serve     Start the inference server
blazr info      Display model architecture and configuration
blazr list      List available local models
blazr pull      Download a model from HuggingFace Hub

Generation Options

Flag	Default	Description
--model	required	Local path or HuggingFace model ID
--prompt	required	Input text
--max-tokens	100	Maximum tokens to generate
--temperature	0.7	Sampling temperature (0 = greedy)
--top-p	0.9	Nucleus sampling threshold
--top-k	40	Top-k sampling
--vocab	auto	Tokenizer vocabulary (auto-detected from model)
--cpu	false	Force CPU inference

Server Options

Flag	Default	Description
--model	required	Local path or HuggingFace model ID
--port	8080	Port to listen on
--host	0.0.0.0	Host to bind to
--cpu	false	Force CPU inference

Tokenizer

Uses splintr for BPE tokenization. Vocabulary is auto-detected from model config or can be specified with --vocab.

Vocabulary	Models	Vocab Size
llama3	Llama 3.x, Mistral	~128k
cl100k_base	GPT-4, GPT-3.5-turbo	~100k
o200k_base	GPT-4o	~200k
deepseek_v3	DeepSeek V3/R1	~129k

GGUF files include an embedded tokenizer which is extracted automatically.

Architecture

blazr is a thin application layer built on boostr (ML framework) and numr (numerical computing). All model architectures, tensor operations, and quantization kernels live in boostr. blazr provides the CLI, HTTP server, model loading, and request orchestration.

blazr (CLI + HTTP server + model lifecycle)
  |
boostr (model architectures + quant kernels + NN modules)
  |
numr (tensors + linalg + multi-backend: CPU/CUDA/WebGPU)

Requirements

• Rust 1.70+
• (Optional) CUDA 12.x for GPU acceleration

License

Apache-2.0 - see LICENSE for details.

Related Projects

• boostr - ML framework (model architectures, quantization)
• numr - Foundational numerical computing (tensors, linalg, multi-backend)
• oxidizr - Training framework for hybrid architectures
• splintr - High-performance BPE tokenizer
• compressr - Model conversion and compression