speculative-decoding-runtime

Speculative decoding runtime with rejection sampling, adaptive gamma, and provable correctness guarantees.

1.41x speedup on CPU | 1.25x speedup on GPU (Fair Baseline) | Greedy exact match verified

---

What It Does

Speculative decoding accelerates LLM inference by using a small draft model to generate candidate tokens, then verifying them against a larger target model in a single forward pass.

• Draft model (fast): Generates gamma candidate tokens.
• Target model (accurate): Verifies all gamma in one forward pass.
• Rejection sampling: Ensures output distribution matches the target model exactly.

---

Features

• Standard Speculative Decoding with rejection sampling.
• Adaptive Gamma: Adjusts speculation length dynamically based on acceptance rates.
• Correctness Guarantee: Validated empirically (greedy exact match).
• GPU Support: Optimized for FP16 inference on CUDA devices.
• Model Pair Validation: Vocabulary compatibility check at startup.

---

Quick Start

# Install dependencies
pip install -r requirements.txt

# CPU benchmark
PYTHONPATH=. python benchmarks/baseline_vs_spec.py --device cpu

# GPU benchmark (Fair baseline comparison)
PYTHONPATH=. python benchmarks/gpu_fair_benchmark.py --device cuda

---

Benchmark Results

Configuration: 20 tokens generated, median of 10 iterations after 5 warmup rounds.
Hardware: NVIDIA RTX 2070 8GB (GPU) | Intel i7-10700K (CPU).
Model Pair: Qwen2-0.5B (Draft) / Qwen2-1.5B (Target).

1. GPU Benchmark (FP16)

To isolate the algorithmic efficiency of speculative decoding, we use a Fair Baseline (manual autoregressive loop without KV Cache) to match the current runtime's architecture.

Method	Latency (20 tokens)	Speedup	Correctness
Manual Autoregressive (Fair Baseline)	1118.66 ms	1.00x	-
Speculative Decoding (Gamma=2)	896.23 ms	1.25x	Verified

Technical Note: GPU speedup (1.25x) is lower than CPU speedup (1.41x). This is counter-intuitive but expected without KV cache reuse. On GPU, individual forward passes are so fast that the relative overhead of prefix re-encoding becomes proportionally larger. Implementing KV cache reuse would eliminate this gap.

2. CPU Benchmark (FP32)

gamma	speedup	acceptance
1	1.21x	73.3%
2	1.41x	74.2%
4	1.18x	58.3%

Comparison Against Production Baselines

• This Project (Fair Baseline): 1.25x speedup on GPU.
• HuggingFace .generate() (with KV Cache): Current gap reflects the KV cache reuse advantage in production engines, not algorithmic divergence.

---

Correctness

Speculative decoding with rejection sampling provably preserves the target model's output distribution.

• Greedy Mode: Speculative output exactly matches target autoregressive output (token-by-token).
• Verification: Verified across both CPU (FP32) and GPU (FP16) implementations.
• Unit Tests: 31/31 tests passing, covering rejection sampling math.

---

Project Structure

• src/engine/speculative.py: Core speculative decoding algorithm.
• src/engine/verification.py: Rejection sampling and prefix acceptance logic.
• src/strategies/adaptive_gamma.py: Dynamic speculation length adjustment.
• benchmarks/gpu_fair_benchmark.py: GPU performance validation.

---

Limitations & Future Work

• KV Cache Reuse: Current implementation re-encodes the full prefix each round. Implementing KV Cache would unlock >2x speedups.
• Batching: Currently optimized for single-sequence inference.

---

License

MIT - Joao Felipe De Souza, 2026