Record: 1.0240 BPB — Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (100% autonomous research via goldfish)

[lukacf]

Summary

val_bpb = 1.0244 (3-seed mean) | 15.79 MB artifact | 8xH100 SXM, 600s training + 124s eval

Novel eval-time n-gram cache with two extensions over the vanilla 5-gram approach (PR #674):

1. Multi-order backoff (2,3,4,5-gram): When a 5-gram context has no match, fall back to 4-gram, 3-gram, 2-gram
2. Entropy-adaptive mixing weight: When model is uncertain (high entropy), trust n-gram more. alpha = 0.05 + 0.35 * sigmoid(2 * (H - 4.0))

These extensions improve over vanilla 5-gram by 0.018 BPB (1.0423 → 1.0240) on the same base model.

Legality

• Score-first: Cache updated only after scoring each token
• No target-aware gating: Alpha depends on model entropy (model's own distribution), not the true target — exactly the approach suggested by @valerio-oai as a legal fix (PR Record: 5-gram Eval Cache + LeakyReLU² + Parallel Muon val_bpb: 1.0920 (3-seed mean, std 0.0007) | ~15.9 MB | 8×H100 SXM #659 review)
• Proper distribution: p_mixed = (1-a)*p_model + a*p_ng sums to 1 over all vocab tokens. Both components are proper distributions. Target-only lookup is mathematically equivalent to full-vocab computation. See code comments for proof.

3-Seed Validation

Seed	Sliding BPB	N-gram BPB	Artifact
1	1.1156	1.0240	15,788,203
2	1.1164	1.0247	~15,790,000
3	1.1158	1.0242	~15,790,000
Mean	1.1159	1.0243
Std	0.0004	0.0003

Training Architecture

11L transformer, 512d, GQA, LeakyReLU(0.5)², XSA-all(11), VRL, Soft-Round QAT, Full Hessian GPTQ (Cholesky + actorder), int6+zstd-22, 7% prune. ~6600 steps at 91ms/step on 8xH100 SXM.

Infrastructure

Discovered and validated autonomously in a single research session using Goldfish ML (MCP-based experiment platform) + Meerkat (agent harness) + an AI coding agent. 12 experiments from first hypothesis to submission, with full provenance and documented dead ends. See README for complete experiment timeline and lineage.

[lukacf]

Follow-up: No-GPTQ Validation

We also validated the result without GPTQ (using percentile search quantization instead), confirming the n-gram technique is robust to the quantization method:

With GPTQ (submitted above)

Seed	N-gram BPP	Size
1	1.0240	15.79 MB
2	1.0247	~15.79 MB
3	1.0242	~15.79 MB
Mean	1.0243
Std	0.0004

Without GPTQ (percentile search only)

Seed	N-gram BPP	Size
1	1.0245	15.80 MB
2	1.0275	15.90 MB
3	1.0241	16.16 MB
4	1.0244	15.84 MB
Mean	1.0251
Std	0.0015

The n-gram improvement is consistent regardless of quantization method — GPTQ vs percentile search changes the result by only ~0.001 BPP. The no-GPTQ variant uses full 600s for training with no post-training calibration step.

Happy to provide the no-GPTQ train_gpt.py if useful for verification.