Podracing II: Electric Bugaloo — 0.9625 BPB (3-seed mean, all sub-0.964)

[newjordan]

Results

Seed	Sliding BPB	7-gram Backoff BPB	Artifact
42	1.1210	0.9631	15.59 MB
2045	1.1196	0.9620	15.71 MB
7	1.1202	0.9624	15.59 MB
Mean	1.1203	0.9625	—

Progression

PR	Mean BPB	Notes
#190	—	The Stinky Frost Recipe
#390, #401	1.1295, 1.1243	Sponge Bath TTT + EMA/SWA/QAT
#445	1.1236	Late Training Replay + GPTQ-lite
#498, #499	1.1378	The Frugendorff
#508, #578	1.1215	GPTQ + Early QAT + Legal TTT
#533, #577	1.1207	GPTQ + Short TTT
#587	1.1208	XSA + quantization tuning
#656	1.1195	Three Breadsticks
#706	1.0461	Podracing I (fixed 5-gram)
#753	0.9625	Podracing II (backoff + adaptive)

What Changed vs Podracing I

Two eval-time improvements, no training changes:

1. Multi-order backoff (2-7): longest context first, cascade on miss
2. Entropy-adaptive alpha: trust n-gram more when model is uncertain

Compliance

• Score-first, backward-looking cache
• Alpha from model entropy only — no target access
• GPTQ calibration inside training phase
• Training logs + submission.json included

Credits

• N-gram eval cache: @deanbrr (Record: 5-gram Eval Cache + LeakyReLU² + Parallel Muon val_bpb: 1.0920 (3-seed mean, std 0.0007) | ~15.9 MB | 8×H100 SXM #659)
• Backoff + adaptive alpha: @Asukabot0 (Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727)
• Base architecture: @signalrush (Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414)

Reproduce

SEED=2045 MLP_ACT=leaky_relu_sq MLP_LEAKY_SLOPE=0.5 XSA_LAST_N=4 BIGRAM_VOCAB_SIZE=1536 ROPE_DIMS=24 NGRAM_EVAL_ORDER=7 NGRAM_EVAL_ADAPTIVE=1 TTT_EVAL_ENABLED=0 torchrun --nproc_per_node=8 train_gpt.py

8xH100 SXM, 600s training + ~140s eval.