The home of the napkin-gamemaster series — five one-file PyTorch experiments on a 6GB laptop GPU, each answering one sharp question and buying one measured lesson, building to this repo's finale: real Atari Pong, through the actual ALE emulator, from raw pixels, beaten in fourteen minutes on the laptop — with the run-to-run variance that headline hides reported next to it.
Every repo in the series follows the same rules: one file, one registered hypothesis written before the results, a selfcheck that asserts the claims instead of assuming them, ≥5 seeds with IQM + bootstrap CIs (ties reported as ties), an INSIGHTS.md of what actually broke, and end-to-end reproducibility in minutes-to-hours on the same 6GB card. The finale runs 3 draws per arm rather than 5 — a declared exception for a 14-minute-per-run grid — and reports censored takeoff counts instead of hiding the sample size in a mean.
The agent (green paddle) playing the same 2KB cartridge binary shipped in 1978. Rendered by play from the committed checkpoint's training run.
The goal: a little AI trained on a 6GB laptop — in one evening — beats real Atari Pong, the actual 1978 cartridge running in an emulator, seeing only the screen. No cloud farm, no secret sauce, and every design decision measured rather than copied from a textbook. Five small projects, each answering one question the next one needs.
1. napkin-returns — which parts of the learning recipe actually matter? The textbook teaches a ladder of four increasingly clever upgrades for turning experience into improvement. Built all four, ran them twenty times each: the celebrated statistical tricks bought a rounding error, while one humble change — study each batch of experience four times instead of once — multiplied the score by six. Bonus discovery: single test runs lie; close competitors need ~15 repeats before you can honestly name a winner. That discipline became the series' law.
2. napkin-pixels — what does it cost to see instead of know? The final AI must play from the screen, not from the game's internal numbers. The cost turned out brutal: picture-learners plateaued below half of what a numbers-learner reached. A four-frame flip-book (so motion is visible) is non-negotiable; learnable eyes are non-negotiable (frozen random ones scored zero — a Pong ball is nine bright pixels in a sea of four thousand, and random circuitry smears the needle into invisibility); the fashionable "also learn to redraw the screen" homework was a dead tie — skipped.
3. napkin-replay — which ingredient of the classic Atari recipe is load-bearing? The famous DQN recipe ships four add-ons everyone copies blindly: the experience diary, the frozen answer key, the second-opinion skeptic, the multi-step lookahead. Delete one at a time and see what breaks — plus dial how hard the diary gets re-studied, betting repo 1's "reuse wins" law holds off-policy too. The verdict was humbling for the textbook: two of the four celebrated add-ons were dead ties — only the diary itself (deleting it cost 58%) and multi-step lookahead (25%) were load-bearing. And re-studying harder stopped paying almost immediately: four and eight times the study effort scored the same as one.
4. napkin-dreams — can the AI practice inside its own imagination? Learn the game's physics by watching, then train inside a mental simulation — thousands of imaginary rallies per real one. The catch: dreams drift, and the player learns to exploit dream glitches that don't exist in reality. Dreamers with 3-, 15- and 45-step imaginations measured how long a dream stays trustworthy. It lost badly — thirteen times less efficient than plain learning — and a follow-up probe found the reason, which was not the idea but this particular dream: its sense of when a rally ends is worse than a coin-flip guess after five imagined steps. So the finale plays the real game, not a dreamed one, and says why.
5. napkin-gamemaster — this repo. Everything above becomes one recipe, pointed at real Pong. It won 21–10 after fourteen minutes of training on the laptop — the evening budgeted for it was never needed. Then each inherited lesson was removed and rerun to see which ones actually carried the win. One did: study each batch four times, and deleting it stopped learning dead. One didn't, and not merely by being harmless: deleting the four-frame flip-book that was mandatory in project 2 made learning start roughly 40% sooner, every run. The standard image-cleanup step everyone copies from the Atari canon already smears a moving ball across two positions, so the motion was in the picture all along and the extra frames were paying rent for nothing. And the sobering part: run the exact same recipe six times and it wins five times and completely fails once, so any single run — including a triumphant one — is a story, not a measurement.
The thread through all five: build the smallest honest experiment, write the prediction down before the result, let a self-test prove the code isn't lying, and publish whichever way it lands. The scoreboard on the registered predictions so far is roughly half right, half instructively wrong — which is exactly the point.
| # | repo | question | the lesson it bought |
|---|---|---|---|
| 1 | napkin-returns | which rung of the PG ladder is load-bearing? | The famous rungs (baseline, GAE) bought ~nothing; 4 epochs of data reuse bought 6× and the clip was free insurance. GAE under delayed reward is worse than no critic. Close winners are ties — naming a unique best took 15 seeds. |
| 2 | napkin-pixels | what does seeing cost? | Pixels are a ceiling, not a sample tax (42% of privileged-state performance at matched steps). Framestack and learned convs are load-bearing; frozen random features collapse on sparse frames; reconstruction auxiliaries are a tie. |
| 3 | napkin-replay | which DQN ingredient is load-bearing? | Only the replay buffer (−58%) and 3-step returns (−25%) mattered; the target network and double-Q were ties. Reuse saturates at replay ratio 1 — ratios 4 and 8 bought nothing for 4–8× the FLOPs. |
| 4 | napkin-dreams | how far can you trust a learned model? | Dreaming cost 13× the real frames vs model-free DQN, and no horizon won. A trust probe located the failure: the model's done head is worse than a constant from dream step 5. Verdict scoped to this model class → the finale goes model-free. |
| 5 | napkin-gamemaster (this repo) | can the lessons beat a real game in one evening? | Pong beaten in 14 minutes (21–10), 5 of 6 runs of the same recipe. Of the two inherited lessons, data reuse transferred (removing it: 0/3 runs learned at all) and the framestack did not (removing it cost nothing — the deflicker step already leaks motion into single frames). Outcomes are bimodal, so the finale reports steps to takeoff, censored. |
1. Lineage choice, decided by the upstream numbers, not by taste. On-policy PPO, model-free. napkin-dreams measured dreaming at 13× worse real-frame efficiency than model-free DQN on the same env and located the fault at the world model's done head, so the dream lineage is out on its own evidence. Between the two model-free lineages, napkin-replay's DQN ingredients are largely value-based-only (target networks and double-Q have no PPO analog, as the series plan said up front), while napkin-returns' load-bearing finding — data reuse, 6× — is exactly a PPO knob. PPO also carries the repo-1/2 implementation forward verbatim, which is what makes the leave-one-out an honest test of transfer rather than of a rewrite.
2. The headline run — beat Pong from raw pixels inside the declared one-evening exception.
3. Leave-one-lesson-out, one arm per inherited lesson, each changing exactly one thing:
| arm | what is removed | which repo's lesson |
|---|---|---|
full |
nothing — every lesson applied | — |
no-reuse |
4 PPO epochs → 1 | napkin-returns (data reuse, the 6× finding) |
no-stack |
4-frame stack → 1 frame | napkin-pixels (the flip-book) |
no-motion |
4 frames → 4 copies of the newest frame | follow-up: same tensor shape and parameter count as full, zero history — added after no-stack's result to separate the stack's information from the cost of a 4× wider input |
- The headline run beats Pong (score > 0; stretch ≥ +15) within 1.5M agent steps.
- In the leave-one-lesson-out: removing data reuse hurts most (
no-reuse, napkin-returns' 6× lesson), removing the framestack second (no-stack, napkin-pixels' provably-blind-to-velocity lesson). Both should be decisive, not marginal, at 500k-step budget. - The async collector (this repo's own contribution) is worth ≥1.5× wall-clock over sync at 32 envs —
benchreports the measured number.
(Prediction 2 named the budget as well as the ranking, and both halves turned out to matter — see Results. no-motion was designed after seeing no-stack's outcome and is labelled as the follow-up it is, not as a registered prediction.)
1.5M agent steps (6M raw frames) at ~1,780 steps/s on the RTX 4050 — 14 minutes, not the evening the series budgeted for. The committed checkpoint's game is the gif above: 21–10.
But one run is not a result, and this is where the finale got its sharpest lesson. Six runs of the identical full recipe at 1.5M — three headline runs (two of them on a different GPU) plus the leave-one-out's three — ended at +16.2, +15.8, +19.6, +10.9, +0.4, −20.7. Five of six beat Pong; one never left the floor. The distribution is not a spread around a mean, it is bimodal: by 1.5M a run has either passed the ignition point or is still at −20, and nothing sits in between.
And the failing run is not a "bad seed". Two of those runs are the same seed with the same config (+10.9 and −20.7), with curves already differing by update 10. The cause turned out not to be the obvious suspect: --sync runs are also irreproducible at a fixed seed, so it is not the async collector's thread interleaving. It is nondeterministic CUDA kernels — they perturb the weights at ~1e-7, the perturbation feeds back through action sampling, and at a budget where outcomes are bimodal that decides whether the run learns Pong at all. repro demonstrates both halves in one command, and set_deterministic() + --sync makes runs bit-identical (verified). So every "seed k" here is a draw, not a fixed point.
Hypothesis 1: met, with the caveat it didn't ask for. Score > 0 in 5 of 6 runs, stretch (≥ +15) in 3 of 6, well inside budget. The prediction quietly assumed a single run answers the question; at this budget it doesn't.
Hypothesis 3: refuted, and it cost more than it bought. The async collector was predicted to be worth ≥1.5× wall-clock at 32 envs. Measured by bench: 1,945 vs 1,884 steps/s — 1.03× over 123k steps (1.09× over 25k, where fixed emulator-startup flatters it). Three percent. And the same overlap that buys those three percent is what made me suspect the collector for the irreproducibility above — it was innocent, but a mechanism that complicated the diagnosis and returns 3% is a bad trade. Kept, documented, and not recommended: --sync costs 3% and is the path repro can make deterministic.
Because outcomes are bimodal, a mean over runs lands in a gap where no run lives (full's three LOO runs average −0.2, a score none of them scored). The summary that survives is steps to takeoff — the first checkpoint at a non-losing score — with runs that never got there recorded as censored at the budget, not averaged in. plotloo reports both, and assets/loo.json has every number.
| arm | final scores (3 runs, last 10%) | took off | leaves the floor (≥ −15) | stops losing (≥ 0) |
|---|---|---|---|---|
full |
−20.7 / −0.7 / +16.7 | 2 / 3 | 0.97M, 0.99M, never | 1.06M, 1.50M, never |
no-reuse |
−19.3 / −20.1 / −20.4 | 0 / 3 | never ×3 | never ×3 |
no-stack |
+18.7 / +16.3 / +18.5 | 3 / 3 | 0.57M, 0.59M, 0.59M | 0.89M, 0.95M, 1.20M |
no-motion |
+8.5 / +10.9 / +6.8 | 3 / 3 | 0.69M, 0.85M, 1.13M | 0.99M, 1.17M, 1.32M |
Read the plot before the table. Every arm that learns at all is still rising at 1.5M — none has plateaued. So the final-score column is mostly a readout of when a run started learning, not of how good it eventually gets, which is why the two onset columns are the honest comparison and why no-motion's lower finals are a later start rather than a lower ceiling.
Data reuse is the load-bearing lesson, and it transferred exactly. no-reuse (1 PPO epoch instead of 4) took off in 0 of 3 runs and sat at the floor for the entire budget. napkin-returns measured reuse as worth 6× on CartPole; on real Pong at this budget it is the difference between learning and not learning at all. It is also the same arithmetic that had already bitten this repo once from the other direction — a 4× loss of gradient steps per sample, caused by the minibatch rather than the epoch count, is what made the first headline run fail (INSIGHTS #1).
The framestack lesson did not transfer — prediction 2 is refuted by sign reversal. Removing napkin-pixels' 4-frame stack was predicted to hurt decisively, second only to reuse. Instead no-stack was the fastest arm: all three runs left the floor at 0.57–0.59M steps against full's 0.97–0.99M, with no overlap between the arms and full's third run never starting at all. Removing the stack made learning begin roughly 40% earlier, every time. (Three draws per arm: enough for a non-overlapping ordering on onset, not enough for a confidence interval on it.)
Why: the mechanism is in a line every arm inherits. The standard Atari deflicker step max-pools the last two raw frames; on a moving ball that unions two positions into one smear. motion measures it: the ball covers 2.99 pixels deflickered versus 2.00 for a single position, and 62% of single frames carry the ball's displacement. So no-stack removes the stack's extra history, not all motion — napkin-pixels' mini-Pong had no such leak, which is exactly why its stack was load-bearing and this one isn't. Guard against the obvious wrong explanation: a scripted policy that reads the ball's y from one frame and chases it scores −20.0 (random ≈ −21), so this is not "Pong is trivially reactive" — the learned policy is doing something a position-tracker isn't.
And the follow-up arm splits the credit. no-stack changes two things at once — the information and the input width — so no-motion holds the information at exactly one frame while restoring full's tensor shape and parameter count (four copies of the newest frame). It learned in 3 of 3 runs, which kills any suggestion that the stack's history is necessary. But its onset (0.69–1.13M) sits with full rather than with no-stack: four identical copies of one frame are almost as slow to start as four real frames, despite carrying no more information than a single frame. The delay therefore tracks the width of the observation, not the content of the history. Which specific consequence of that width does the damage — 4× the first-layer parameters, or the 4× larger activations that identical channels produce at initialization — this grid does not separate, and it is left as an open question rather than a story.
N parallel ALE instances behind one batched interface: minimal action set, frame-skip 4 with reward summed across skipped frames, flicker max-pool over the last two raw frames, grayscale → crop → 2× subsample → 80×80 uint8, framestack 4. selfcheck proves: bit-identical determinism over 200 steps; frame-skip conserves reward against a manually stepped reference emulator; the framestack is replay-verified; the two ablation observations are exactly what they claim (1 frame; 4 copies of 1 frame); the takeoff statistic censors a run that never crosses instead of scoring it at the budget; GAE identities against an independent recursion; random play scores −20.8 per game (losing, as it should).
One inherited line turned out to be doing more than flicker removal: max-pooling the last two raw frames also unions the ball's two positions, so a single frame is not motion-free — measured by motion, and the reason no-stack survives (see Results). Preprocessing you copy from the canon is part of the experimental design whether you meant it to be or not.
pip install --target .deps "numpy<2" ale-py
PYTHONPATH=.deps python3.10 napkin_gamemaster.py selfcheck # ~4 min
PYTHONPATH=.deps python3.10 napkin_gamemaster.py train # checkpointed, resumable
PYTHONPATH=.deps python3.10 napkin_gamemaster.py play # render a full game, score overlay
PYTHONPATH=.deps python3.10 napkin_gamemaster.py loo # leave-one-lesson-out grid (resumable)
PYTHONPATH=.deps python3.10 napkin_gamemaster.py plotloo # curves + steps-to-takeoff
PYTHONPATH=.deps python3.10 napkin_gamemaster.py motion # is one frame motion-free? (2 min, CPU)
PYTHONPATH=.deps python3.10 napkin_gamemaster.py bench # async vs sync throughput
PYTHONPATH=.deps python3.10 napkin_gamemaster.py repro # what is and isn't reproducibletrain --sync --deterministic is the bit-reproducible path (3% slower); the default is not
reproducible at a fixed seed, and repro shows you that rather than asserting it.
No gym wrapper stack — the env layer is ~80 asserted lines against ale-py directly. No reward clipping (Pong is already ±1), no lives handling (Pong has none), no evaluation-mode protocol beyond play (training curves are the measurement, as everywhere in the series). No hyperparameter search: every number is either inherited from a measured upstream lesson or the series' long-standing default.
And three limits worth naming rather than burying. The leave-one-out runs 3 draws per arm, which is enough to separate "never learns" from "learns" (no-reuse) and enough for a non-overlapping ordering on onset, but nowhere near enough for a confidence interval on the size of that ordering — so no effect size is quoted for no-stack over full. Runs are not reproducible at a fixed seed unless you ask for it: nondeterministic CUDA kernels are enough to flip an outcome at this budget, and set_deterministic() + --sync is the path that is bit-identical (repro proves it). And this is one game: Pong's deflicker-leaks-motion result is a fact about Pong's sprites, not a general licence to drop framestacking.
Nature-style CNN sized for 80×80 (conv 8×8/4 ×32, 4×4/2 ×64, 3×3/1 ×64 → fc 512), shared policy/value heads. PPO: γ=0.99, λ=0.95, clip 0.2, 4 epochs (napkin-returns' load-bearing lesson), minibatch 256, Adam 2.5e-4, 32 envs × 128 horizon, value coefficient 0.5. The minibatch is not a free parameter: it is what keeps gradient-steps-per-sample at the reference value once the rollout is 32 envs wide (INSIGHTS #1). Observation: 4-frame stack for the headline run — and see Results for what happened when it was removed.