makemore — character-level name generator

A character-level autoregressive language model that generates new, name-like words, following Andrej Karpathy's makemore series. Trained on a dataset of ~32k names.

What's in this repo

• makemore.ipynb — bigram (counting) model: the simplest baseline, predicting the next character from just the previous one.
• makemorept2.ipynb — full bigram training loop plus a small neural network version of the same idea.
• makemoreFV.ipynb — final, cleaned-up version: a WaveNet-style hierarchical MLP (Karpathy's Part 5 architecture), with my own experiments on top (see below).

Architecture (final version)

Instead of concatenating the full character context into one hidden layer at once, the model fuses context characters pairwise in stages (a shallow tree-like structure), inspired by DeepMind's WaveNet. This lets the network build up structure over a longer context more effectively than a flat MLP.

Baseline results (from the original notebook)

Change	Params	Train loss	Val loss
Original (3-char context, 200 hidden)	12K	2.058	2.105
Context 3 → 8	22K	1.918	2.027
Flat → hierarchical	22K	1.941	2.029
Fix batchnorm bug	22K	1.912	2.022
Scale up (n_embd 24, n_hidden 128)	76K	1.769	1.993

My experiment: batch size vs. generalization

Starting from the scaled-up model above, I tried increasing batch size from 32 → 64 to speed up training, and tracked how it affected generalization:

Change	Train loss	Val loss
Batch size 32 (baseline)	1.769	1.993
Batch size 64 (plain)	1.668	2.122
+ weight decay, step LR decay	—	2.033
+ cosine LR decay w/ warmup	—	2.031
+ cosine decay, higher LR	—	2.021
+ stronger weight decay	1.751	1.996

Takeaway: the larger batch size initially hurt validation loss (classic overfitting signature — lower train loss, higher val loss) despite trying several learning-rate schedules (step decay, cosine decay, warmup, higher peak LR) to fix it — none of which meaningfully moved val loss. The actual fix was increasing weight decay, which brought val loss back in line with the batch-32 baseline while training on larger batches. This isolated regularization strength, not the LR schedule, as the real lever for this specific change.

Sample generated names

arlij
chetta
heago
rocklei
hendrix
jamylie
broxin
denish
marianah
astavia
annayve
jayce
niyelle
jaylene
aiyan
aubreana

Running it

Open makemoreFV.ipynb and run all cells top to bottom. Requires PyTorch.

Notes

Karpathy's own experiment log (context length, flat→hierarchical, batchnorm fix, scale-up) is preserved above as the baseline his notebook arrives at — the batch size / weight decay ablation below it is my own addition on top of that architecture.