SmallGPT

A from-scratch implementation of character-level language models in PyTorch, built for learning how GPTs work under the hood. Follows Andrej Karpathy's "makemore" / nanoGPT series. Trained on the Tiny Shakespeare dataset.

Files

File	Description
bigram_model.py	Minimal bigram language model — predicts the next character using only the previous one via a learned embedding table
transformer_model.py	Full GPT-style transformer — multi-head self-attention, feed-forward blocks, residual connections, and layer norm
tiny_shakespeare.txt	Training corpus (~1MB of Shakespeare text)
sample_output.txt	Text generated by the trained transformer model

Models

Bigram

The simplest possible language model. Each character predicts the next using a single embedding lookup — no context, no attention. A good starting point for understanding how tokens and logits work.

Transformer

A decoder-only GPT with 6 layers, 6 attention heads, and ~10M parameters. Implements the full modern stack: causal self-attention, positional embeddings, residual connections, and pre-norm LayerNorm. Shows how scaling context and depth leads to much more coherent text generation compared to the bigram baseline.

The gap between these two models is the gap between a lookup table and a real language model.