An interactive, real 3,725,177-parameter decoder-only Transformer language model — built from scratch, with every step of prediction, generation, and training exposed to inspection.
Live Demo · Architecture · What You Can See · Running Locally
Type a sentence, click Follow the prediction, and watch it travel through a real Transformer — tokens, IDs, embeddings, attention, layers, scores, probabilities — all the way to the next piece being chosen and appended back into your sentence. No installation, runs entirely in your browser.
This is a real, working decoder-only Transformer — the same architecture family behind GPT-style models, at a scale small enough to fully inspect. It converts text into tokens, looks those tokens up as trainable embeddings, runs them through repeated blocks of causal self-attention and a feed-forward network, produces a score for every token in its vocabulary, turns those scores into probabilities, and samples the next piece. Every one of those steps is hand-implemented in plain JavaScript — no PyTorch, no TensorFlow, no autodiff library — and every number the UI shows (token IDs, embeddings, attention weights, logits, probabilities, gradients, parameters, losses, generated tokens) is read live from that real model, not precomputed or faked.
Your text
↓
Tokens (BPE)
↓
Token IDs (lookup addresses)
↓
Embeddings (256-dim, trainable)
↓
Causal self-attention × 4 layers
↓
Logits (one score per vocabulary token)
↓
Softmax → probabilities
↓
Next token chosen → appended to your text
↓
Repeat
The app walks one real prediction through the whole model, then opens up training, pre-training, fine-tuning, and full internals for anyone who wants to go deeper:
- Tokens → IDs → embeddings — your text broken into real BPE pieces, each with a real learned 256-number vector
- Attention — real per-position attention weights, causal-masked, for the exact sentence you typed
- Layers — the same position's attention pattern re-examined across all 4 real transformer blocks
- Scores → probabilities — real logits for every candidate next piece, transformed into a real softmax distribution
- Generation — the same predict → choose → append loop, automated, using real streamed output (greedy or sampled)
- Training — a real example, a real forward pass, real loss, a real gradient, a real parameter update — then the same prediction run again to see the change
- Pre-training — the real checkpoint history (3,000 real training steps; real start/end loss; real held-out loss) behind every other section
- SFT — a real supervised fine-tuning run showing the actual trade-off: the targeted behavior improving while a real held-out measurement regresses
- Engineering / Open the machine — every block's real parameter count and dimensions, free exploration of real attention across every layer and head, and a training lab to run more real steps yourself
Vocabulary: 2,169 tokens (byte-pair encoding)
Embedding: 2,169 × 256 (tied with output projection)
Context length: 32 tokens
Layers: 4 × [LayerNorm → causal 8-head self-attention → residual
→ LayerNorm → feed-forward (GELU, d_ff=1,024) → residual]
Heads: 8 (head dim 32)
Output: tied embedding projection + bias → softmax
Loss: cross-entropy
Optimizer: SGD
Parameters: 3,725,177
No framework, no attention library — causal self-attention, layer normalization, and backpropagation are all hand-derived and independently gradient-checked against the live model.
prediction → loss → gradient → parameter update → repeat
Every training step — in the guided walkthrough, the training lab, pre-training, and SFT — is real, hand-derived backpropagation through the exact same code path, not a framework call or a separate implementation:
- Prediction — a real forward pass produces a probability for every one of the 2,169 vocabulary pieces
- Loss — cross-entropy,
L = -log(p[target]) - Gradient — the analytical derivative of the loss with respect to every one of the 3,725,177 parameters, backpropagated through the output projection, every attention block, and the embeddings
- Update — plain SGD
- Repeat — the pre-trained checkpoint shipped with this app came from 3,000 real steps of exactly this loop
The pre-training, SFT, and "how it learns" sections all show the real before/gradient/after values for a chosen parameter, and the real loss numbers behind the checkpoint — never invented or interpolated data.
Before release, the model and tokenizer were independently audited and numerically re-verified — not just re-run:
3,725,177 parameters, computed from the live model, not asserted
Forward pass and backpropagation independently re-derived, gradient by gradient
Full numerical gradient check across every parameter category
Causal masking verified: every future-position attention weight is exactly zero
Deterministic reset — identical results across repeated runs from the same seed
Browser regression testing at 1440px, 1280px, 1024px, and 375px
Run the same checks yourself — see Running Locally below.
tiny-transformer-app/
├── src/
│ ├── model/ Pure math: config, forward/backward pass, attention,
│ │ feed-forward, layer norm, loss, gradient checking.
│ ├── tokenizer/ Byte-pair encoding tokenizer, trained on the app's corpus.
│ ├── training/ Training-step orchestration on top of src/model.
│ ├── worker/ The Web Worker that owns the live, trainable model —
│ │ every prediction/train/generate call goes through it.
│ ├── hooks/ React hooks bridging the UI to the worker.
│ └── components/ Presentation — the Rail, attention visuals, training
│ and inspection panels. Call into the worker rather
│ than reimplementing any math.
├── scripts/ Node-side training, generation, and validation —
│ the same source of truth the browser build reads.
├── checkpoints/ Real trained weights (pretrained + SFT), used by
│ the Node scripts.
└── public/checkpoints/ The same real weights, served to the browser at runtime.
screenshots/ Screenshots used in this README.
.github/workflows/ CI: lint, validate, audit, build, deploy to GitHub Pages.
cd tiny-transformer-app
npm install
npm run devThen open the printed local URL.
npm run build # production build
npm run lint # eslint
npm run validate:tokenizer # scripts/validate-tokenizer.mjs
npm run validate:transformer # scripts/validate-transformer.mjs
npm run validate:full-model # scripts/validate-full-model.mjs
npm run audit # scripts/audit-full-check.mjs — the full numerical auditReact and Vite — that's the entire dependency list for the application itself. The model, tokenizer, training loop, and gradient checking are plain JavaScript with no ML framework.
This repository's first version was Tiny Language Model — a much smaller (98,100-parameter) word-embedding-and-hidden-layer model with no attention mechanism, built as an earlier step toward this one. It's preserved at tiny-language-model-app/ with its own README, and is no longer the app this repository deploys, but the code and its own audit trail remain exactly as they were.