GPT-2, stage by stage

Type a sentence and follow it through every stage of GPT-2 small (the 124-million-parameter model of 2019): words → tokens → vectors → block 1, …, block 12 → output layer → loss. The model runs inside this page (its weights, 167 MB, are downloaded once and then cached by your browser), so every number you see is computed from your sentence. The diagram on the left shows where in the architecture you are, and the panel on the right lists the hyperparameters that matter at that stage. Companion page for the MATH 289 lectures on LLM architecture.

try:

loading …

← → keys also move; click a box in the diagram to jump there. The last stage is the loss in training mode and sampling in inference mode.

Stage 0 of 16 — words

Hyperparameters of GPT-2 small

vocabulary $V$50,257 tokens

tokenizerbyte-level BPE: 256 bytes + 50,000 merges + 1 special token

vector dimension $d$768

context length1,024 positions (generation stops when the context is full)

positionslearned table $P$, 1,024 × 768 (added once, before block 1)

blocks $L$12, identical in shape, different weights

heads $h$12 per block, each $d_k = 64$ ($h \cdot d_k = d$)

MLP width$4d = 3{,}072$, GELU (tanh form)

normalizationLayerNorm, pre-norm, $\varepsilon = 10^{-5}$

output matrixtied to $E$: logits $= z\,E^{\mathsf T}$

parameters124.4M = $E$ 38.6M + $P$ 0.8M + 12 × 7.09M (per block: attention 2.36M, MLP 4.72M)

trainingnext-token cross-entropy on WebText (40 GB of web text), 2019

inferenceweights frozen; one forward pass per generated token; temperature $\tau$ rescales the logits before the softmax

Here: the same weights, 167 MB (blocks rounded to int8, $E$ in 16-bit floats), run by ONNX Runtime Web in your browser; the numbers agree with the full-precision model to about 0.01.

`?`	Toggle this help
`/`	Search
`f`	Link hints (vim-like)
`t`	Toggle dark mode
`j` / `k`	Scroll down / up
`g` / `G`	Top / bottom
`o`	Jump back
`l`	Cycle language (en→zh→fr)
`H` / `L`	History back / forward
`r`	Reload
`F`	Fullscreen
`i`	Idle in the Matrix
`a`	ASCII Aquarium
`Esc`	Close / cancel

Stage 0 of 16 — words

Keyboard Shortcuts