Introduction to LLM architecture for mathematicians, and an interactive demo of GPT-2

Life cycle of words and tokens behind the chat box

For the Stanford MATH 289 lectures on LLM architecture, I made a small interactive page with Claude Code to demonstrate the life cycle of words and tokens during training and inference:

GPT-2 small, stage by stage →

Lecture notes

The lecture notes are Google Colab notebooks: LLM architecture I, LLM architecture II.

They cover each component of the architecture words → tokens → vectors → $L\times$[attention + MLP (with Add and Norm)] → output layer → loss or next token and discuss some common modern design practices.

Philosophies of DL and LLMs

1. Represent the information, goals, and heuristics with linear-algebraic structures, which are controlled by a finite set of parameters.
2. Only the linear parts are learnable; the nonlinear parts are fixed during training. By the universal approximation theorem and real-life observations, this is enough to approximate most functions and tasks, even though the nonlinear parts are fixed in advance.
3. Believe that the machine can magically learn and make your wish come true, given a reasonable loss function (a proxy for the success rate on tasks), a reasonable optimizer, a huge amount of data, and some careful engineering. That is, if the machine did not learn well, retry with more engineering.