TL;DR
Building a GPT from Scratch: Let's Reproduce GPT-2 (and ChatGPT?)
We build a character-level Transformer language model from scratch using PyTorch, training on Tiny Shakespeare, and explain the architecture of GPT.
Andrej Karpathy · 1h 56m · about 3 min to read · about 1h 53m saved · 8 key points · timestamped
- GPT stands for Generative Pre-trained Transformer and is a decoder-only Transformer trained autoregressively.
- The Transformer architecture was introduced in the 2017 'Attention is All You Need' paper.
- Self-attention allows each token to attend to all previous tokens via queries, keys, and values.
- Multi-head attention runs multiple attention heads in parallel and concatenates their outputs.
The lecture begins with a demonstration of ChatGPT and explains that its underlying architecture is the Transformer, introduced in the 2017 paper 'Attention is All You Need'. The goal is to build a simpler, character-level version of GPT using PyTorch, trained on the Tiny Shakespeare dataset. The process starts with tokenization, where text is converted to integer sequences using a vocabulary of 65 characters. Data is split into training and validation sets, and batches of fixed-length chunks are created for training. A bigram language model is implemented first, which predicts the next character based only on the current character. This baseline achieves a validation loss of about 2.5. The core innovation is self-attention: tokens communicate by emitting query, key, and value vectors. The attention scores are computed as dot products between queries and keys, masked to prevent future tokens from attending to past ones, then softmaxed and used to aggregate values. This is efficiently implemented using matrix multiplication with a lower triangular mask. Multi-head attention runs multiple self-attention heads in parallel, concatenating their outputs. A feed-forward network (a simple MLP) is added after attention for per-token computation. Residual connections and layer normalization are introduced to stabilize training of deeper networks. The model is scaled up to 384 embedding dimension, 6 heads, 6 layers, and dropout of 0.2, achieving a validation loss of 1.48 on Tiny Shakespeare. The resulting generated text is nonsensical but resembles Shakespearean style. The lecture concludes by contrasting the decoder-only Transformer (used in GPT) with the encoder-decoder architecture from the original paper. The pre-trained model is a document completer; to turn it into a helpful assistant like ChatGPT, additional fine-tuning stages are needed, including supervised fine-tuning and reinforcement learning from human feedback (RLHF). The full code is available in the nanoGPT repository.
- Introduction and MotivationChatGPT demo; language models; plan to build a GPT-like Transformer.
- Setup and TokenizationDownloading Tiny Shakespeare; character-level tokenizer; encoder/decoder.
- Data Loading and BatchingTrain/val split; batch and block size; creating input and target tensors.
- Bigram Language ModelSimple bigram model using embedding table; loss calculation; generation.
- Efficient Aggregation with Matrix MultiplyMathematical trick to compute averages of past tokens using lower triangular matrices.
- Self-Attention MechanismQueries, keys, values; dot product affinities; masking; softmax; weighted aggregation.
- Single Head Attention ImplementationImplementing one head of self-attention in PyTorch.
- Multi-Head AttentionRunning multiple heads in parallel and concatenating outputs.
- Feed-Forward and Transformer BlockAdding a simple MLP after attention; combining into a block.
- Residual Connections and Layer NormSkip connections; pre-layer normalization for stable training.
- Scaling Up the ModelIncreasing batch size, block size, embedding dimension, heads, layers; adding dropout.
- Results and Generated TextValidation loss 1.48; sample output resembling Shakespeare.
- nanoGPT Code WalkthroughOverview of model.py and train.py; comparison to lecture code.
- From Pre-training to ChatGPTComparison to GPT-3; need for fine-tuning and RLHF to align the model.