GPT and next-token prediction
Shannon's game, played by a transformer. A language model assigns a probability to the next token given everything before it; causal masking lets one pass train every position; byte-pair encoding decides what a token is; GPT-1, BERT, GPT-2 and GPT-3 show what scaling this did; and temperature, top-k and top-p turn a distribution back into text.
55 min
After this lesson you can
- Write the chain-rule factorisation of a sequence's probability and the next-token cross-entropy loss
- Explain how causal masking lets one forward pass produce a training signal at every position
- Run byte-pair encoding by hand, and explain why GPT-2 applies it to bytes
- Place GPT-1, BERT, GPT-2 and GPT-3 in order, with their objectives and sizes, and say what in-context learning is
- Implement greedy, temperature, top-k and nucleus (top-p) decoding and predict their effect on entropy
In this lesson
- A language model predicts the next token
- Training: every position at once
- What is a token? Byte-pair encoding
- Scaling up: GPT, BERT, GPT-2, GPT-3
- From a distribution to text: decoding
- Training a small GPT for real
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.