Cross-entropy, KL and perplexity
What it costs to code the world with the wrong probabilities. Cross-entropy is that cost, KL divergence is the waste, perplexity is the same number as a count of choices, and minimising cross-entropy on text is exactly how every GPT is trained.
45 min
After this lesson you can
- Define cross-entropy as the average code length when data from p is coded with a code built for q
- Define KL divergence, prove it is never negative (Gibbs' inequality), and show it is not symmetric
- Convert between cross-entropy in bits or nats and perplexity, and read perplexity as an effective number of choices
- Explain why minimising cross-entropy on text is maximum likelihood, and is the training loss of GPT-style models
In this lesson
- Every model is a code
- Cross-entropy: coding p with q’s code
- KL divergence: the price of being wrong
- Three facts about KL
- From distributions to data: the loss of every language model
- Perplexity: surprise as a number of choices
- Is there a floor?
- The code
- In the papers
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.