Scaling laws and their limits
Loss falls as a smooth power law in parameters, data and compute across many orders of magnitude — so smoothly that you can plan a training run from small experiments. How the laws were found, how Chinchilla corrected them, what "emergence" means, and where the laws must stop.
50 min
After this lesson you can
- Write down the Kaplan and Chinchilla scaling laws and say what each term means
- Derive the compute-optimal model size and data from L(N, D) = E + A/N^α + B/D^β under C ≈ 6ND
- Explain why Kaplan et al. and Hoffmann et al. disagreed, and why the published Chinchilla constants need care
- Evaluate claims of emergent abilities, including the metric-choice critique
- Name the concrete limits — irreducible loss, data, test-time compute — and what is known about each
In this lesson
- Power laws, found by accident
- The Chinchilla correction
- The floor: irreducible loss and the entropy of text
- Emergent abilities — or emergent metrics?
- Limits: data, repetition and test-time compute
- The four questions
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.