Grokking
A small transformer memorises its training set within a hundred steps, looks hopeless on new inputs for thousands more, and then suddenly generalises perfectly. We run it, open it up, find the Fourier algorithm it learned, and compare the competing explanations.
50 min
After this lesson you can
- Describe grokking and the conditions under which it appears (data fraction, weight decay)
- Explain the Fourier multiplication algorithm that networks learn for modular addition
- Read the memorisation, circuit-formation and clean-up phases from training data
- Compare the main explanations — weight norm, circuit efficiency, pattern-learning speed, lazy-to-rich — and say what each predicts
In this lesson
- The discovery
- What the network learned: a Fourier algorithm
- Watching the algorithm form in our run
- Three phases
- Why? Competing explanations
- Code: the model in PyTorch
- The four questions
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.