Double descent
Make a model bigger and its test error falls, then rises — the textbook U — and then, past the point where it can fit its training data exactly, falls again. Why the peak appears, why the second descent happens, and why a little regularisation makes the peak vanish.
45 min
After this lesson you can
- Describe model-wise, epoch-wise and sample-wise double descent and where each was observed
- Derive, with the SVD, why the least-squares fit explodes at the interpolation threshold p = n
- Explain why the minimum-norm interpolant improves as the number of parameters grows
- Show why ridge regularisation removes the peak, and state the main critiques of double descent
In this lesson
- The classical and the modern regime
- Why the peak: dividing by almost zero
- Why the second descent: the smallest interpolant
- Regularisation removes the peak
- Deep double descent
- The counterpoint: are we counting parameters right?
- Code: the whole experiment in NumPy
- The four questions
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.