Width versus depth
With a fixed budget of parameters, should a network be wide or deep? Theory says depth can be exponentially better, experiments say shape often barely matters, and infinitely wide networks turn into kernels. Here is how all three are true at once.
45 min
After this lesson you can
- State the universal approximation theorem and say precisely what it does not promise
- Explain, with Telgarsky's folding construction, why depth can beat width exponentially
- Read results from a matched-parameter experiment and say when width or depth wins
- Explain vanishing gradients and why residual connections fix them
- Define the neural tangent kernel, the lazy regime and µP, and say what each predicts as width grows
In this lesson
- One wide layer can represent anything — in principle
- Depth folds, width adds
- A matched-parameter experiment
- Why deep networks used to be hard to train
- Infinitely wide networks
- Feature learning at infinite width: mean field and µP
- Code: laziness in thirty lines
- The four questions
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.