Training in practice
Why deep networks were hard to train, and the toolkit that fixed it — initialisation that preserves variance, momentum and Adam, learning-rate schedules, normalisation, dropout, and residual connections.
50 min
After this lesson you can
- Explain why weights cannot start at zero, and derive the Xavier and He initialisation scales from a variance argument
- Recognise vanishing and exploding activations and gradients, and connect them to products of many layer Jacobians
- Decode the update rules of momentum, RMSProp, Adam and AdamW, and say what problem each one addresses
- Describe warmup and cosine schedules, batch and layer normalisation, dropout, and residual connections — what each does, and what is still debated about why it works
In this lesson
- 1. Where to start: initialisation
- 2. How to step: momentum, RMSProp, Adam
- 3. How big a step, over time: schedules
- 4. Keeping activations in range: normalisation
- 5. Regularising by noise: dropout
- 6. A shortcut for the gradient: residual connections
- Putting it together
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.