Gradient descent
Solve the same least-squares problem by walking downhill — derive the update rule, the exact learning-rate limit (2 over the largest curvature), why elongated bowls make descent zig-zag, and why the stochastic version is the one that scales.
40 min
After this lesson you can
- Derive the gradient-descent update and explain why the negative gradient is the direction of steepest descent
- Derive the exact stability condition for gradient descent on a quadratic loss
- Explain how the condition number of the Hessian causes zig-zagging, and how feature standardisation fixes it
- Implement full-batch, minibatch and stochastic gradient descent, and say what each trades away
- Decide when the closed form beats gradient descent and when it does not
In this lesson
- Which way is downhill?
- The picture
- The exact analysis on a quadratic
- Condition number: why the bowl’s shape sets the speed
- Stochastic and minibatch gradient descent
- Momentum, briefly
- Closed form or gradient descent?
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.