Newton, quasi-Newton and natural gradients
Use curvature, not just slope — derive Newton's step from a quadratic model, see why it converges quadratically and what that costs, make it safe with line searches and damping, and meet its practical descendants: Gauss–Newton, Levenberg–Marquardt, BFGS, L-BFGS, and the natural gradient with its large-scale approximations.
45 min
After this lesson you can
- Derive the Newton step as the minimiser of the second-order Taylor model, and explain why it is exact on quadratics and affine invariant
- Show why Newton's method converges quadratically near a minimum, and list its costs and failure modes
- Explain damping, backtracking line search and Levenberg–Marquardt regularisation of the Hessian
- Derive Gauss–Newton for nonlinear least squares from the structure of the Hessian
- Derive the secant equation and the BFGS update, and explain how L-BFGS stores it in O(md) memory
- Define the natural gradient with the Fisher information matrix and say what K-FAC and Shampoo approximate
In this lesson
- The Newton step: minimise the model
- Quadratic convergence: digits double
- What it costs, and how it fails
- Gauss–Newton and Levenberg–Marquardt: curvature for free
- Quasi-Newton: learn the curvature from the gradients
- The natural gradient: curvature of the model, not the loss
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.