Derivatives, gradients and the chain rule
A derivative is a sensitivity, meaning how much the output moves when you nudge the input. The gradient collects every sensitivity into one uphill arrow, and the chain rule multiplies sensitivities through a pipeline, which is all backpropagation does.
45 min
After this lesson you can
- Read a derivative as a sensitivity and use it for linear approximation
- Compute partial derivatives and assemble them into a gradient
- Prove that the gradient points in the direction of steepest ascent and is perpendicular to contour lines
- Derive the chain rule from nudges, including the sum-over-paths version for several variables
- Check derivatives numerically with finite differences and with PyTorch autograd
In this lesson
- The derivative is a sensitivity
- Many inputs: partial derivatives
- The gradient: all the sensitivities in one arrow
- The chain rule: sensitivities multiply
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.