Backpropagation
The chain rule, applied backwards through a computational graph — why that is efficient, the full matrix derivation for a two-layer network with softmax cross-entropy, and how to check it with finite differences.
50 min
After this lesson you can
- Run a forward and a backward pass by hand on a computational graph, using "local derivative × upstream gradient" at every node
- Explain why reverse mode computes all parameter gradients for about the cost of a few forward passes, while finite differences need one pass per parameter
- Derive every gradient of a two-layer MLP with softmax cross-entropy in matrix form, checking each shape
- Implement backpropagation in NumPy and verify it with a centred finite-difference gradient check
In this lesson
- Computational graphs
- Why backwards? The cost of the two directions
- The full derivation: a two-layer MLP with softmax cross-entropy
- Checking your gradients
- Backprop by hand, in NumPy
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.