The CNN lineage: LeNet to ResNet
From a cat's visual cortex to a 152-layer network — how convolutional nets got deep, why depth hit a wall in 2015, and the one-line fix, y = F(x) + x, that let gradients through.
55 min
After this lesson you can
- Trace the chain of ideas from Hubel & Wiesel's simple and complex cells to the Neocognitron, LeNet and AlexNet
- Say what changed in 2012 (data, GPUs, ReLU, dropout, augmentation) and quote the ImageNet numbers correctly
- Explain the degradation problem, and why it is not overfitting
- Derive why an identity shortcut keeps gradients from vanishing, and say what it does not guarantee
- Place the Vision Transformer in the story as a trade of inductive bias for data
In this lesson
- Chapter 1: simple cells and complex cells
- Chapter 2: the Neocognitron, and then backpropagation
- Chapter 3: data, GPUs, and 2012
- Chapter 4: deeper
- The wall: when deeper is worse
- Residual learning: y=F(x)+x
- Why the shortcut helps gradients
- The code
- Where the lineage went next
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.