Vision transformers, detection and segmentation
A classifier answers "what is in this picture?". Real vision systems must also say where, how many and exactly which pixels — and in 2020 the transformer showed that an image can be read as a sentence of patches.
55 min
After this lesson you can
- Turn an image into a sequence of patch tokens and write down the Vision Transformer's input equation with every shape
- State accurately what the ViT paper found about data size and inductive bias, and what DeiT added
- Compute IoU, run greedy non-maximum suppression by hand, and explain how COCO-style AP is computed
- Contrast two-stage (R-CNN family) and one-stage (YOLO) detectors, and say what DETR removed
- Distinguish semantic from instance segmentation and explain the U-Net and Mask R-CNN designs
In this lesson
- The data: a grid with local structure
- Part 1: the Vision Transformer
- Part 2: detection — boxes, overlaps and suppression
- Part 3: segmentation — a label for every pixel
- Learning to see without labels
- Open problems
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.