Scaled dot-product and multi-head attention
The transformer’s attention fits on one line, softmax(QKᵀ/√dₖ)V. Here is every symbol and every shape, the reason for the square root, how masks decide who may see whom, why there are many heads — and what it all costs.
50 min
After this lesson you can
- Compute softmax(QKᵀ/√dₖ)V by hand on a small example and state the shape of every intermediate matrix
- Derive why the variance of a dot product grows with dₖ and explain how that saturates the softmax
- Apply causal and padding masks, and distinguish self-attention from cross-attention
- Implement multi-head attention from scratch and count its parameters
In this lesson
- Why get rid of the recurrence?
- Three roles for every token
- The formula, decoded
- Why divide by the square root of the key size?
- Masks: who may see whom
- Self-attention and cross-attention
- Many heads
- The code
- What it costs
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.