Sparse attention
Instead of letting every token look at every other token, choose a small set of pairs, either by a fixed pattern or by a learned selection, and let depth carry information the rest of the way.
45 min
After this lesson you can
- Describe attention as a mask over the grid and relate the mask's number of nonzeros to the cost
- Build sliding-window, dilated, strided, fixed, global and random masks and count how their cost grows with
- Explain how stacking layers widens the receptive field, and use it to explain Mistral 7B's 131K theoretical span
- Place learned sparsity (NSA, DSA), linear attention and FlashAttention correctly relative to fixed sparse patterns
In this lesson
- Attention with a mask
- A gallery of fixed patterns
- Depth relays information
- What the theory says, and what it does not
- Let the model choose: learned sparsity
- Two things that are not sparse attention
- Code: build the masks and count
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.