The cost of attention
Count where the arithmetic and the bytes go when a transformer reads a long prompt and writes a long answer, and see why the two halves of inference hit two different walls.
40 min
After this lesson you can
- Count the FLOPs of one transformer layer as a function of sequence length and width , and find where the term takes over
- Explain why generating text is a sequential loop, and name its two phases, prefill and decode
- Compute arithmetic intensity and use a roofline to decide whether a computation is compute-bound or memory-bound
- Say which of the three costs of long context (FLOPs, memory capacity, memory traffic) each later lesson in this part attacks
In this lesson
- Counting the work
- Generation is a loop
- Compute-bound or memory-bound?
- Why long context is expensive
- Code: count, then measure
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.