The KV cache
Why generating text naively recomputes the same keys and values over and over, how caching them makes each new token cheap, and the memory bill that comes with it, along with the four ideas (MQA, GQA, PagedAttention, latent attention) that shrink it.
50 min
After this lesson you can
- Explain exactly which quantities can be reused between decoding steps, and why queries and attention weights are not cached
- Implement a decode loop with and without a KV cache and show they produce identical outputs
- Compute the KV-cache size of a real model from its configuration
- Explain how multi-query, grouped-query and multi-head latent attention shrink the cache, and how PagedAttention stops it being wasted
In this lesson
- What stays the same from one step to the next
- Counting what the cache saves
- What exactly is stored, and how big it gets
- Four ways to pay less
- Code: a decode loop with and without a cache
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.