Attention: learning where to look
An encoder–decoder that squeezes a whole sentence into one vector forgets long inputs. Letting the decoder look back at every source word, with weights it learns, fixed that — and the same trick became the core of the transformer.
45 min
After this lesson you can
- Explain why a single fixed-length context vector is a bottleneck for long inputs
- Write Bahdanau attention — energies, softmax weights, context vector — and say every symbol aloud
- Compare additive, dot-product and general scoring functions
- Describe attention as a soft dictionary lookup with queries, keys and values, and say what the softmax guarantees
In this lesson
- The bottleneck
- The fix: keep every word, and look back softly
- The equations
- A family of scores
- Queries, keys and values: attention as a soft lookup
- The code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.