Seq2seq and the bottleneck
In 2014 two LSTMs, one reading and one writing, learned to translate whole sentences — squeezing each source sentence into a single fixed-length vector. It worked better than anyone expected, and the squeeze is exactly what attention was invented to remove.
45 min
After this lesson you can
- Describe the encoder–decoder architecture and write
- Explain teacher forcing, beam search and the source-reversal trick, and why reversal helped
- State the fixed-length bottleneck precisely and summarise the evidence for and against it (Sutskever et al.; Cho et al.; Bahdanau et al.)
- Measure, in code, how much of a sequence a fixed-size state can hold
- Explain how keeping every encoder state removes the bottleneck, as the bridge to attention
In this lesson
- The encoder–decoder
- What made it work
- The bottleneck
- The picture: how much can one vector hold?
- The way out: keep every state
- The code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.