LSTM and GRU
The long short-term memory gives a recurrent network a memory cell whose contents are carried forward by addition, not by repeated matrix multiplication. Gates learn when to write, keep, forget and read — and the gradient rides the same additive path back through time.
55 min
After this lesson you can
- Derive the constant error carousel from the requirement that an error signal neither shrink nor grow
- Write the full LSTM equations (with the forget gate), say every symbol aloud, and count parameters
- Explain why the additive cell update lets gradients survive many steps, and how the forget gate controls that
- Explain what the forget gate (Gers et al., 2000) added to the 1997 LSTM
- Write the GRU and compare it with the LSTM
In this lesson
- The constant error carousel
- The missing gate: learning to forget
- The picture: one cell, one toy sequence
- The equations
- Why the gradient survives
- GRU: a simpler gated cell
- The code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.