Descent
XI.3Seeing and RememberingPro

Recurrent networks

Sentences, speech and prices arrive one step at a time and can be any length. A recurrent network reuses one cell at every step and carries a memory forward. Training it means backpropagating through time — and multiplying the same Jacobian again and again, which makes gradients vanish or explode.

55 min

After this lesson you can

  • Explain why fixed-size windows are a poor fit for sequences, using Elman's shifted-pattern example
  • Write the recurrence with every shape, and count its parameters
  • Unroll an RNN and derive backpropagation through time
  • Show that the gradient across many steps is a product of Jacobians, and relate its growth or decay to the spectral radius of
  • Implement a character-level RNN with BPTT and gradient clipping in NumPy

In this lesson

  1. Why a window isn’t enough
  2. The recurrence
  3. The picture: unrolling through time
  4. Backpropagation through time
  5. The problem: a product of Jacobians
  6. A patch for exploding gradients: clipping
  7. The code

This lesson is part of Descent Pro

Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.

See plansSign in / Sign up

Descent Pro

Start your free trial to keep reading

This lesson, and every other lesson on the route, is part of Descent Pro. Every plan starts with a free trial, and students with a .edu email get longer. Cancel before the trial ends and you pay nothing.

Start free trialSign in / Sign up