Deep Q-learning and actor–critic methods
Put a neural network where the Q-table was and Q-learning can blow up. Why — the deadly triad — and the fixes that made deep RL work — replay, target networks, double estimators — then the same ideas carried to continuous actions in DDPG, TD3 and SAC.
55 min
After this lesson you can
- State the deadly triad and show, with Baird's counterexample, that it can make values diverge
- Explain what experience replay and a target network each fix, and what they do not
- Derive why a max over noisy estimates is biased upward, and write the Double DQN target
- Decode the DDPG, TD3 and SAC objectives and say what each change is for
In this lesson
- Why the obvious thing breaks: the deadly triad
- What DQN changed
- A max over noisy estimates is biased upward
- Continuous actions: the actor takes over the max
- The family tree
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.