Model-based RL and planning
An agent that can predict what its actions will do can think before it acts. Dyna learns from imagined experience; Monte Carlo tree search looks ahead at decision time; AlphaGo, AlphaZero and MuZero stripped away human knowledge one piece at a time until even the rules were learned. The catch is always the same — the agent will exploit its model's mistakes.
50 min
After this lesson you can
- Distinguish direct and indirect RL, and implement Dyna-Q
- Run the four steps of Monte Carlo tree search and explain the UCB rule inside it
- Say what AlphaGo Zero, AlphaZero and MuZero each removed from their predecessor
- Explain model bias and why agents exploit learned models
In this lesson
- Dyna: learning from imagined experience
- Decision-time planning: Monte Carlo tree search
- From AlphaGo to MuZero: removing the human, then the rules
- Learning in imagination: world models and Dreamer
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.