Exploration
An agent can only learn from rewards it finds. Random dithering finds them with probability that shrinks exponentially with distance; optimism, counts, curiosity and posterior sampling go looking on purpose. Montezuma's Revenge shows how hard "on purpose" is.
50 min
After this lesson you can
- Show, by computing it, why ε-greedy needs exponentially many episodes on a chain
- Explain optimism in the face of uncertainty and implement a count-based bonus
- Derive a pseudo-count from a density model and say why it generalises
- Describe curiosity (ICM) and RND bonuses, and the noisy-TV failure they must avoid
- Say what Go-Explore changed for Montezuma's Revenge, and what it relied on
In this lesson
- Dithering is not exploring
- Optimism in the face of uncertainty
- Counting where you cannot count: pseudo-counts
- Curiosity: reward yourself for being surprised
- Montezuma’s Revenge and Go-Explore
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.