Policy gradients and PPO
Instead of learning values and acting greedily, adjust the policy itself in the direction that raises expected reward. The log-derivative trick makes that gradient computable, baselines make it usable, and PPO's clipped objective makes it safe.
50 min
After this lesson you can
- Derive the policy gradient with the log-derivative trick, and explain why the environment's dynamics drop out
- Implement REINFORCE and explain why a baseline reduces variance without adding bias
- Define the advantage and describe actor–critic methods and GAE
- Read PPO's clipped surrogate objective and say, case by case, when a sample gets no gradient
In this lesson
- The objective and the obstacle
- The log-derivative trick
- REINFORCE on a bandit, and the variance problem
- Reward-to-go, advantage and the critic
- Steps that are too big: trust regions and PPO
- Where this goes next
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.