RLVR and reasoning models
When an answer can be checked by a program, the reward needs no model and no human. Reinforcement learning against such checkers — with GRPO as the workhorse — produced models that think for thousands of tokens before answering. What it actually teaches them is still being argued.
55 min
After this lesson you can
- Define reinforcement learning with verifiable rewards and give examples of verifiers
- Decode the GRPO objective and compute group-relative advantages by hand
- Summarise what the DeepSeek-R1 paper and OpenAI's o1 announcement actually reported, and attribute each claim
- Explain pass@k, and the evidence on both sides of the “sharpening versus expanding” debate
- Describe how reward hacking shows up even when the reward is a program
In this lesson
- Rewards from checkers
- GRPO: PPO without a critic
- What the reasoning-model papers reported
- The open question: does RLVR teach new reasoning, or sharpen what is there?
- Reward hacking with a perfect judge
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.