RLHF: learning from preferences
A pretrained model predicts text, not helpful answers. Nobody can write down a reward for “helpful” — but people can say which of two answers is better. RLHF turns those comparisons into a reward model and optimises against it, carefully; DPO skips the reward model altogether.
55 min
After this lesson you can
- Describe the three-stage InstructGPT recipe and the signal each stage uses
- Derive the Bradley–Terry reward-model loss and train one
- Decode the KL-regularised RLHF objective and derive its closed-form optimum
- Explain reward over-optimisation (Goodhart's law) and what Gao et al. measured
- Derive the DPO loss from the RLHF objective, and say what it keeps and what it drops
In this lesson
- From Atari games to InstructGPT
- The three-stage recipe
- Goodhart’s law, measured
- DPO: the reward model was inside the policy all along
- Feedback from AI: Constitutional AI and RLAIF
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.