Imitation and offline RL
Learning from someone else's data instead of your own trial and error. Behavioural cloning is supervised learning — until the learner drifts somewhere the expert never went, and its errors compound. DAgger fixes that by asking the expert about the learner's own mistakes; offline RL has no expert to ask, and must learn to distrust its own optimism.
50 min
After this lesson you can
- Explain why behavioural cloning's errors grow quadratically with the horizon, and state the bound
- Describe DAgger and why training on the learner's own states makes errors grow linearly
- Say what inverse RL recovers, and why the answer is never unique
- Explain distribution shift and extrapolation error in offline RL, and how BCQ and CQL respond
- Connect imitation and offline RL to supervised fine-tuning and preference optimisation of LLMs
In this lesson
- Behavioural cloning and compounding errors
- DAgger: ask the expert about your own mistakes
- Inverse RL: learning what the expert wants
- Offline RL: better than the data, from the data
- Where large language models fit
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.