Uncertainty in deep learning
A network's softmax says how sure it sounds, not how sure it should be. The two kinds of uncertainty, the practical ways to estimate the one that matters off the training data (deep ensembles, MC dropout, Bayesian networks), out-of-distribution detection, and conformal prediction, which gives a coverage guarantee that holds whatever the model.
50 min
After this lesson you can
- Distinguish aleatoric from epistemic uncertainty and say which one more data removes
- Explain why a calibrated softmax is not a measure of uncertainty about the model
- Build a deep ensemble and decompose its predictive variance into the two kinds
- Describe MC dropout and the main criticism of its Bayesian interpretation
- Derive the split-conformal coverage guarantee and state exactly what it does and doesn't promise
In this lesson
- Two kinds of uncertainty
- Deep ensembles
- MC dropout
- Detecting out-of-distribution inputs
- Conformal prediction: a guarantee that doesn’t trust the model
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.