Calibration and proper scoring rules
When a model says 80%, is it right 80% of the time? Reliability diagrams and expected calibration error (and why the binning matters), the scoring rules that reward honest probabilities and the proof that they do, and three ways to fix a miscalibrated model after training — Platt scaling, isotonic regression and temperature scaling.
45 min
After this lesson you can
- Define calibration for binary and multiclass classifiers, and explain why it is separate from accuracy and AUC
- Read a reliability diagram and compute expected calibration error, knowing how bins and sample size distort it
- Prove that the Brier score and log loss are proper scoring rules, and show that absolute error is not
- Recalibrate a model with Platt scaling, isotonic regression or temperature scaling, fitted on held-out data
- Summarise what Guo et al. (2017) found about modern networks, and what later work revised
In this lesson
- What calibrated means
- Seeing it: the reliability diagram
- Expected calibration error, and its caveats
- Scores that reward honesty
- Why models are miscalibrated
- Fixing it after training
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.