Imbalance and distribution shift
Two ways the data a model meets can differ from the balanced, familiar world it was scored on. Rare classes (and what class weights, resampling and threshold moving really do to a model's probabilities), then distribution shift — covariate, label and concept shift — how to detect it, and when reweighting can fix it.
50 min
After this lesson you can
- Show that class weights and resampling are equivalent to changing the class prior, and predict what that does to probabilities
- Choose between threshold moving, reweighting and resampling for a rare class, and avoid the evaluation mistakes they invite
- Distinguish covariate, label and concept shift by which factor of the joint distribution changes
- Derive importance weighting for covariate shift and the prior correction for label shift, and estimate both without target labels
- Detect shift with a two-sample test, and explain why overlap between the distributions limits every fix
In this lesson
- Part 1: rare classes
- Part 2: when the world moves
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.