Generalization, bias–variance and regularization
A model is judged on data it has never seen. See overfitting happen as a polynomial gets more flexible, derive the bias–variance decomposition, and tame flexibility with ridge (closed form, derived) and lasso (exact zeros) — choosing their strength by cross-validation.
45 min
After this lesson you can
- Distinguish training error from test error, and use train / validation / test splits correctly
- Derive the bias–variance decomposition of expected squared error
- Derive the ridge solution , show it always exists for , and explain shrinkage via the SVD
- Explain geometrically why the lasso produces exact zeros and has no closed form in general
- Choose a regularisation strength with -fold cross-validation
In this lesson
- Training error lies
- The bias–variance decomposition
- Ridge regression: penalise large weights
- The lasso: penalise absolute values, get exact zeros
- Choosing λ (or the degree): validation and cross-validation
- A twist from modern deep learning
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.