Gradient boosting
Instead of averaging independent trees, add small trees one at a time, each fitted to the mistakes of the ones before. See why this is gradient descent in the space of functions, build it from scratch with stumps, and meet XGBoost, LightGBM and CatBoost — and the evidence on why boosted trees still rule tabular data.
45 min
After this lesson you can
- State the boosting question (can weak learners make a strong one?) and describe AdaBoost's reweighting at a high level
- Derive least-squares boosting as repeatedly fitting residuals, and generalise it to any differentiable loss via pseudo-residuals
- Explain gradient boosting as gradient descent in function space, and the roles of shrinkage, tree size, subsampling and early stopping
- Derive XGBoost's optimal leaf weight from a second-order expansion, and summarise what LightGBM and CatBoost change
- Summarise the benchmark evidence on boosted trees versus neural networks for tabular data, including its limits
In this lesson
- AdaBoost: reweight the mistakes
- Boosting is forward stagewise additive modelling
- Squared loss: fit the residuals
- Any loss: fit the negative gradient
- Regularising a boosted model
- The modern libraries: XGBoost, LightGBM, CatBoost
- Why boosted trees still rule tables
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.