Bagging and random forests
One deep tree is unstable; the average of hundreds of them is not. Derive exactly how much averaging helps — and why correlation between the trees sets a floor — then decorrelate them with random feature subsets, estimate error for free with out-of-bag samples, and learn when feature importances lie.
40 min
After this lesson you can
- Explain bootstrap aggregation and why it reduces variance but not bias
- Derive the variance of an average of B correlated estimators, , and explain what it implies
- Describe how random forests decorrelate trees, and the role of the number of candidate features
- Compute and interpret out-of-bag error
- Compare impurity-based and permutation feature importances, and recognise their failure modes
In this lesson
- Averaging reduces variance: a calculation
- Bagging: bootstrap aggregation
- Measuring the floor
- Random forests: decorrelate the trees
- Out-of-bag error: cross-validation for free
- Feature importance, and how it misleads
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.