How sure are you? Bootstrap and tests
A score measured on a finite test set, after one training run, is a random number. Standard errors and confidence intervals, the bootstrap, paired comparisons of two models (McNemar's test and the paired bootstrap), variance across random seeds, the multiple-comparisons trap, and why a significant difference can still be a meaningless one.
50 min
After this lesson you can
- Compute the standard error of an accuracy and a 95% confidence interval, and interpret the interval correctly
- Build a bootstrap confidence interval for any metric, and say what randomness it does and doesn't capture
- Compare two models on the same test set with McNemar's test and the paired bootstrap, and explain why pairing helps
- Account for variation across random seeds and avoid best-of-k reporting
- Recognise the multiple-comparisons problem, apply Holm's correction, and separate statistical significance from effect size
In this lesson
- How big is the noise? Standard errors
- The bootstrap: error bars for anything
- Comparing two models: pair them
- The other randomness: seeds
- Many comparisons
- Significant is not the same as important
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.