Validation design and leakage
A test score is only honest if nothing about the test set reached the model. The jobs of training, validation and test data; holdout, k-fold and nested cross-validation; splits that respect groups and time; and the ways information leaks — through features, preprocessing, duplicates, repeated peeking and, for language models, the training corpus itself.
45 min
After this lesson you can
- State what each of the training, validation and test sets is for, and why a test set wears out with use
- Explain why the best cross-validation score in a hyperparameter search is optimistic, and fix it with nested cross-validation
- Choose grouped or time-ordered splits when examples are not independent, and see what random splits leak
- Recognise target leakage, preprocessing leakage, duplicate leakage and adaptive overfitting, with real examples of each
- Explain benchmark contamination in large language models
In this lesson
- Three sets, three jobs
- Cross-validation, and the winner’s curse
- When examples aren’t independent
- The many ways information leaks
- Contamination: when the training set is the internet
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.