Resampling Methods
Adapted from Kameron Decker Harris’ 2022 weekly notes.
1 Resampling methods
Resampling methods draw repeated samples from the data we have, and fit the same model again to each one. Comparing the fits shows what a single fit cannot, such as how well the model predicts new data.
2 Holding out a validation set
The simplest approach splits the data once, into the three parts in Figure 1. We fit each candidate model on the training set, choose the candidate with the smallest validation error, and report the chosen model’s test error.
With only one candidate, nothing is chosen. A validation set is then not needed, and the two-part split in Figure 2 suffices.
2.1 Weighing a single split
A single split assumes that the observations are independent and identically distributed. Even then, it has three drawbacks:
The held-out parts shrink the training set.
The validation error depends on which observations the random split puts in the validation set.
As an estimate of the chosen model’s risk, the validation error can therefore have a high variance.
Hastie and Tibshirani describe the validation-set approach, from 8:11, and its drawbacks, from 13:14 [1], video 5.1.
Abu-Mostafa covers model selection with a validation set in his lecture on validation, from 30:47 [2], Lecture 13.
The CS229 lecture covers data splits, models and cross-validation [3], Lecture 8.
Lall compares validation approaches in his EE104 lecture on validation, from 14:20 [4], Lecture 4.
3 K-fold cross-validation
\(K\)-fold cross-validation splits the data into \(K\) folds. This \(K\) is not the number of neighbors in \(K\)-nearest neighbors. Both names are standard, so the letter is reused. Figure 3 shows \(K = 4\).
The test fold rotates through the \(K\) folds. Each turn fits the model on the other \(K - 1\) folds and scores it on the test fold. The \(K\)-fold cross-validation estimate of the risk combines the \(K\) scores. When the folds have the same size, the cross-validation estimate is the sample mean of the \(K\) scores. Every observation is scored once, so the cross-validation estimate varies less than the validation error of a single split. The price is \(K\) fits instead of one.
Figure 4 runs \(K\)-fold cross-validation on the Auto data, one fold at a time. The mean squared errors of the folds vary from fold to fold, and their sample mean is the \(K\)-fold cross-validation estimate.
3.1 Leave-one-out cross-validation
Leave-one-out cross-validation is the extreme case \(K = n\), in which each fold holds a single observation.
Figure 5 compares three estimates of the test mean squared error on the Auto data. For the degree-2 fit, the ten estimates from a random validation set run from 17.5 to 21, the ten 10-fold estimates run from 19.2 to 19.5, and leave-one-out cross-validation gives 19.2. Averaging over folds makes the 10-fold estimate far less variable than a single split.
Abu-Mostafa explains leave-one-out cross-validation, from 49:03 [2], Lecture 13.
Starmer explains \(K\)-fold cross-validation [6].
Abu-Mostafa explains \(K\)-fold cross-validation, from 1:06:25 [2], Lecture 13.
Lall covers cross-validation in his EE104 lecture on validation, from 36:24 [4], Lecture 4.
GeeksforGeeks explains cross-validation [7], including \(K\)-fold and leave-one-out.
The University of Washington’s CSE 446 slides cover train-test splits and cross-validation [8], Lecture 5.
