Cross-validation is a statistical method for estimating how well a model will perform on new data. It is especially useful when a dataset is not large enough to waste a big portion on a single test split. Instead of judging a model from one lucky or unlucky split, cross-validation uses several different splits.
This makes the performance estimate more stable and trustworthy.
Key Facts
- In K-fold cross-validation, the dataset is split into K approximately equal folds.
- Each round uses K - 1 folds for training and 1 fold for validation.
- Every data point is used for validation exactly once across the K rounds.
- Mean validation score = (score1 + score2 + ... + scoreK) / K.
- A common choice is K = 5 or K = 10 because it balances reliability and computing cost.
- Cross-validation estimates generalization performance, but a final untouched test set is still best for the final report.
Vocabulary
- Cross-validation
- A resampling method that evaluates a model by training and validating it on several different splits of the same dataset.
- Fold
- One of the equal or nearly equal parts into which a dataset is divided during K-fold cross-validation.
- Validation set
- The data used to evaluate a model during model selection or tuning, not to fit the model parameters.
- Generalization
- A model's ability to make accurate predictions on new data that was not used during training.
- Performance metric
- A numerical measure, such as accuracy, mean squared error, or F1 score, used to judge how well a model performs.
Common Mistakes to Avoid
- Training on the validation fold, which is wrong because validation data must stay unseen during that round to give a fair performance estimate.
- Averaging results from unequal procedures, which is wrong because each fold should follow the same preprocessing, training, and evaluation steps.
- Preprocessing before splitting the data, which is wrong when steps like scaling or feature selection use information from the full dataset and cause data leakage.
- Using cross-validation as the final test result after model tuning, which is wrong because repeated tuning can make the validation estimate overly optimistic.
Practice Questions
- 1 A dataset has 200 examples and is split into K = 5 folds. How many examples are in each validation fold, and how many are used for training in each round?
- 2 A 4-fold cross-validation gives validation accuracies of 0.78, 0.82, 0.80, and 0.76. What is the mean validation accuracy?
- 3 Explain why rotating the validation fold in K-fold cross-validation usually gives a more reliable estimate than using one fixed train-test split.