Cross-validation is a way to test how well a machine learning model will work on new data. Instead of training and testing once, we split the data into several parts called folds and rotate which part is used for testing. This matters because a model can look accurate on one lucky split but fail on different examples.
Cross-validation gives a more reliable estimate of real performance.
In k-fold cross-validation, the dataset is divided into k equal or nearly equal folds. The model trains on k - 1 folds and tests on the remaining fold, then this process repeats until every fold has been the test fold once. The final score is usually the average of all test scores, such as average accuracy or average error.
This helps students and scientists compare models, tune settings, and reduce the chance of being fooled by one random train-test split.
Key Facts
- Cross-validation estimates how well a model performs on data it has not seen before.
- In k-fold cross-validation, the data is split into k folds.
- Each round trains on k - 1 folds and tests on 1 fold.
- Total rounds in k-fold cross-validation = k.
- Average score = (score1 + score2 + ... + scorek) / k.
- More folds often use data more efficiently, but they can take more time to compute.
Vocabulary
- Cross-validation
- A model testing method that repeatedly splits data into training and testing parts to estimate performance.
- Fold
- One section of a dataset used as either training data or testing data during cross-validation.
- Training set
- The part of the data used to teach the model patterns and relationships.
- Test set
- The part of the data used to check how well the trained model performs on examples it did not use for learning.
- Overfitting
- A problem where a model learns the training data too closely and performs poorly on new data.
Common Mistakes to Avoid
- Testing on the same data used for training, because this can make the model seem better than it really is.
- Shuffling data incorrectly before splitting, because sorted or grouped data can make folds unbalanced and give misleading scores.
- Choosing the model based only on the best single fold score, because the average score gives a more stable estimate of performance.
- Letting test fold information affect training, because any data leakage can make cross-validation results unfairly high.
Practice Questions
- 1 A dataset has 100 examples and is split into 5 folds. How many examples are in each fold, and how many examples are used for training in each round?
- 2 A 4-fold cross-validation gives accuracy scores of 0.80, 0.85, 0.75, and 0.90. What is the average accuracy?
- 3 A student says cross-validation is unnecessary because one train-test split is faster. Explain why cross-validation can still be worth using when comparing machine learning models.