Cross-validation is a way to test how well a machine learning model will work on new data. Instead of training and testing once, we split the data into several parts called folds and rotate which part is used for testing. This matters because a model can look accurate on one lucky split but fail on different examples.
Cross-validation gives a more reliable estimate of real performance.
In k-fold cross-validation, the dataset is divided into k equal or nearly equal folds. The model trains on k - 1 folds and tests on the remaining fold, then this process repeats until every fold has been the test fold once. The final score is usually the average of all test scores, such as average accuracy or average error.
This helps students and scientists compare models, tune settings, and reduce the chance of being fooled by one random train-test split.
Understanding AI & Machine Learning: What Is Cross-Validation
Cross-validation is most useful while choosing between possible models and settings. A student might compare a decision tree with a neural network, or try several values for a model setting that controls complexity. The same validation process gives each choice a fair test.
However, the scores can become misleading if you keep trying hundreds of choices until one happens to score well. In that case, the model selection process has partly learned from the validation data. Good projects keep one final test set untouched until every important choice is finished.
A major danger is data leakage. Leakage happens when information from a test portion reaches the training process by accident. For example, suppose you calculate the average height of all students before splitting the data, then use that average to scale height values.
The test portion has influenced the training transformation. The correct method is to calculate averages, fill missing values, select useful features, and tune settings using only the training portion in each round.
Then the learned transformation is applied to that round's test portion. This rule matters for every step that learns something from data.
The way data is divided should match the problem. In a dataset where only a small number of cases are positive, ordinary random folds may put too few positive cases into one portion. A stratified split keeps roughly the same class balance in each portion.
This is common when detecting spam, disease, fraud, or rare equipment failures. Some examples belong together and must stay together.
Records from one patient, one school, or one household should not be scattered across different portions. Otherwise, the model may recognize the group instead of learning a pattern that works for new groups.
Time creates another important exception. Randomly mixing old and new records is usually wrong for forecasting. A weather model predicting tomorrow should train on earlier days, then be checked on later days.
A sales forecast should not see future prices or future buying patterns during training. Time-based validation moves forward through time, using the past to predict what comes next.
This better copies the real job of the model. Students should always ask whether their data has an order, a shared source, or repeated measurements from the same people.
The final average score is useful, but it is not the whole story. Look at how much the scores vary across portions. A model with a high average but wildly different results may be unreliable.
The right measurement depends on the task. Accuracy can hide poor performance when one class is much more common than another. Precision and recall can show different kinds of mistakes in spam filters or medical screening.
Mean absolute error is often clearer for predicting values such as house prices or temperatures. Cross-validation takes extra computing time, so simple models may be tried first. Its real value is careful comparison, honest checking, and attention to how data was collected.
Key Facts
- Cross-validation estimates how well a model performs on data it has not seen before.
- In k-fold cross-validation, the data is split into k folds.
- Each round trains on k - 1 folds and tests on 1 fold.
- Total rounds in k-fold cross-validation = k.
- Average score = (score1 + score2 + ... + scorek) / k.
- More folds often use data more efficiently, but they can take more time to compute.
Vocabulary
- Cross-validation
- A model testing method that repeatedly splits data into training and testing parts to estimate performance.
- Fold
- One section of a dataset used as either training data or testing data during cross-validation.
- Training set
- The part of the data used to teach the model patterns and relationships.
- Test set
- The part of the data used to check how well the trained model performs on examples it did not use for learning.
- Overfitting
- A problem where a model learns the training data too closely and performs poorly on new data.
Common Mistakes to Avoid
- Testing on the same data used for training, because this can make the model seem better than it really is.
- Shuffling data incorrectly before splitting, because sorted or grouped data can make folds unbalanced and give misleading scores.
- Choosing the model based only on the best single fold score, because the average score gives a more stable estimate of performance.
- Letting test fold information affect training, because any data leakage can make cross-validation results unfairly high.
Practice Questions
- 1 A dataset has 100 examples and is split into 5 folds. How many examples are in each fold, and how many examples are used for training in each round?
- 2 A 4-fold cross-validation gives accuracy scores of 0.80, 0.85, 0.75, and 0.90. What is the average accuracy?
- 3 A student says cross-validation is unnecessary because one train-test split is faster. Explain why cross-validation can still be worth using when comparing machine learning models.