Machine learning is a way for computers to find patterns in data and use those patterns to make predictions or decisions. To build a useful model, scientists do not use one big pile of data all at once. They usually split the data into training, validation, and test sets so they can learn, tune, and check performance fairly.
This matters because a model that only works on examples it has already seen may fail in the real world.
Understanding AI & Machine Learning: Training, Validation, and Test Data
A model learns by changing many internal numbers, often called weights. During training, it makes a prediction for an example, compares that prediction with the known answer, then adjusts its weights to reduce the error. This cycle happens many times.
For a photo classifier, the known answer might be cat or dog. For a weather model, it might be tomorrow's temperature.
The examples need reliable labels. If labels are wrong, incomplete, or based on unfair human judgments, the model can learn those mistakes as if they were facts.
Validation data guides decisions made while building the model. A team may try different numbers of training rounds, different input features, or different model sizes. Each choice is checked against validation results.
If training performance keeps improving while validation performance gets worse, the model is likely becoming too specific to its practice examples. Builders can respond by stopping earlier, simplifying the model, collecting more varied data, or using methods that discourage it from relying too heavily on tiny details.
The goal is not the highest training score. The goal is dependable performance on cases outside the training set.
The final test set must be protected from repeated peeking. If people keep checking test results and changing the model after each check, they slowly tune their choices to that test set. Then the final score is no longer an honest estimate of new-world performance.
This is called test leakage. Leakage can happen in less obvious ways too.
A student appearing in both training and test records, duplicate photos in both groups, or future information included in past predictions can make results look much better than they really are. Data should be split so that the test examples match the kind of new cases the model will face.
Students can see these ideas in spam filters, video recommendations, face recognition, medical screening, and apps that predict travel times. The data used for each job matters more than a single percentage score. A medical screening tool may need to catch nearly every serious case, even if it creates some false alarms.
A fraud detector may need to avoid wrongly blocking ordinary purchases. Accuracy means correct predictions divided by all predictions, but it can hide problems when one outcome is much more common than another. When learning this topic, pay attention to what counts as a new example, who is represented in the data, which errors cause harm, and whether the evaluation setup matches real use.
Key Facts
- Training data is used to fit the model's parameters by learning patterns from examples.
- Validation data is used to compare choices such as model type, features, or settings before the final test.
- Test data is used only at the end to estimate how well the model works on new data.
- Accuracy = number of correct predictions / total number of predictions.
- A common split is 70% training, 15% validation, and 15% test, but the best split depends on the project.
- Overfitting happens when a model learns noise or memorizes training examples instead of learning a pattern that generalizes.
Vocabulary
- Training data
- Training data is the set of examples a machine learning model uses to learn patterns and adjust its internal settings.
- Validation data
- Validation data is a separate set of examples used to tune model choices and compare versions during development.
- Test data
- Test data is a final held-out set of examples used to estimate how well the finished model performs on new cases.
- Model
- A model is a mathematical or computational system that takes inputs and produces predictions or classifications.
- Overfitting
- Overfitting is when a model performs very well on training data but poorly on new data because it learned details that do not generalize.
Common Mistakes to Avoid
- Using test data during training is wrong because the final score will no longer be an honest measure of performance on new data.
- Choosing the model with the best training accuracy is wrong because high training accuracy can mean the model memorized examples instead of learning useful patterns.
- Mixing duplicate or nearly identical examples across splits is wrong because the model may effectively see the answer before it is tested.
- Assuming one data split is always perfect is wrong because small or unbalanced datasets may need careful sampling or cross-validation to get a reliable estimate.
Practice Questions
- 1 A dataset has 1,000 labeled images. If it is split into 70% training, 15% validation, and 15% test, how many images go in each set?
- 2 A model makes 180 correct predictions out of 240 test examples. What is its test accuracy?
- 3 A model gets 99% accuracy on training data but only 62% accuracy on test data. Explain what problem this suggests and name one way to reduce it.