Machine learning is a way for computers to find patterns in data and use those patterns to make predictions or decisions. To build a useful model, scientists do not use one big pile of data all at once. They usually split the data into training, validation, and test sets so they can learn, tune, and check performance fairly.
This matters because a model that only works on examples it has already seen may fail in the real world.
Key Facts
- Training data is used to fit the model's parameters by learning patterns from examples.
- Validation data is used to compare choices such as model type, features, or settings before the final test.
- Test data is used only at the end to estimate how well the model works on new data.
- Accuracy = number of correct predictions / total number of predictions.
- A common split is 70% training, 15% validation, and 15% test, but the best split depends on the project.
- Overfitting happens when a model learns noise or memorizes training examples instead of learning a pattern that generalizes.
Vocabulary
- Training data
- Training data is the set of examples a machine learning model uses to learn patterns and adjust its internal settings.
- Validation data
- Validation data is a separate set of examples used to tune model choices and compare versions during development.
- Test data
- Test data is a final held-out set of examples used to estimate how well the finished model performs on new cases.
- Model
- A model is a mathematical or computational system that takes inputs and produces predictions or classifications.
- Overfitting
- Overfitting is when a model performs very well on training data but poorly on new data because it learned details that do not generalize.
Common Mistakes to Avoid
- Using test data during training is wrong because the final score will no longer be an honest measure of performance on new data.
- Choosing the model with the best training accuracy is wrong because high training accuracy can mean the model memorized examples instead of learning useful patterns.
- Mixing duplicate or nearly identical examples across splits is wrong because the model may effectively see the answer before it is tested.
- Assuming one data split is always perfect is wrong because small or unbalanced datasets may need careful sampling or cross-validation to get a reliable estimate.
Practice Questions
- 1 A dataset has 1,000 labeled images. If it is split into 70% training, 15% validation, and 15% test, how many images go in each set?
- 2 A model makes 180 correct predictions out of 240 test examples. What is its test accuracy?
- 3 A model gets 99% accuracy on training data but only 62% accuracy on test data. Explain what problem this suggests and name one way to reduce it.