A train-test split is a simple but powerful way to check whether a machine learning model has actually learned a pattern or only memorized examples. The full dataset is divided so the model trains on one part and is evaluated on data it has not seen before. This matters because real-world predictions are always made on new cases, not on the same examples used during learning.
A good split gives a more honest estimate of future performance.
In many workflows, the data is separated into training, validation, and testing sets. The training set is used to fit model parameters, the validation set is used to compare models and tune settings, and the test set is saved for the final evaluation. Data leakage happens when information from validation or test examples accidentally influences training, making performance look better than it really is.
Careful splitting, preprocessing inside the training workflow, and using time-aware or group-aware splits when needed help prevent misleading results.
Key Facts
- A common split is 80% training and 20% testing, but the best ratio depends on dataset size and task.
- Training set: used to fit the model parameters.
- Validation set: used to tune hyperparameters and choose among models.
- Test set: used once at the end to estimate performance on unseen data.
- Generalization means performing well on new data, not just on training data.
- If N is the total number of examples and p is the test fraction, test size = pN and training size = (1 - p)N.
Vocabulary
- Training set
- The portion of the data used by the algorithm to learn patterns and fit model parameters.
- Validation set
- A separate portion of the data used to tune model choices before the final test.
- Test set
- The held-out portion of the data used to estimate how well the final model works on unseen examples.
- Generalization
- The ability of a model to make accurate predictions on new data that was not used for training.
- Data leakage
- A mistake where information from validation or test data influences training and gives an overly optimistic performance estimate.
Common Mistakes to Avoid
- Testing on the training set is wrong because the model has already seen those examples, so the score may reflect memorization instead of real prediction ability.
- Tuning hyperparameters using the test set is wrong because it turns the test set into part of the model selection process and makes the final score biased.
- Preprocessing the full dataset before splitting is wrong when steps like scaling, imputation, or feature selection use information from all examples, because test data information can leak into training.
- Randomly splitting time-series or grouped data without care is wrong because future records or related records can end up in training, making the model seem better than it will be in real use.
Practice Questions
- 1 A dataset has 1,000 examples. If you use an 80% training and 20% testing split, how many examples go into each set?
- 2 A dataset has 2,400 examples and is split into 70% training, 15% validation, and 15% testing. How many examples are in each part?
- 3 A student scales every feature using the mean and standard deviation of the full dataset before making the train-test split. Explain why this can cause data leakage and describe the correct procedure.