A hyperparameter is a setting chosen before a machine learning model begins training. It controls how the learning process happens, like turning knobs on a training control panel. Hyperparameters matter because they can strongly affect whether a model learns useful patterns or performs poorly.
Students can think of them as instructions for how the model should practice, not as facts the model discovers by itself.
During training, the model adjusts its internal parameters, such as weights in a neural network or split choices in a decision tree, based on data. Hyperparameters guide that process by setting limits, speeds, and structure, such as learning rate, batch size, number of trees, max depth, or number of neighbors. To choose good hyperparameters, data scientists often compare many settings using validation data.
The goal is to find settings that help the model make accurate predictions on new examples, not just memorize the training data.
Understanding AI & Machine Learning: What Is a Hyperparameter
A useful way to understand tuning is to think about the model's mistakes over time. At first, a model may be too simple to capture an important pattern. This is called underfitting.
A model can have the opposite problem. It may fit tiny details, noise, or accidental patterns in its practice examples. This is called overfitting.
Settings that control model complexity often create a tradeoff between these errors. A deeper tree, for example, can represent more detailed rules. If it grows too freely, its rules may match one particular dataset rather than the wider real-world pattern.
A separate validation set is important because training results can be misleading. The model has already used the training examples, so a very high training score does not prove that it will work elsewhere. Students might see the same idea when studying for a test.
Memorising answers from one worksheet is different from understanding a topic well enough to solve unfamiliar problems. Data scientists try a setting, train the model, then measure its validation performance.
They keep the final test set untouched until the end. Using the test set repeatedly to make choices would slowly turn it into another practice set.
Settings do not usually work alone. A choice that works well for one dataset may work badly for another. Larger datasets can often support more complex models.
Noisy data may need stronger limits to prevent the model from chasing random variation. Some settings change the time and computer memory required for training. This matters in real applications such as spam filters, photo sorting, route prediction, and medical image analysis.
A model may need to be accurate, but it may need to make predictions quickly enough for the job. Developers therefore balance quality, speed, memory use, and the cost of collecting more data.
Testing many possible settings can be done in organized ways. Grid search tries a planned list of combinations. Random search samples combinations from chosen ranges and can find good options with fewer trials when only a few settings matter strongly.
More advanced methods use earlier results to decide which trial to run next. None of these methods guarantees a perfect answer. Results can change because of random starting values, shuffled data, or a small validation set.
Good practice includes recording each setting, using the same data split for fair comparisons, and checking results more than once. When learning this topic, pay attention to the reason for each choice rather than trying to memorize a single best value.
Key Facts
- A hyperparameter is set before training begins, while a parameter is learned during training.
- Learning rate controls step size during training: new weight = old weight - learning rate × gradient.
- Batch size is the number of training examples used before one model update.
- In a random forest, more trees can improve stability but usually increase training time.
- In a decision tree, max depth limits how many splits the tree can make from root to leaf.
- In k-nearest neighbors, k is the number of nearby examples used to make a prediction.
Vocabulary
- Hyperparameter
- A hyperparameter is a machine learning setting chosen before training that controls how the model learns.
- Parameter
- A parameter is a value the model learns from data during training, such as a weight in a neural network.
- Training Data
- Training data is the set of examples used to teach a machine learning model patterns.
- Validation Data
- Validation data is a separate set of examples used to compare model settings during development.
- Overfitting
- Overfitting happens when a model learns the training data too closely and performs poorly on new data.
Common Mistakes to Avoid
- Calling learned weights hyperparameters is wrong because weights are parameters adjusted by the model during training.
- Using the test set to pick hyperparameters is wrong because it leaks information and makes the final accuracy look too good.
- Choosing the largest model every time is wrong because a very complex model may overfit and fail on new examples.
- Changing many hyperparameters at once without recording results is wrong because it becomes hard to know which setting improved or hurt performance.
Practice Questions
- 1 A student trains a model with batch size 20 on 1,000 training examples. How many batches are needed for one full pass through the data?
- 2 A random forest model is tested with 25 trees, 50 trees, and 100 trees. The validation accuracies are 82%, 86%, and 85%. Which number of trees should be chosen if accuracy is the only goal?
- 3 A model gets 99% accuracy on training data but only 70% accuracy on validation data. Explain what this suggests and name one hyperparameter change that might help.