A hyperparameter is a setting chosen before a machine learning model begins training. It controls how the learning process happens, like turning knobs on a training control panel. Hyperparameters matter because they can strongly affect whether a model learns useful patterns or performs poorly.
Students can think of them as instructions for how the model should practice, not as facts the model discovers by itself.
During training, the model adjusts its internal parameters, such as weights in a neural network or split choices in a decision tree, based on data. Hyperparameters guide that process by setting limits, speeds, and structure, such as learning rate, batch size, number of trees, max depth, or number of neighbors. To choose good hyperparameters, data scientists often compare many settings using validation data.
The goal is to find settings that help the model make accurate predictions on new examples, not just memorize the training data.
Key Facts
- A hyperparameter is set before training begins, while a parameter is learned during training.
- Learning rate controls step size during training: new weight = old weight - learning rate × gradient.
- Batch size is the number of training examples used before one model update.
- In a random forest, more trees can improve stability but usually increase training time.
- In a decision tree, max depth limits how many splits the tree can make from root to leaf.
- In k-nearest neighbors, k is the number of nearby examples used to make a prediction.
Vocabulary
- Hyperparameter
- A hyperparameter is a machine learning setting chosen before training that controls how the model learns.
- Parameter
- A parameter is a value the model learns from data during training, such as a weight in a neural network.
- Training Data
- Training data is the set of examples used to teach a machine learning model patterns.
- Validation Data
- Validation data is a separate set of examples used to compare model settings during development.
- Overfitting
- Overfitting happens when a model learns the training data too closely and performs poorly on new data.
Common Mistakes to Avoid
- Calling learned weights hyperparameters is wrong because weights are parameters adjusted by the model during training.
- Using the test set to pick hyperparameters is wrong because it leaks information and makes the final accuracy look too good.
- Choosing the largest model every time is wrong because a very complex model may overfit and fail on new examples.
- Changing many hyperparameters at once without recording results is wrong because it becomes hard to know which setting improved or hurt performance.
Practice Questions
- 1 A student trains a model with batch size 20 on 1,000 training examples. How many batches are needed for one full pass through the data?
- 2 A random forest model is tested with 25 trees, 50 trees, and 100 trees. The validation accuracies are 82%, 86%, and 85%. Which number of trees should be chosen if accuracy is the only goal?
- 3 A model gets 99% accuracy on training data but only 70% accuracy on validation data. Explain what this suggests and name one hyperparameter change that might help.