A random forest is a machine learning model that uses many decision trees to make a better prediction than one tree alone. Each tree asks a series of simple yes or no questions about the data, such as whether a value is above or below a cutoff. By combining many trees, the model can reduce the chance that one unusual pattern leads to a wrong answer.
Random forests are used in tasks such as classifying emails, predicting house prices, detecting fraud, and helping computers recognize patterns.
Understanding AI & Machine Learning: Random Forests Explained
The strength of a forest comes from making its individual trees disagree in useful ways. During training, a tree may receive a training set in which some rows appear more than once while others are left out. This means no two trees see exactly the same evidence.
A tree can become strongly influenced by a few unusual examples. When many differently trained trees are combined, that influence is less likely to control the final result.
This works best when tree errors are not all linked. If every tree makes the same mistake for the same reason, collecting more trees will not fix it.
Inside each tree, the program searches for a split that makes the groups below it more uniform. For a task with labels, it prefers a split that separates the labels clearly. For a task with numbers, it prefers a split that makes the remaining number values less spread out.
The exact score may be based on impurity or variation, but the main idea is simple. A useful question divides confusing data into simpler groups. Trees are often allowed to grow quite deep.
A deep single tree can memorize details of its training examples, including noise. The forest reduces this overfitting mainly by reducing variation in the result, rather than by making each individual tree simple.
The examples left out of a tree's training sample are valuable. They can be used to test that tree without needing a separate test prediction from the same examples. This is called out of bag evaluation.
It gives a quick estimate of how well the model may perform on new data. It is still important to keep a final test set that was never used during model choices. Students should watch for data leakage.
Leakage happens when information from the answer, or from the future, enters an input feature. A model may then seem highly accurate in testing but fail in real use.
Random forests can report feature importance, which suggests which inputs helped the predictions most. This is useful, but it does not prove that a feature causes an outcome. For example, a postcode may help predict house price because it is connected to location, income, school access, and many other factors.
Related features can share importance in confusing ways. Real datasets need careful preparation before training. Missing values, text categories, unbalanced classes, and unfair historical patterns can all affect results.
In fraud detection or medical screening, a rare positive case may matter far more than an ordinary case. Accuracy alone can hide poor performance on the cases that need the most attention.
Key Facts
- A random forest is an ensemble model made from many decision trees.
- Each tree is trained on a random sample of the training data, often using bootstrap sampling.
- At each split, a tree considers only a random subset of features, which helps trees become different from each other.
- For classification, the forest often predicts by majority vote: prediction = most common class among trees.
- For regression, the forest often predicts by averaging: prediction = (tree1 + tree2 + ... + treeN) / N.
- More trees can improve stability, but they also increase computing time and do not always improve accuracy.
Vocabulary
- Decision Tree
- A decision tree is a model that makes predictions by following a path of yes or no questions from a root to a final answer.
- Random Forest
- A random forest is a group of decision trees that combine their predictions to make a more reliable result.
- Feature
- A feature is an input variable used by a model, such as age, height, temperature, or number of clicks.
- Bootstrap Sample
- A bootstrap sample is a random sample made by choosing from the training data with replacement.
- Majority Vote
- Majority vote is a method where the class predicted by the most trees becomes the final classification.
Common Mistakes to Avoid
- Thinking one tree is the same as a random forest. A random forest uses many trees, so it can average out errors that a single tree might make.
- Using all features at every split. Random forests usually test only a random subset of features at each split, which helps the trees make different mistakes.
- Assuming more trees always means perfect accuracy. More trees can make predictions more stable, but bad data, weak features, or the wrong problem setup can still limit performance.
- Testing the model on the same data used for training. This can make the model look better than it really is because it may have memorized patterns instead of learning to generalize.
Practice Questions
- 1 A random forest has 9 trees for a classification problem. If 6 trees predict Cat and 3 trees predict Dog, what is the final prediction using majority vote?
- 2 A random forest regression model has 5 trees with predictions 12, 15, 14, 10, and 13. What is the final prediction using the average?
- 3 A single decision tree fits the training data almost perfectly but performs poorly on new data. Explain why a random forest might perform better on the same problem.