A decision tree is a statistical and machine learning model that makes predictions by splitting data into smaller and smaller groups. Each split asks a question about one variable, such as whether a study time is above 5 hours or whether a tumor size is below 2 cm. The final endpoints, called leaves, give a predicted class or numerical value.
Decision trees matter because they are easy to visualize, explain, and use for both classification and regression problems.
A tree is built by choosing splits that make the resulting groups more pure or more useful for prediction. For classification, common split criteria include Gini impurity and entropy, which measure how mixed the classes are in a node. For regression, a tree often chooses splits that reduce the sum of squared errors within the groups.
Because a tree can become too detailed and memorize training data, pruning and ensemble methods such as random forests are used to improve performance on new data.
Understanding Statistics: Decision Trees
A tree learns its rules from examples with known outcomes. At each possible branch, the program tries many candidate questions. For a numerical feature, it may test several cutoff values found in the data.
For a category such as transport type, it may separate one category from the rest or form groups of categories. It compares how well each choice separates outcomes.
A useful split puts similar cases together, so the next prediction is less uncertain. The program usually favours the split with the largest improvement, then repeats the process inside each new group.
The choice of split can be understood with a small example. Suppose a school has data on attendance, homework completion, and final course result. A group containing equal numbers of passes and fails is hard to predict.
If a split based on homework completion produces one group with mostly passes and another with mostly fails, it is valuable. The score for the split must consider group size.
A rule that creates a tiny perfectly pure group but leaves nearly all records mixed may not help much. This is why the impurity of child groups is weighted by the number of examples in each group.
A tree can make mistakes when its training data is limited or unrepresentative. A very deep tree may create rules that fit random accidents in the sample. For example, it might learn that students with a particular locker number are likely to pass, simply because of a coincidence in past records.
Such a rule may fail completely for new students. A validation set helps detect this problem.
It contains data held back during training, so it provides a fairer test of performance. Pruning removes weak lower branches, while limits on depth, minimum group size, or minimum improvement can stop the tree from growing unnecessarily.
Decision trees appear in many familiar systems. A bank may use them as one part of a model that estimates loan risk. A hospital may use carefully tested rules to support clinical decisions.
An online shop may group customers to estimate which items they might buy. In science class, a tree can classify organisms from observable features. Students should remember that a prediction is not a fact about an individual.
It is an estimate based on patterns in earlier data. They should inspect which variables were used, check whether important groups were underrepresented, and consider whether a rule could reflect unfair past decisions. Easy-to-read rules are helpful, but they still need evidence, testing, and responsible interpretation.
Key Facts
- A decision tree predicts by following a path from the root node through decision nodes to a leaf node.
- For classification, a leaf often predicts the most common class among the training examples in that leaf.
- Gini impurity for a node is G = 1 - sum(p_i^2), where p_i is the fraction of examples in class i.
- Entropy for a node is H = -sum(p_i log2(p_i)), where lower entropy means a purer group.
- Information gain is IG = impurity(parent) - weighted average impurity(children).
- Random forests combine many decision trees and often predict by majority vote for classification or averaging for regression.
Vocabulary
- Root node
- The root node is the first node in a decision tree where all training data enter before any splits occur.
- Split
- A split is a rule that divides data into groups based on the value of one feature.
- Leaf node
- A leaf node is an endpoint of the tree that gives the final prediction.
- Impurity
- Impurity measures how mixed the classes are within a node, with lower impurity meaning the node is more uniform.
- Overfitting
- Overfitting happens when a model matches the training data too closely and performs poorly on new data.
Common Mistakes to Avoid
- Choosing the split with the most branches automatically, which is wrong because a split should be judged by how much it improves prediction or reduces impurity.
- Reading every branch in the tree, which is wrong because each prediction follows only one path from the root to one leaf.
- Assuming a deeper tree is always better, which is wrong because very deep trees can memorize noise and overfit the training data.
- Ignoring the size of leaf groups, which is wrong because predictions from very small leaves can be unstable and sensitive to individual data points.
Practice Questions
- 1 A node contains 30 examples: 18 are class A and 12 are class B. Compute the Gini impurity of the node using G = 1 - sum(p_i^2).
- 2 A parent node with 100 examples has Gini impurity 0.48. A split creates a left child with 40 examples and Gini impurity 0.20, and a right child with 60 examples and Gini impurity 0.30. Compute the weighted child impurity and the information gain.
- 3 A decision tree performs very well on training data but much worse on test data. Explain what is likely happening and describe one way to reduce the problem.