A decision tree is a machine learning model that makes predictions by asking a sequence of yes or no questions or choosing among labeled options. It looks like a flowchart, with a starting question at the top, branches for possible answers, and final predictions at the bottom. Decision trees matter because they are easy to read, explain, and connect to everyday choices like sorting emails, recommending videos, or deciding whether an image contains a cat.
They are often one of the first AI models students learn because the logic is visible instead of hidden.
Understanding AI & Machine Learning: Decision Trees Explained
A tree is built from examples that already have known outcomes. Each example has features, which are pieces of input information such as screen size, word count, account age, or past purchases. During training, the program tries many possible rules for each feature.
For a number, it might test whether a value is below a chosen cutoff. For a label, it might group one option against the others. It chooses a rule that makes the resulting groups as different as possible in their known outcomes.
Then it repeats this process inside each new group. The order of the rules is learned from the data, not written by a person in advance.
The model needs a way to measure whether a group is mixed or mostly uniform. A group containing only spam messages is easy to classify. A group with nearly equal numbers of spam and ordinary messages is uncertain.
Gini impurity gives a score for this mixture. It is found by taking one minus the sum of each class proportion squared. Entropy measures uncertainty in a related way, using class proportions and logarithms.
Lower impurity or lower entropy means cleaner groups. A useful split produces a large drop in the weighted impurity across its child groups.
This drop is called information gain. The weighting matters because a tiny, perfectly clean group should not outweigh a large, messy group.
Trees can learn patterns that are too specific to their training examples. This problem is called overfitting. For example, a tree might keep adding rules until it memorizes unusual details from a small set of customer records.
It may then perform poorly on new customers. To control this, developers limit the maximum depth, require a minimum number of examples before making a split, or remove weak branches after training. This removal is called pruning.
Students should understand that a highly accurate training result is not enough. The model must be tested on separate data that it did not see while learning.
In real systems, the choice of features matters as much as the tree rules. A loan model may use income, repayment history, and employment length. Some features can reflect unfair past decisions or differences between groups of people.
A tree can copy those patterns even when a sensitive feature is not included directly. Checking accuracy for different groups is important. Feature importance scores can show which inputs influenced many splits, but they do not prove that a feature caused an outcome.
Decision trees are useful for understanding one model's reasoning. Larger systems often combine many trees in a random forest or boosting model, which can improve accuracy but makes the final decision harder to explain.
Key Facts
- A decision tree starts at a root node, follows branches based on feature values, and ends at a leaf node with a prediction.
- For classification, the output is a category, such as spam or not spam.
- For regression, the output is a number, such as predicted price or temperature.
- Gini impurity measures mixed labels in a group: Gini = 1 - sum(p_i^2).
- Entropy is another measure of uncertainty: Entropy = -sum(p_i log2(p_i)).
- Information gain measures how useful a split is: Gain = impurity before split - weighted impurity after split.
Vocabulary
- Decision Tree
- A decision tree is a machine learning model that predicts an answer by following a branching set of rules.
- Root Node
- The root node is the first question or test at the top of a decision tree.
- Branch
- A branch is a path from one node to another that represents the result of a test.
- Leaf Node
- A leaf node is an endpoint of the tree where the final prediction is made.
- Feature
- A feature is an input variable the model uses to make a decision, such as age, color, score, or time.
Common Mistakes to Avoid
- Thinking the first split is chosen randomly. The model tests possible splits and usually chooses the one that best separates the training data.
- Confusing a branch with a prediction. A branch is only a path based on an answer, while the prediction happens at a leaf node.
- Assuming a deeper tree is always better. Very deep trees can memorize training data and perform poorly on new examples.
- Ignoring the training data labels. A decision tree learns from examples with known answers, so incorrect or biased labels can lead to unreliable predictions.
Practice Questions
- 1 A small dataset has 10 examples at a node: 6 are labeled Yes and 4 are labeled No. Compute the Gini impurity using Gini = 1 - sum(p_i^2).
- 2 A split sends 8 examples to the left node with Gini 0.25 and 12 examples to the right node with Gini 0.50. What is the weighted Gini impurity after the split?
- 3 A decision tree for recommending whether to play outside first asks, Is it raining? Explain why this might be a useful root question, and describe one situation where it might not be enough to make a good prediction.