A decision tree is a machine learning model that makes predictions by asking a sequence of yes or no questions or choosing among labeled options. It looks like a flowchart, with a starting question at the top, branches for possible answers, and final predictions at the bottom. Decision trees matter because they are easy to read, explain, and connect to everyday choices like sorting emails, recommending videos, or deciding whether an image contains a cat.
They are often one of the first AI models students learn because the logic is visible instead of hidden.
Key Facts
- A decision tree starts at a root node, follows branches based on feature values, and ends at a leaf node with a prediction.
- For classification, the output is a category, such as spam or not spam.
- For regression, the output is a number, such as predicted price or temperature.
- Gini impurity measures mixed labels in a group: Gini = 1 - sum(p_i^2).
- Entropy is another measure of uncertainty: Entropy = -sum(p_i log2(p_i)).
- Information gain measures how useful a split is: Gain = impurity before split - weighted impurity after split.
Vocabulary
- Decision Tree
- A decision tree is a machine learning model that predicts an answer by following a branching set of rules.
- Root Node
- The root node is the first question or test at the top of a decision tree.
- Branch
- A branch is a path from one node to another that represents the result of a test.
- Leaf Node
- A leaf node is an endpoint of the tree where the final prediction is made.
- Feature
- A feature is an input variable the model uses to make a decision, such as age, color, score, or time.
Common Mistakes to Avoid
- Thinking the first split is chosen randomly. The model tests possible splits and usually chooses the one that best separates the training data.
- Confusing a branch with a prediction. A branch is only a path based on an answer, while the prediction happens at a leaf node.
- Assuming a deeper tree is always better. Very deep trees can memorize training data and perform poorly on new examples.
- Ignoring the training data labels. A decision tree learns from examples with known answers, so incorrect or biased labels can lead to unreliable predictions.
Practice Questions
- 1 A small dataset has 10 examples at a node: 6 are labeled Yes and 4 are labeled No. Compute the Gini impurity using Gini = 1 - sum(p_i^2).
- 2 A split sends 8 examples to the left node with Gini 0.25 and 12 examples to the right node with Gini 0.50. What is the weighted Gini impurity after the split?
- 3 A decision tree for recommending whether to play outside first asks, Is it raining? Explain why this might be a useful root question, and describe one situation where it might not be enough to make a good prediction.