Classification and regression are two major types of supervised learning used in statistics and machine learning. Both start with input data and a known target, then learn a rule for making predictions on new cases. The key difference is the kind of answer they predict: classification predicts a category, while regression predicts a number.
This distinction matters because it determines the model, loss function, evaluation metric, and how the result should be interpreted.
In classification, the model learns boundaries that separate classes, such as spam versus not spam or disease present versus absent. In regression, the model learns a numerical relationship, such as predicting temperature, price, or time. Some algorithms, such as decision trees, random forests, and neural networks, can be adapted for both tasks, but they use different output layers or training goals.
Choosing the correct task type helps avoid misleading predictions and incorrect measures of performance.
Key Facts
- Classification predicts a discrete label, such as cat, dog, or bird.
- Regression predicts a continuous value, such as house price, height, or speed.
- A simple regression model can be written as y = mx + b.
- Classification often estimates class probabilities, such as P(class = spam) = 0.93.
- Common classification metrics include accuracy = correct predictions / total predictions, precision, recall, and F1 score.
- Common regression metrics include MAE = mean(|y - y_hat|), MSE = mean((y - y_hat)^2), and RMSE = sqrt(MSE).
Vocabulary
- Classification
- Classification is a supervised learning task where the model predicts which category or class an input belongs to.
- Regression
- Regression is a supervised learning task where the model predicts a numerical value from input data.
- Target variable
- The target variable is the output the model is trained to predict, such as a class label or a number.
- Decision boundary
- A decision boundary is the line, curve, or surface that separates predicted classes in a classification model.
- Loss function
- A loss function measures how wrong a model's predictions are during training.
Common Mistakes to Avoid
- Treating any number as a regression target is wrong because some numbers are actually category codes, such as 0 for no and 1 for yes.
- Using accuracy to judge a regression model is wrong because regression predictions are usually not exactly equal to the true value.
- Using linear regression for a category prediction without adjustment is wrong because its outputs are not guaranteed to be valid class probabilities.
- Ignoring class imbalance in classification is wrong because a model can look accurate by mostly predicting the majority class.
Practice Questions
- 1 A model predicts whether 200 emails are spam or not spam. It correctly classifies 174 emails. What is its accuracy?
- 2 A regression model predicts house prices with errors of 25,000, 30,000. What is the mean absolute error?
- 3 A hospital wants to predict whether a patient has flu, COVID, or neither based on symptoms. Is this a classification or regression problem, and why?