Multiple regression is a statistical method for predicting one outcome using two or more predictor variables. It is useful when a result depends on several factors at the same time, such as predicting a house price from size, location, and age. Instead of fitting a line through points in two dimensions, multiple regression fits a prediction surface in higher dimensions.
This lets us estimate how each predictor is related to the outcome while accounting for the others.
The basic model is y = b0 + b1x1 + b2x2 + ... + bkxk + e, where each coefficient describes the expected change in the outcome for a one-unit change in that predictor. The phrase holding other variables constant is essential because each coefficient is interpreted after controlling for the other predictors in the model. R-squared shows the proportion of outcome variation explained by the model, but it usually increases when predictors are added.
Adjusted R-squared is often better for comparing models because it penalizes unnecessary predictors.
Key Facts
- Multiple regression model: y = b0 + b1x1 + b2x2 + ... + bkxk + e.
- b0 is the intercept, the predicted value of y when all predictors equal 0.
- bj is the expected change in y for a 1-unit increase in xj, holding all other predictors constant.
- Predicted value: yhat = b0 + b1x1 + b2x2 + ... + bkxk.
- Residual: e = y - yhat, the difference between the observed value and the predicted value.
- Adjusted R-squared = 1 - [(1 - R-squared)(n - 1)/(n - k - 1)], where n is sample size and k is the number of predictors.
Vocabulary
- Multiple regression
- A statistical model that predicts one outcome variable using two or more predictor variables.
- Coefficient
- A number in a regression model that estimates how much the predicted outcome changes when one predictor increases by one unit, holding the others constant.
- Intercept
- The predicted value of the outcome when every predictor in the model equals zero.
- Residual
- The difference between an observed outcome value and the value predicted by the regression model.
- Adjusted R-squared
- A version of R-squared that adjusts for the number of predictors and sample size to discourage adding weak predictors.
Common Mistakes to Avoid
- Interpreting a coefficient without holding other variables constant is wrong because multiple regression estimates each predictor's effect after accounting for the other predictors.
- Assuming a larger R-squared always means a better model is wrong because R-squared can increase just by adding more predictors, even weak ones.
- Treating correlation as causation is wrong because regression can show association, but causal claims need study design, randomization, or strong supporting evidence.
- Ignoring multicollinearity is wrong because highly related predictors can make coefficient estimates unstable and hard to interpret.
Practice Questions
- 1 A model predicts exam score using yhat = 42 + 5(study hours) + 3(practice tests). What is the predicted score for a student who studies 6 hours and takes 4 practice tests?
- 2 A regression model has R-squared = 0.80, n = 50, and k = 4 predictors. Calculate the adjusted R-squared using Adjusted R-squared = 1 - [(1 - R-squared)(n - 1)/(n - k - 1)].
- 3 A salary model includes years of experience and education level. Explain what it means if the coefficient for years of experience is 2500, holding education level constant.