Multiple regression is a statistical method for predicting one outcome using two or more predictor variables. It is useful when a result depends on several factors at the same time, such as predicting a house price from size, location, and age. Instead of fitting a line through points in two dimensions, multiple regression fits a prediction surface in higher dimensions.
This lets us estimate how each predictor is related to the outcome while accounting for the others.
The basic model is y = b0 + b1x1 + b2x2 + ... + bkxk + e, where each coefficient describes the expected change in the outcome for a one-unit change in that predictor. The phrase holding other variables constant is essential because each coefficient is interpreted after controlling for the other predictors in the model. R-squared shows the proportion of outcome variation explained by the model, but it usually increases when predictors are added.
Adjusted R-squared is often better for comparing models because it penalizes unnecessary predictors.
Understanding Statistics: Multiple Regression
A regression program chooses the coefficients by making the residuals as small as possible overall. More precisely, it squares each residual, then finds the set of coefficients with the smallest total of these squared values. Squaring prevents positive and negative errors from cancelling out.
It gives extra weight to large mistakes, which is useful but can make the model sensitive to unusual cases. A single home with an extremely high price, or one student with an unusually large study time, can pull the fitted results strongly. Students should inspect a scatterplot and check for outliers rather than trusting a table of coefficients alone.
The words holding other variables constant describe a comparison that may not be common in the data. For house prices, the coefficient for floor area compares homes with the same location, age, and every other included feature. This helps separate effects that are mixed together.
It does not work perfectly when predictors overlap too much. For example, homes with more bedrooms often have more floor area. The model may then struggle to decide how much of the price difference belongs to each variable.
This issue is called multicollinearity. It can produce coefficients that change greatly when a new predictor is added, even when the overall predictions remain reasonable.
Multiple regression appears in many everyday studies. A school might estimate a final exam score from attendance, earlier test results, homework completion, and study time. A health study might estimate blood pressure from age, activity level, diet, and body mass.
Companies use related models to forecast demand from price, season, advertising, and local conditions. In each case, prediction is not the same as proof of cause. If students who study more tend to earn higher scores, the model shows a relationship after accounting for measured variables.
Motivation, access to tutoring, sleep, and other unmeasured factors may still affect both study time and scores. Careful experiments are stronger evidence for causation.
A useful model needs more than a high adjusted R squared value. Residual plots should show a random cloud around zero. A curved pattern suggests that a straight effect is missing an important shape.
A fan shaped spread suggests that errors become larger for some values, which makes uncertainty estimates less reliable. The errors should be roughly independent, especially for data collected over time, since yesterday's temperature is related to today's temperature. Predictions are safest within the range of the original data.
A model built from small homes should not be trusted automatically for a huge mansion. When comparing models, prefer predictors with a clear reason for inclusion, enough data for the number of variables, and performance checked on new data that were not used to fit the model.
Key Facts
- Multiple regression model: y = b0 + b1x1 + b2x2 + ... + bkxk + e.
- b0 is the intercept, the predicted value of y when all predictors equal 0.
- bj is the expected change in y for a 1-unit increase in xj, holding all other predictors constant.
- Predicted value: yhat = b0 + b1x1 + b2x2 + ... + bkxk.
- Residual: e = y - yhat, the difference between the observed value and the predicted value.
- Adjusted R-squared = 1 - [(1 - R-squared)(n - 1)/(n - k - 1)], where n is sample size and k is the number of predictors.
Vocabulary
- Multiple regression
- A statistical model that predicts one outcome variable using two or more predictor variables.
- Coefficient
- A number in a regression model that estimates how much the predicted outcome changes when one predictor increases by one unit, holding the others constant.
- Intercept
- The predicted value of the outcome when every predictor in the model equals zero.
- Residual
- The difference between an observed outcome value and the value predicted by the regression model.
- Adjusted R-squared
- A version of R-squared that adjusts for the number of predictors and sample size to discourage adding weak predictors.
Common Mistakes to Avoid
- Interpreting a coefficient without holding other variables constant is wrong because multiple regression estimates each predictor's effect after accounting for the other predictors.
- Assuming a larger R-squared always means a better model is wrong because R-squared can increase just by adding more predictors, even weak ones.
- Treating correlation as causation is wrong because regression can show association, but causal claims need study design, randomization, or strong supporting evidence.
- Ignoring multicollinearity is wrong because highly related predictors can make coefficient estimates unstable and hard to interpret.
Practice Questions
- 1 A model predicts exam score using yhat = 42 + 5(study hours) + 3(practice tests). What is the predicted score for a student who studies 6 hours and takes 4 practice tests?
- 2 A regression model has R-squared = 0.80, n = 50, and k = 4 predictors. Calculate the adjusted R-squared using Adjusted R-squared = 1 - [(1 - R-squared)(n - 1)/(n - k - 1)].
- 3 A salary model includes years of experience and education level. Explain what it means if the coefficient for years of experience is 2500, holding education level constant.