Multiple regression is used to predict one response variable from two or more explanatory variables. This cheat sheet helps students organize the model equation, interpret coefficients, and check how well a model fits data. It is especially useful when several factors may affect the same outcome, such as predicting test scores from study time, attendance, and sleep.
Key Facts
- A multiple regression model with two predictors is written as , where is the predicted response.
- The intercept is the predicted value of when all explanatory variables equal , if that situation makes sense in context.
- A slope coefficient estimates the change in for a -unit increase in while all other predictors are held constant.
- A residual is the prediction error for one data point, calculated by .
- The coefficient of determination is the proportion of variation in explained by the regression model, with .
- Adjusted penalizes unnecessary predictors and is often better than for comparing models with different numbers of explanatory variables.
- Multicollinearity occurs when predictors are strongly related to each other, which can make coefficient estimates unstable and hard to interpret.
- A prediction should usually be made only within the range of the original data because extrapolation can be unreliable.
Vocabulary
- Multiple Regression
- A statistical method that predicts one response variable using two or more explanatory variables.
- Response Variable
- The variable being predicted or explained, usually represented by .
- Explanatory Variable
- A variable used to predict the response variable, often represented by , , and so on.
- Coefficient
- A number in the regression equation that shows how a predictor is associated with the predicted response when other predictors are held constant.
- Residual
- The difference between an observed value and its predicted value, calculated as .
- Multicollinearity
- A problem that occurs when explanatory variables are highly correlated with each other.
Common Mistakes to Avoid
- Interpreting a coefficient without saying other variables are held constant is wrong because each slope in multiple regression adjusts for the other predictors in the model.
- Assuming a larger always means a better model is wrong because adding more predictors can increase even when those predictors are not useful.
- Using the model far outside the data range is wrong because extrapolated predictions may not follow the same pattern seen in the sample.
- Treating correlation between predictors as harmless is wrong because strong multicollinearity can make slopes change dramatically when the model changes.
- Confusing residuals with predicted values is wrong because a residual measures error, while is the model's predicted response.
Practice Questions
- 1 A model predicts final exam score with , where is study hours and is hours of sleep. Find when and .
- 2 For one student, the observed score is and the predicted score is . Find the residual .
- 3 In the model , where is years of experience and is commute distance in miles, interpret the coefficient in context.
- 4 A model has a high , but two predictors are strongly correlated with each other. Explain why the model may still be difficult to interpret.
Understanding Multiple Regression
Most regression programs fit a model by making the total of the squared residuals as small as possible. Squaring gives large mistakes extra weight and prevents positive errors from cancelling negative errors. The fitted surface is the best compromise for the data that were collected, not a rule guaranteed to work everywhere.
A few unusual observations can pull this surface toward themselves. Students should inspect the raw data and identify points with unusually large residuals or extreme predictor values before trusting a result.
Each coefficient has meaning only in the context of the other variables in the model. For example, a model of house prices might include floor area, number of bedrooms, and neighborhood. The coefficient for floor area describes a comparison between homes that are alike on the other included features.
This can feel artificial because real variables often change together. Larger homes usually have more bedrooms.
When two predictors carry nearly the same information, small changes in the data can produce very different coefficient values. The model may still predict reasonably well, but its separate explanations become less reliable.
A strong model is not proven by a high R squared value alone. R squared tends to rise when more predictors are added, even if a new predictor contributes little useful information. Adjusted R squared helps reveal this problem because it accounts for model complexity.
A better test is to set aside some observations, build the model using the remaining data, then check predictions on the held-out observations. This is called validation. It shows whether the model learned a useful pattern rather than memorizing random features of one sample.
Residual plots are important because they reveal patterns hidden by summary statistics. Residuals should appear as a fairly random cloud around zero when plotted against predicted values or each predictor. A curved pattern suggests that a straight-line effect is too simple.
A fan-shaped pattern suggests that prediction errors become larger for some values than for others. Data collected over time may have related errors from one day to the next.
Such issues can make uncertainty estimates less dependable. A model can sometimes improve by using a transformed variable, a curved term, or a meaningful interaction between predictors.
Regression describes associations in observed data. It does not automatically show that one variable causes another. For instance, students who sleep more may earn higher scores, but family support, health, or study habits could affect both sleep and scores.
Missing influences can distort coefficients. When making a prediction, use values that are realistic and similar to those in the original sample.
Report that a prediction is an estimate, not a promise. Pay attention to units, data quality, unusual cases, and whether the model makes sense in the real situation.