Multicollinearity occurs when predictors in a regression model contain overlapping information. This cheat sheet helps students recognize when correlated explanatory variables make coefficient estimates unstable or hard to interpret. It is especially useful for multiple linear regression, model diagnostics, and applied data analysis.
The goal is to separate prediction accuracy from reliable inference about individual predictors.
The main diagnostic is the variance inflation factor, written as , where comes from regressing predictor on the other predictors. A large means the variance of is inflated because is strongly explained by the other predictors. Tolerance is the reciprocal, .
Multicollinearity does not necessarily bias least squares estimates, but it can increase standard errors, widen confidence intervals, and make signs or significance tests unreliable.
Key Facts
- Multicollinearity means one predictor is approximately a linear combination of other predictors, such as .
- The variance inflation factor for predictor is , where is from an auxiliary regression of on the other predictors.
- Tolerance is , so small tolerance indicates stronger multicollinearity.
- If , then , meaning no variance inflation from linear dependence with the other predictors.
- The standard error inflation factor is , so is multiplied by compared with an uncorrelated predictor setting.
- A common warning threshold is , while is often treated as severe, though context matters.
- Ordinary least squares remains unbiased under multicollinearity if the usual exogeneity condition holds.
- Perfect multicollinearity makes singular, so the ordinary least squares estimator cannot be computed.
Vocabulary
- Multicollinearity
- A condition in multiple regression where two or more predictors are strongly linearly related.
- Variance Inflation Factor
- A diagnostic measure that shows how much multicollinearity inflates the variance of .
- Tolerance
- The quantity that measures the proportion of variation in not explained by the other predictors.
- Auxiliary Regression
- A regression used to predict one explanatory variable from the remaining explanatory variables in order to compute .
- Perfect Multicollinearity
- A situation where a predictor is an exact linear combination of other predictors, making regression coefficients not uniquely estimable.
- Coefficient Instability
- A pattern where estimated regression coefficients change substantially when variables or observations are added or removed.
Common Mistakes to Avoid
- Treating a high for the main regression as proof of multicollinearity is wrong because VIF uses from an auxiliary regression for each predictor.
- Using only pairwise correlations to rule out multicollinearity is wrong because one predictor can be explained by a combination of several others even when no single pairwise correlation is extreme.
- Concluding that multicollinearity biases is wrong because multicollinearity mainly increases variance and standard errors when still holds.
- Automatically deleting every variable with is wrong because theory, prediction goals, sample size, and variable importance should guide model changes.
- Interpreting individual coefficients without considering scale and related predictors is wrong because correlated predictors can make signs, magnitudes, and -values unstable.
Practice Questions
- 1 A predictor has auxiliary regression value . Compute and .
- 2 A regression output gives for . By what factor is the standard error of inflated, using ?
- 3 If from regressing on all other predictors, compute and explain whether this suggests serious multicollinearity.
- 4 A model has strong predictive accuracy but several important predictors have large values and nonsignificant tests. Explain why prediction can still be good while individual coefficient inference is weak.
Understanding Multicollinearity and the Variance Inflation Factor
Regression estimates each predictor's effect while holding the other predictors fixed. That phrase creates the main difficulty. If two predictors nearly move together, the data contain very few cases where one changes while the other stays similar.
The model must divide a shared pattern between their coefficients. Small changes in the sample can then shift a large part of the effect from one coefficient to the other.
This is why a coefficient may change sign after a related variable is added, even when both variables have sensible relationships with the outcome. The issue is about the limited independent variation available for estimating separate effects.
An auxiliary regression is useful because it asks how much unique information a predictor has left after the others are used. Focus on the residuals from that regression. They represent the part of the predictor that the other predictors cannot account for.
The main regression relies on this leftover part to estimate that predictor's separate coefficient. When the residuals have very little spread, the separate estimate rests on a weak signal. Pairwise correlations can miss this problem.
A predictor may have only moderate correlation with each other predictor separately, yet be closely reconstructed by several predictors taken together. This is why checking a full set of variance inflation factors is more informative than scanning one correlation table.
Students often meet this issue in data with related measurements. A health study might include body mass, waist size, and body fat percentage. An education study might include family income, parent education, and neighborhood resources.
A business model might use advertising spending, sales staff size, and store count. These variables can all be useful descriptions, but the data may not clearly identify their individual contributions. Predictions for new cases can remain fairly accurate because the combined information is strong.
Claims about which single factor matters most require much more care. Wide intervals, large changes across similar model specifications, and coefficients that conflict with subject knowledge are practical warning signs.
There is no universal rule that demands removing a variable at a particular variance inflation factor value. First check whether predictors measure nearly the same underlying idea. If they do, choose one using theory, measurement quality, or the purpose of the analysis.
A combined index can sometimes match the research goal better than several competing measures. For interaction terms and polynomial terms, centering a predictor around its mean can reduce collinearity created by the construction of those terms. It does not solve overlap between genuinely related variables.
Collecting more observations helps only when the new data include meaningful independent variation. Perfect overlap needs a model change, such as omitting one redundant indicator category. When reporting results, state that coefficient estimates are sensitive to correlated predictors rather than treating a single significance result as decisive.