Multicollinearity occurs when predictors in a regression model contain overlapping information. This cheat sheet helps students recognize when correlated explanatory variables make coefficient estimates unstable or hard to interpret. It is especially useful for multiple linear regression, model diagnostics, and applied data analysis.
The goal is to separate prediction accuracy from reliable inference about individual predictors.
The main diagnostic is the variance inflation factor, written as , where comes from regressing predictor on the other predictors. A large means the variance of is inflated because is strongly explained by the other predictors. Tolerance is the reciprocal, .
Multicollinearity does not necessarily bias least squares estimates, but it can increase standard errors, widen confidence intervals, and make signs or significance tests unreliable.
Key Facts
- Multicollinearity means one predictor is approximately a linear combination of other predictors, such as .
- The variance inflation factor for predictor is , where is from an auxiliary regression of on the other predictors.
- Tolerance is , so small tolerance indicates stronger multicollinearity.
- If , then , meaning no variance inflation from linear dependence with the other predictors.
- The standard error inflation factor is , so is multiplied by compared with an uncorrelated predictor setting.
- A common warning threshold is , while is often treated as severe, though context matters.
- Ordinary least squares remains unbiased under multicollinearity if the usual exogeneity condition holds.
- Perfect multicollinearity makes singular, so the ordinary least squares estimator cannot be computed.
Vocabulary
- Multicollinearity
- A condition in multiple regression where two or more predictors are strongly linearly related.
- Variance Inflation Factor
- A diagnostic measure that shows how much multicollinearity inflates the variance of .
- Tolerance
- The quantity that measures the proportion of variation in not explained by the other predictors.
- Auxiliary Regression
- A regression used to predict one explanatory variable from the remaining explanatory variables in order to compute .
- Perfect Multicollinearity
- A situation where a predictor is an exact linear combination of other predictors, making regression coefficients not uniquely estimable.
- Coefficient Instability
- A pattern where estimated regression coefficients change substantially when variables or observations are added or removed.
Common Mistakes to Avoid
- Treating a high for the main regression as proof of multicollinearity is wrong because VIF uses from an auxiliary regression for each predictor.
- Using only pairwise correlations to rule out multicollinearity is wrong because one predictor can be explained by a combination of several others even when no single pairwise correlation is extreme.
- Concluding that multicollinearity biases is wrong because multicollinearity mainly increases variance and standard errors when still holds.
- Automatically deleting every variable with is wrong because theory, prediction goals, sample size, and variable importance should guide model changes.
- Interpreting individual coefficients without considering scale and related predictors is wrong because correlated predictors can make signs, magnitudes, and -values unstable.
Practice Questions
- 1 A predictor has auxiliary regression value . Compute and .
- 2 A regression output gives for . By what factor is the standard error of inflated, using ?
- 3 If from regressing on all other predictors, compute and explain whether this suggests serious multicollinearity.
- 4 A model has strong predictive accuracy but several important predictors have large values and nonsignificant tests. Explain why prediction can still be good while individual coefficient inference is weak.