Cook's distance measures how much a regression model changes when one observation is removed. This cheat sheet helps students identify influential points that can strongly affect fitted coefficients, predictions, and conclusions. It is especially useful in college statistics when checking linear regression assumptions and diagnosing unusual data points.
The core idea is to combine residual size with leverage, because a point is influential when it is both unusual in outcome and positioned far from the center of the predictor values. Important tools include the hat matrix, leverage values, studentized residuals, and Cook's distance. Rules of thumb such as or help flag observations for further investigation, not automatic deletion.
Key Facts
- In linear regression, Cook's distance for observation can be written as .
- The residual for observation is , which measures vertical distance from the fitted regression line or plane.
- The leverage value is , the th diagonal entry of the hat matrix .
- The average leverage in a regression model is , where is the number of estimated parameters including the intercept.
- A common leverage flag is or, more conservatively, .
- A common Cook's distance screening rule is , while is often considered a stronger warning sign.
- An observation with a large residual but low leverage may be an outlier without being highly influential.
- An observation with high leverage and a small residual may have little influence on the fitted model, because it already follows the fitted pattern.
Vocabulary
- Cook's distance
- A diagnostic measure that estimates how much the fitted regression model changes when one observation is removed.
- Influential point
- An observation that substantially changes regression coefficients, fitted values, or statistical conclusions when included or excluded.
- Leverage
- A measure of how far an observation's predictor values are from the center of the predictor space.
- Residual
- The difference between an observed response and its predicted value, written as .
- Hat matrix
- The matrix that maps observed responses to fitted values in linear regression.
- Studentized residual
- A residual scaled by an estimate of its standard deviation so residuals from different observations are easier to compare.
Common Mistakes to Avoid
- Deleting every point with large Cook's distance, which is wrong because a flagged value requires investigation of data quality, context, and model sensitivity before removal.
- Treating as an absolute rule, which is wrong because it is only a screening guideline and not a formal hypothesis test.
- Confusing outliers with influential points, which is wrong because an outlier has an unusual response value while influence depends on both residual size and leverage.
- Ignoring high-leverage observations with small residuals, which is wrong because they may still control the fitted line even if they do not appear unusual on a residual plot.
- Forgetting that includes the intercept, which is wrong because leverage averages and Cook's distance formulas use the total number of estimated regression parameters.
Practice Questions
- 1 A regression has observations and estimated parameters. Compute the average leverage and the common high-leverage cutoff .
- 2 For one observation, , , , and . Compute .
- 3 A data set has observations. Use the rule to find the Cook's distance screening cutoff.
- 4 Explain why a point with high leverage but a very small residual might not have a large Cook's distance, yet still deserves attention in regression diagnostics.
Understanding Cook's Distance and Influential Points
Cook's distance is best understood as a comparison between two fitted models. First, fit the regression using every case. Then temporarily leave out one case and fit it again.
If the slope, intercept, or predicted values move noticeably, that case has influence. The statistic summarizes this movement across the model rather than looking only at one vertical error.
A point can pull a line toward itself, especially when it sits at an extreme predictor value. This is why a visually reasonable line can still depend heavily on just one or two records.
Residuals and leverage describe different kinds of unusual behavior. A residual concerns the response value after the predictors have been considered. For example, a student who studies a typical number of hours but earns a very unexpected score can have a large residual.
Leverage concerns the predictor values before the response is considered. A student who studies far more hours than everyone else has high leverage.
Such a student gives the model information about the far end of the study range, but the estimate there may be unstable if no similar students are present. Influence becomes important when these features combine in a way that changes the fitted relationship.
The number of predictors matters when judging leverage. Adding predictors gives the model more flexibility and raises the typical leverage value. A case may look ordinary on each predictor separately yet be unusual in their combination.
For instance, a customer may have an ordinary age and income, while the particular age and income pair is rare in the data. Regression software can calculate leverage for this situation even when a graph cannot show every dimension. Students should learn to inspect diagnostic plots, not just a table of results.
A residual versus fitted plot can reveal patterns in errors. A leverage plot or influence plot can reveal cases that deserve closer checking.
A flagged case is not proof of a mistake. It may be a data entry error, a measurement problem, a real but rare event, or an important subgroup that the model should represent. Check the original record first.
Then consider whether the model form makes sense for that case. A curved relationship, a missing predictor, or unequal error spread can make a valid observation appear troublesome. It is useful to report results with the case included and with it removed when conclusions differ.
Do not delete a point merely because it is inconvenient. The goal of regression diagnostics is to understand how dependent the conclusion is on particular observations and whether the model describes the population fairly.