In regression, a single unusual data point can change the equation of a fitted line and lead to misleading predictions. Some unusual points are outliers in the y-direction, some are far away in the x-direction, and some strongly affect the slope or intercept. Understanding these points matters because regression is often used to make decisions from real data.
A good analysis checks whether the model describes the overall pattern or is being pulled by a few observations.
Leverage measures how far a point's x-value is from the center of the x-values, while influence measures how much the fitted regression changes when that point is removed. A high-leverage point is not automatically influential, because it may still fall close to the existing trend. Cook's distance combines residual size and leverage to flag points that may strongly affect the regression line.
Comparing the original regression line with the line after removing or adding an unusual point helps reveal whether conclusions are stable.
Key Facts
- Residual = observed y - predicted y = y - y_hat.
- A vertical outlier has a large residual but may have low leverage if its x-value is near the center of the data.
- A high-leverage point has an x-value far from the mean of x, so it can strongly affect the fitted line.
- An influential point is one whose removal noticeably changes the regression equation, predictions, or conclusions.
- Simple linear regression model: y_hat = b0 + b1x.
- Cook's distance is often flagged for review when D_i > 4/n, where n is the sample size.
Vocabulary
- Outlier
- An outlier is a data point that lies far from the general pattern of the other observations.
- Leverage
- Leverage measures how far an observation's x-value is from the average x-value in the data set.
- Influential point
- An influential point is an observation that substantially changes the fitted regression line when it is removed.
- Residual
- A residual is the vertical difference between an observed y-value and the y-value predicted by the regression model.
- Cook's distance
- Cook's distance is a diagnostic measure that combines residual size and leverage to estimate how much one point affects a regression fit.
Common Mistakes to Avoid
- Calling every outlier influential is wrong because a point can have a large residual but little effect on the slope if its x-value is near the center of the data.
- Ignoring high-leverage points that lie close to the line is wrong because such points may still control the slope and make the trend look more certain than it is.
- Deleting unusual points automatically is wrong because a point may represent a real case, and removal should be justified by measurement error, data entry error, or a clear analysis plan.
- Using only the correlation coefficient to judge a regression is wrong because correlation can be greatly increased or decreased by a single high-leverage observation.
Practice Questions
- 1 A regression model predicts y_hat = 2 + 3x. For a point with x = 4 and observed y = 20, calculate the predicted value and the residual.
- 2 A data set has n = 40 observations. Using the rule D_i > 4/n, what Cook's distance value should be used as a rough cutoff for reviewing influential points?
- 3 A point is far to the right of all other x-values but lies almost exactly on the current regression line. Explain whether it is an outlier, a high-leverage point, and likely influential.