In regression, a single unusual data point can change the equation of a fitted line and lead to misleading predictions. Some unusual points are outliers in the y-direction, some are far away in the x-direction, and some strongly affect the slope or intercept. Understanding these points matters because regression is often used to make decisions from real data.
A good analysis checks whether the model describes the overall pattern or is being pulled by a few observations.
Leverage measures how far a point's x-value is from the center of the x-values, while influence measures how much the fitted regression changes when that point is removed. A high-leverage point is not automatically influential, because it may still fall close to the existing trend. Cook's distance combines residual size and leverage to flag points that may strongly affect the regression line.
Comparing the original regression line with the line after removing or adding an unusual point helps reveal whether conclusions are stable.
Understanding Statistics: Influential Points and Leverage
A fitted line is chosen by a least squares rule. The rule makes the total of the squared vertical errors as small as possible. Squaring matters because one large error counts much more than several small ones.
A point near the middle of the horizontal data can sit far above or below the line. It increases the total error, yet the line can often stay in nearly the same position because many points surround it.
A point at the far edge has more power over the line's tilt. Changing its vertical position can rotate the line around the main cluster of data.
Think about a class study of weekly study time and test score. Most students may study from two to eight hours. One student reports twenty hours.
That student lies far beyond the usual range, so the fitted line must extend toward that value. If the reported score agrees with the general pattern, the point may support the same slope as the other students. If the score is very different, it can make study time appear much more or much less important than the rest of the class data suggests.
The important issue is not whether a value looks unusual by itself. The issue is whether the result of the analysis depends heavily on it.
Several checks help separate a data problem from a meaningful rare case. A residual plot shows whether errors are scattered randomly or whether one observation stands far from the rest. A plot of the raw data is essential because a table of numbers can hide the shape of the situation.
Analysts often refit the model with a flagged observation left out. They compare the slope, intercept, predicted values, and any claim based on the slope.
Cook's distance is useful because it considers both an observation's position along the horizontal axis and its mismatch with the fitted line. A large value is a signal to investigate, not an automatic reason to delete data.
Students should be careful not to remove observations simply because they are inconvenient. First check for typing mistakes, wrong units, duplicated records, or a measurement taken under different conditions. A value can be real and still require a different model.
For example, a very large business may behave differently from small businesses, or an extreme weather day may follow different physical conditions. Predictions beyond the observed horizontal range need special caution. A line that is strongly shaped by one distant point can give very unreliable predictions there.
Reporting a sensitivity check makes the conclusion more honest. It shows whether the main pattern remains when a single observation is examined closely.
Key Facts
- Residual = observed y - predicted y = y - y_hat.
- A vertical outlier has a large residual but may have low leverage if its x-value is near the center of the data.
- A high-leverage point has an x-value far from the mean of x, so it can strongly affect the fitted line.
- An influential point is one whose removal noticeably changes the regression equation, predictions, or conclusions.
- Simple linear regression model: y_hat = b0 + b1x.
- Cook's distance is often flagged for review when D_i > 4/n, where n is the sample size.
Vocabulary
- Outlier
- An outlier is a data point that lies far from the general pattern of the other observations.
- Leverage
- Leverage measures how far an observation's x-value is from the average x-value in the data set.
- Influential point
- An influential point is an observation that substantially changes the fitted regression line when it is removed.
- Residual
- A residual is the vertical difference between an observed y-value and the y-value predicted by the regression model.
- Cook's distance
- Cook's distance is a diagnostic measure that combines residual size and leverage to estimate how much one point affects a regression fit.
Common Mistakes to Avoid
- Calling every outlier influential is wrong because a point can have a large residual but little effect on the slope if its x-value is near the center of the data.
- Ignoring high-leverage points that lie close to the line is wrong because such points may still control the slope and make the trend look more certain than it is.
- Deleting unusual points automatically is wrong because a point may represent a real case, and removal should be justified by measurement error, data entry error, or a clear analysis plan.
- Using only the correlation coefficient to judge a regression is wrong because correlation can be greatly increased or decreased by a single high-leverage observation.
Practice Questions
- 1 A regression model predicts y_hat = 2 + 3x. For a point with x = 4 and observed y = 20, calculate the predicted value and the residual.
- 2 A data set has n = 40 observations. Using the rule D_i > 4/n, what Cook's distance value should be used as a rough cutoff for reviewing influential points?
- 3 A point is far to the right of all other x-values but lies almost exactly on the current regression line. Explain whether it is an outlier, a high-leverage point, and likely influential.