The least squares regression line is a straight line used to model the relationship between two quantitative variables. It is called a best-fit line because it balances the data points so that the overall prediction error is as small as possible. This matters because regression lets us describe trends, make predictions, and compare how strongly variables move together.
In a scatterplot, the regression line gives a clear visual summary of a data cloud.
Understanding Statistics: The Least Squares Regression Line
A regression line makes one prediction for every possible input value. The vertical gap from a point to that prediction is called its residual. A point above the line has a positive residual because its observed value is greater than predicted.
A point below the line has a negative residual. Looking at residuals is important because a line can appear reasonable on a scatterplot while making a pattern of mistakes.
For a suitable linear model, the residuals should form a fairly random band above and below zero. A curve, a fan shape, or separate clusters in a residual plot can show that a straight line is not the right model.
The word squared in least squares has a practical purpose. If positive and negative residuals were simply added, they could cancel out. A prediction that is too high by ten units would cancel a prediction that is too low by ten units, even though both are large errors.
Squaring makes every error positive. It gives extra weight to large misses, since an error of four units contributes sixteen squared units, while an error of two units contributes four squared units.
This is why one distant point can pull the line strongly toward itself. The least squares line is best according to this particular rule, not necessarily best for every real decision.
The slope describes the predicted change in the response when the explanatory variable rises by one unit. Its units matter. If study time is measured in hours and test score is measured in points, a slope of two means the predicted score rises by two points for each extra hour studied.
This does not prove that studying causes the score increase. Students who study longer may differ in preparation, attendance, sleep, or access to help.
Regression describes an association in the data. A causal claim needs a well designed experiment or strong evidence that other influences have been controlled.
The intercept needs careful interpretation. It is the predicted response when the explanatory variable is zero. Sometimes zero is realistic, such as zero hours of practice.
Sometimes it is outside the data range, such as predicting adult height at zero years of age from a sample containing only teenagers. Predictions are most reliable within the range of observed input values.
Extending a line far beyond that range is called extrapolation, and it can fail when a real relationship changes direction or levels off. Students should inspect the scatterplot before calculating anything, check the units, identify unusual points, and report predictions as estimates rather than exact facts.
Key Facts
- The least squares regression line has the form y hat = a + bx, where b is the slope and a is the y-intercept.
- A residual is residual = observed y - predicted y = y - y hat.
- Least squares means the line minimizes the sum of squared residuals: SSE = sum(y - y hat)^2.
- The slope can be found with b = r(s_y / s_x), where r is correlation, s_y is the standard deviation of y, and s_x is the standard deviation of x.
- The intercept is a = y bar - b x bar, so the regression line always passes through the point (x bar, y bar).
- Example: If b = 2.5, a = 4, and x = 6, then y hat = 4 + 2.5(6) = 19.
Vocabulary
- Least squares regression line
- The line that minimizes the sum of the squared vertical distances between the observed data points and the predicted values.
- Residual
- The vertical difference between an observed y-value and the y-value predicted by the regression line.
- Slope
- The amount the predicted y-value changes for each 1-unit increase in x.
- Intercept
- The predicted y-value when x equals 0, if that value of x is meaningful in context.
- Correlation
- A number from -1 to 1 that describes the direction and strength of a linear relationship between two quantitative variables.
Common Mistakes to Avoid
- Using horizontal distances as residuals is wrong because residuals in ordinary least squares are vertical distances from each data point to the line.
- Forgetting to square residuals is wrong because positive and negative errors would cancel, hiding the total size of prediction errors.
- Interpreting the intercept outside the data context is wrong because x = 0 may not be realistic or included in the observed range.
- Using the regression line to predict far beyond the data is wrong because extrapolation assumes the same linear pattern continues where no data were observed.
Practice Questions
- 1 A regression line is y hat = 12 + 3x. Predict y when x = 7, and find the residual if the observed y-value is 35.
- 2 A data set has r = 0.80, s_x = 5, s_y = 10, x bar = 20, and y bar = 50. Find the slope b, the intercept a, and the regression equation.
- 3 A scatterplot shows a curved pattern, but a least squares regression line is drawn through it. Explain why the line may give misleading predictions even if it passes through the middle of the data.