Least squares data fitting is a method for finding the line or curve that best matches a set of data points. Students use it to model trends, make predictions, and judge how well a model represents real data. This cheat sheet helps organize the formulas and interpretation steps needed for regression problems.
It is especially useful when working with scatterplots, calculators, spreadsheets, or data science tools.
The main idea is to choose model parameters that make the sum of squared residuals as small as possible. For a linear model y = mx + b, the slope and intercept can be found using summary statistics or matrix equations. Residuals, correlation, and R-squared help measure the quality of the fit.
A good model should have small, pattern-free residuals and make sense in the context of the data.
Key Facts
- A residual is the prediction error for one data point, calculated as residual = observed y - predicted y.
- The least squares method chooses model parameters that minimize SSE = sum(residuals^2).
- For a line y = mx + b, the least squares slope is m = sum((x - xbar)(y - ybar)) / sum((x - xbar)^2).
- For a line y = mx + b, the least squares intercept is b = ybar - m xbar.
- The correlation coefficient r measures the strength and direction of a linear relationship, with -1 <= r <= 1.
- The coefficient of determination is R-squared = 1 - SSE/SST, where SST = sum((y - ybar)^2).
- In matrix form, the least squares solution to Ax = b satisfies the normal equation A^T A x = A^T b.
- Extrapolation means making predictions outside the data range, and it is usually less reliable than interpolation.
Vocabulary
- Least Squares
- A fitting method that chooses model parameters by minimizing the sum of the squared prediction errors.
- Residual
- The vertical difference between an observed data value and the value predicted by the model.
- Regression Line
- A line of best fit, usually written y = mx + b, that models the trend in a set of data.
- Correlation Coefficient
- A number r from -1 to 1 that describes the direction and strength of a linear association.
- R-squared
- A statistic that gives the fraction of variation in y explained by the fitted model.
- Normal Equation
- The matrix equation A^T A x = A^T b used to find a least squares solution when a system has more equations than unknowns.
Common Mistakes to Avoid
- Using predicted y - observed y for residuals without checking the class convention, because many courses define residual = observed y - predicted y.
- Forgetting to square residuals before adding them, because least squares minimizes SSE = sum(residuals^2), not the sum of raw errors.
- Treating a large r value as proof of cause and effect, because correlation only describes association and does not prove causation.
- Using the regression line far outside the data range, because extrapolated predictions can be unreliable when the trend changes.
- Assuming R-squared always proves the model is appropriate, because a high R-squared can still come with curved residual patterns or influential outliers.
Practice Questions
- 1 For the data points (1, 2), (2, 3), and (3, 5), find the least squares line y = mx + b.
- 2 A model predicts y = 12 for a data point with observed value y = 15. Find the residual and the squared residual.
- 3 For a data set, SSE = 20 and SST = 80. Calculate R-squared and interpret what it means.
- 4 A scatterplot has a strong curved pattern, but a linear regression gives R-squared = 0.91. Explain why the linear model might still be a poor choice.