Least squares data fitting is a method for finding the line or curve that best matches a set of data points. Students use it to model trends, make predictions, and judge how well a model represents real data. This cheat sheet helps organize the formulas and interpretation steps needed for regression problems.
It is especially useful when working with scatterplots, calculators, spreadsheets, or data science tools.
The main idea is to choose model parameters that make the sum of squared residuals as small as possible. For a linear model y = mx + b, the slope and intercept can be found using summary statistics or matrix equations. Residuals, correlation, and R-squared help measure the quality of the fit.
A good model should have small, pattern-free residuals and make sense in the context of the data.
Key Facts
- A residual is the prediction error for one data point, calculated as residual = observed y - predicted y.
- The least squares method chooses model parameters that minimize SSE = sum(residuals^2).
- For a line y = mx + b, the least squares slope is m = sum((x - xbar)(y - ybar)) / sum((x - xbar)^2).
- For a line y = mx + b, the least squares intercept is b = ybar - m xbar.
- The correlation coefficient r measures the strength and direction of a linear relationship, with -1 <= r <= 1.
- The coefficient of determination is R-squared = 1 - SSE/SST, where SST = sum((y - ybar)^2).
- In matrix form, the least squares solution to Ax = b satisfies the normal equation A^T A x = A^T b.
- Extrapolation means making predictions outside the data range, and it is usually less reliable than interpolation.
Vocabulary
- Least Squares
- A fitting method that chooses model parameters by minimizing the sum of the squared prediction errors.
- Residual
- The vertical difference between an observed data value and the value predicted by the model.
- Regression Line
- A line of best fit, usually written y = mx + b, that models the trend in a set of data.
- Correlation Coefficient
- A number r from -1 to 1 that describes the direction and strength of a linear association.
- R-squared
- A statistic that gives the fraction of variation in y explained by the fitted model.
- Normal Equation
- The matrix equation A^T A x = A^T b used to find a least squares solution when a system has more equations than unknowns.
Common Mistakes to Avoid
- Using predicted y - observed y for residuals without checking the class convention, because many courses define residual = observed y - predicted y.
- Forgetting to square residuals before adding them, because least squares minimizes SSE = sum(residuals^2), not the sum of raw errors.
- Treating a large r value as proof of cause and effect, because correlation only describes association and does not prove causation.
- Using the regression line far outside the data range, because extrapolated predictions can be unreliable when the trend changes.
- Assuming R-squared always proves the model is appropriate, because a high R-squared can still come with curved residual patterns or influential outliers.
Practice Questions
- 1 For the data points (1, 2), (2, 3), and (3, 5), find the least squares line y = mx + b.
- 2 A model predicts y = 12 for a data point with observed value y = 15. Find the residual and the squared residual.
- 3 For a data set, SSE = 20 and SST = 80. Calculate R-squared and interpret what it means.
- 4 A scatterplot has a strong curved pattern, but a linear regression gives R-squared = 0.91. Explain why the linear model might still be a poor choice.
Understanding Least Squares Data Fitting
Squaring errors does more than prevent positive and negative errors from cancelling. It makes large misses count much more heavily than small misses. An error twice as large contributes four times as much to the total.
This is useful when large mistakes are genuinely more serious, but it means one unusual point can pull a fitted line noticeably. For a straight line, the balancing process has an important result. The line goes through the point formed by the average input value and the average output value.
This gives a quick check on calculator work. If a reported regression line misses that average point, something is wrong with the data entry or calculation.
The normal equations come from viewing fitting as a geometry problem. Each column of a data table represents one feature used for prediction, such as study time, temperature, or advertising cost. The fitted values are built from combinations of those columns.
After fitting, the leftover error has no overall direction that matches any column in the model. In geometric language, the residual vector is perpendicular to the space of possible fitted values. This is why the equations are called normal equations, since normal means perpendicular.
In a simple line fit, this condition says that the residuals balance in several useful ways. Their total is zero, and they have no remaining linear connection with the input values.
A small total squared error is not enough to trust a model. Students should always inspect a residual plot. Random scatter above and below zero supports a linear model.
A curved pattern suggests that the relationship bends, so a quadratic or another model may fit better. A fan shaped pattern shows that the error spread changes as the input grows. This can make predictions less dependable in part of the range.
It is important to separate an outlier from a high leverage point. An outlier has an unusual output compared with nearby points. A high leverage point has an unusually extreme input value.
A point can be both, and then it may strongly change the slope. Removing data is not automatically acceptable. First check for a recording error, then consider whether the point represents a real but unusual case.
R squared compares a model with a basic prediction that uses the average output for every case. A high value means the model improves greatly on that basic prediction within the observed data. It does not prove that the model is scientifically correct.
It does not show that one variable causes another. For example, ice cream sales and sunburn cases may rise together because warm weather affects both. Predictions should include the idea of uncertainty.
Even a strong trend has natural variation around it, so a predicted value is an estimate rather than a guarantee. Interpolation within the measured range is usually safer because the model has evidence there. Outside that range, unseen changes can make a straight trend fail quickly.