Linear regression is a method for predicting a numerical output from one or more input features. In its simplest form, it draws the best fitting straight line through a scatter plot of data. This makes it useful for estimating trends, making forecasts, and understanding how one variable changes with another.
It is one of the most important starting models in statistics and machine learning because it is simple, visual, and interpretable.
A linear regression model learns parameters, usually a slope and an intercept, from training data. The model makes predictions using an equation such as y_hat = mx + b, then compares those predictions with the actual values. A cost function, often mean squared error, measures how wrong the model is overall.
Gradient descent is one way to adjust the parameters step by step until the line fits the data as well as possible.
Understanding Statistics: Linear Regression for Prediction
The fitting process gives large mistakes extra weight. A prediction that misses by ten units is not merely twice as bad as one that misses by five units. After each error is squared, it counts four times as much.
This is useful because a line should not ignore serious misses. It creates a weakness too. One unusual data point can pull the line strongly toward itself.
Students should always inspect the scatter plot before trusting a calculated result. A typing error, a faulty sensor, or an exceptional event can change the model more than expected.
A straight line is most useful when the average pattern is roughly straight. Real points do not need to sit exactly on it. Their vertical gaps from the predicted values should look like a random cloud, without a clear curve or widening fan shape.
A curved pattern suggests that the relationship changes at different input values. For example, braking distance may rise much faster at high speed than at low speed.
A fan shaped pattern means prediction errors become larger or smaller across the range. In such cases, a line can still give an answer, but its uncertainty is uneven and its conclusions need care.
With several input features, the model assigns one coefficient to each feature. A house price model might use floor area, age, distance from a station, and number of bedrooms. Each coefficient describes the predicted change linked to one feature while the other included features are held fixed.
This wording matters. Features in real data often move together. Larger homes may have more bedrooms, for instance.
When inputs are strongly related, the separate coefficients can become unstable. The model may predict fairly well, yet an individual coefficient may not provide a reliable explanation of cause.
A useful model must be checked on data that was not used to fit it. Training error tells how closely the model matches examples it has already seen. Test error shows how well it generalizes to new cases.
A low training error with a much higher test error is a warning sign. Predictions outside the observed range need special caution too. A line fitted to temperatures from ten to thirty degrees has no solid reason to remain accurate at sixty degrees.
This is called extrapolation. In school work, record the units, plot the data, check unusual points, compare training and test results, then state the limits of any prediction plainly.
Key Facts
- Simple linear regression model: y_hat = mx + b
- Residual: e = y - y_hat
- Mean squared error: MSE = (1/n)Σ(y_i - y_hat_i)^2
- A positive slope means predicted y increases as x increases.
- The best fit line minimizes the total squared residuals for the training data.
- Gradient descent update idea: new parameter = old parameter - learning rate × gradient
Vocabulary
- Linear regression
- A statistical and machine learning method that predicts a numerical output using a linear equation.
- Input feature
- A measured variable used by the model to make a prediction.
- Prediction
- The output value estimated by a model for a given input.
- Residual
- The difference between an observed value and the value predicted by the regression model.
- Cost function
- A formula that measures how far the model predictions are from the actual data values.
Common Mistakes to Avoid
- Confusing correlation with causation, because a strong linear trend does not prove that the input feature directly causes the output.
- Using the regression line far outside the data range, because extrapolation can give unreliable predictions when the trend changes beyond the observed values.
- Ignoring residual patterns, because curved or fan shaped residual plots show that a straight line may not be an appropriate model.
- Choosing a learning rate that is too large in gradient descent, because the algorithm can overshoot the minimum and fail to converge.
Practice Questions
- 1 A regression model is y_hat = 3x + 2. What prediction does it make when x = 7?
- 2 For three data points, the actual values are 5, 8, and 10, and the predicted values are 4, 9, and 7. Find the residuals and the mean squared error.
- 3 A scatter plot shows points that curve upward instead of forming a roughly straight cloud. Explain why a linear regression model may give poor predictions for this data.