Linear regression is a method for predicting a numerical output from one or more input features. In its simplest form, it draws the best fitting straight line through a scatter plot of data. This makes it useful for estimating trends, making forecasts, and understanding how one variable changes with another.
It is one of the most important starting models in statistics and machine learning because it is simple, visual, and interpretable.
A linear regression model learns parameters, usually a slope and an intercept, from training data. The model makes predictions using an equation such as y_hat = mx + b, then compares those predictions with the actual values. A cost function, often mean squared error, measures how wrong the model is overall.
Gradient descent is one way to adjust the parameters step by step until the line fits the data as well as possible.
Key Facts
- Simple linear regression model: y_hat = mx + b
- Residual: e = y - y_hat
- Mean squared error: MSE = (1/n)Σ(y_i - y_hat_i)^2
- A positive slope means predicted y increases as x increases.
- The best fit line minimizes the total squared residuals for the training data.
- Gradient descent update idea: new parameter = old parameter - learning rate × gradient
Vocabulary
- Linear regression
- A statistical and machine learning method that predicts a numerical output using a linear equation.
- Input feature
- A measured variable used by the model to make a prediction.
- Prediction
- The output value estimated by a model for a given input.
- Residual
- The difference between an observed value and the value predicted by the regression model.
- Cost function
- A formula that measures how far the model predictions are from the actual data values.
Common Mistakes to Avoid
- Confusing correlation with causation, because a strong linear trend does not prove that the input feature directly causes the output.
- Using the regression line far outside the data range, because extrapolation can give unreliable predictions when the trend changes beyond the observed values.
- Ignoring residual patterns, because curved or fan shaped residual plots show that a straight line may not be an appropriate model.
- Choosing a learning rate that is too large in gradient descent, because the algorithm can overshoot the minimum and fail to converge.
Practice Questions
- 1 A regression model is y_hat = 3x + 2. What prediction does it make when x = 7?
- 2 For three data points, the actual values are 5, 8, and 10, and the predicted values are 4, 9, and 7. Find the residuals and the mean squared error.
- 3 A scatter plot shows points that curve upward instead of forming a roughly straight cloud. Explain why a linear regression model may give poor predictions for this data.