Logistic regression is a statistical method used when the outcome has two categories, such as yes or no, pass or fail, or disease present or absent. Instead of predicting any possible number, it predicts a probability between 0 and 1. This matters because many real decisions are based on estimated chances, not exact quantities.
The S-shaped logistic curve shows how a change in a predictor variable can move an outcome from unlikely to likely.
Understanding Statistics: Logistic Regression
A logistic model begins with examples for which the outcome is already known. Each example has one or more predictor values, such as hours studied, attendance, age, or a test result. The model searches for coefficient values that make the observed outcomes most plausible.
This fitting process gives greater credit to a model that assigns high chances to events that happened and low chances to events that did not happen. It does not try to make every prediction exactly right.
Real data contain overlap. Some students with similar study time will still have different results for reasons the model cannot measure.
The model does its main calculation on odds rather than directly on probability. Odds compare the chance that an event occurs with the chance that it does not occur. A probability of one half has odds of one to one.
A probability of four fifths has odds of four to one. The coefficients describe changes in these odds. For one predictor, increasing it by one unit multiplies the odds by a fixed amount, provided other predictors stay fixed.
This is why reports often convert a coefficient into an odds ratio. An odds ratio above one means higher odds.
An odds ratio below one means lower odds. Odds are not the same as probability, so they must be interpreted carefully.
With several predictors, logistic regression can separate effects that would otherwise be mixed together. A study of exam outcomes might include study time, previous grades, attendance, and access to tutoring. The coefficient for study time then describes its relationship with the outcome after accounting for the other included variables.
This does not prove that extra study time causes a better result. Important missing factors, such as sleep, family support, or motivation, can distort the pattern.
Students should learn the difference between prediction and causation. A model can be useful for estimating risk even when it cannot explain the true cause of that risk.
A probability must often be turned into an action, and the cutoff depends on the situation. The halfway cutoff is common, but it is not automatically best. In medical screening, missing a real illness can be more harmful than sending a healthy person for another test.
A lower cutoff may be chosen to catch more possible cases. This usually creates more false alarms. In spam filtering, a higher cutoff can reduce the chance of hiding an important email, though more spam may remain visible.
Model quality should be checked with more than overall accuracy. Precision, recall, false positive rate, and false negative rate reveal different kinds of mistakes.
Calibration matters too. If cases given a probability near seven tenths occur about seven times out of ten over many similar cases, the model is well calibrated.
Key Facts
- Logistic regression predicts P(y = 1), the probability that an observation belongs to class 1.
- The logistic model is P(y = 1) = 1 / (1 + e^-(b0 + b1x)).
- The logit link is log(p / (1 - p)) = b0 + b1x.
- Odds are defined as odds = p / (1 - p).
- If b1 is positive, increasing x increases the predicted probability; if b1 is negative, increasing x decreases it.
- A common classification rule is predict y = 1 if p >= 0.5 and y = 0 if p < 0.5.
Vocabulary
- Logistic regression
- A model that predicts the probability of a binary outcome using one or more predictor variables.
- Probability
- A number from 0 to 1 that describes how likely an event is to occur.
- Odds
- The ratio of the probability that an event happens to the probability that it does not happen.
- Log-odds
- The natural logarithm of the odds, written as log(p / (1 - p)).
- Decision threshold
- A cutoff probability used to convert a predicted probability into a predicted class.
Common Mistakes to Avoid
- Treating logistic regression like ordinary linear regression is wrong because the output must stay between 0 and 1, while a linear model can predict values below 0 or above 1.
- Interpreting a coefficient as a direct change in probability is wrong because logistic regression coefficients change log-odds, not probability by a fixed amount.
- Using a 0.5 threshold without thinking is a mistake because the best threshold depends on the cost of false positives and false negatives.
- Assuming correlation means causation is wrong because logistic regression can show association, but it does not prove that a predictor causes the outcome.
Practice Questions
- 1 For the model p = 1 / (1 + e^-(2 + 0.8x)), find the predicted probability when x = 0. Round to three decimal places.
- 2 A logistic regression model gives log(p / (1 - p)) = -1.2 + 0.6x. Find the log-odds and odds when x = 3.
- 3 Explain why a logistic curve is more appropriate than a straight line for predicting whether a student passes or fails based on hours studied.