Logistic regression is a statistical method used when the outcome has two categories, such as yes or no, pass or fail, or disease present or absent. Instead of predicting any possible number, it predicts a probability between 0 and 1. This matters because many real decisions are based on estimated chances, not exact quantities.
The S-shaped logistic curve shows how a change in a predictor variable can move an outcome from unlikely to likely.
Key Facts
- Logistic regression predicts P(y = 1), the probability that an observation belongs to class 1.
- The logistic model is P(y = 1) = 1 / (1 + e^-(b0 + b1x)).
- The logit link is log(p / (1 - p)) = b0 + b1x.
- Odds are defined as odds = p / (1 - p).
- If b1 is positive, increasing x increases the predicted probability; if b1 is negative, increasing x decreases it.
- A common classification rule is predict y = 1 if p >= 0.5 and y = 0 if p < 0.5.
Vocabulary
- Logistic regression
- A model that predicts the probability of a binary outcome using one or more predictor variables.
- Probability
- A number from 0 to 1 that describes how likely an event is to occur.
- Odds
- The ratio of the probability that an event happens to the probability that it does not happen.
- Log-odds
- The natural logarithm of the odds, written as log(p / (1 - p)).
- Decision threshold
- A cutoff probability used to convert a predicted probability into a predicted class.
Common Mistakes to Avoid
- Treating logistic regression like ordinary linear regression is wrong because the output must stay between 0 and 1, while a linear model can predict values below 0 or above 1.
- Interpreting a coefficient as a direct change in probability is wrong because logistic regression coefficients change log-odds, not probability by a fixed amount.
- Using a 0.5 threshold without thinking is a mistake because the best threshold depends on the cost of false positives and false negatives.
- Assuming correlation means causation is wrong because logistic regression can show association, but it does not prove that a predictor causes the outcome.
Practice Questions
- 1 For the model p = 1 / (1 + e^-(2 + 0.8x)), find the predicted probability when x = 0. Round to three decimal places.
- 2 A logistic regression model gives log(p / (1 - p)) = -1.2 + 0.6x. Find the log-odds and odds when x = 3.
- 3 Explain why a logistic curve is more appropriate than a straight line for predicting whether a student passes or fails based on hours studied.