Logistic regression is a machine learning model used to make yes or no predictions from data. It is often used for classification, such as deciding whether an email is spam, whether a student may pass a test, or whether a tumor is likely benign or malignant. Even though its name includes regression, its output is a probability that gets turned into a class label.
It matters because many real world AI systems need clear decisions based on uncertain information.
The model starts by combining input features into a single score using weights and a bias. That score is passed through an S-shaped sigmoid function, which squeezes any number into a value between 0 and 1. The result can be interpreted as the probability of belonging to the positive class, such as probability of spam.
A decision threshold, often 0.5, then converts the probability into a binary decision.
Key Facts
- Logistic regression is mainly used for binary classification, such as class 0 versus class 1.
- Linear score: z = w1x1 + w2x2 + ... + b
- Sigmoid function: p = 1 / (1 + e^(-z))
- The output p is a probability between 0 and 1.
- Decision rule with threshold 0.5: if p >= 0.5, predict class 1; if p < 0.5, predict class 0.
- Training adjusts the weights and bias to reduce prediction error, often using a loss function called cross entropy.
Vocabulary
- Feature
- A feature is an input variable the model uses to make a prediction, such as hours studied or number of words in an email.
- Weight
- A weight is a number that shows how strongly a feature affects the model's prediction.
- Bias
- A bias is a constant added to the model's score so the decision boundary can shift to better fit the data.
- Sigmoid Function
- The sigmoid function converts any input score into a probability between 0 and 1.
- Threshold
- A threshold is the cutoff value used to turn a probability into a predicted class.
Common Mistakes to Avoid
- Calling logistic regression a model for predicting any number. It is usually used for classification because its sigmoid output is a probability between 0 and 1.
- Treating the raw score z as the final probability. The score must pass through the sigmoid function before it can be interpreted as a probability.
- Assuming 0.5 is always the best threshold. Different problems may need a higher or lower threshold depending on the cost of false positives and false negatives.
- Ignoring feature scaling when features have very different sizes. Large scale features can make training harder and can cause some weights to dominate unfairly.
Practice Questions
- 1 A logistic regression model uses z = 2x - 3. If x = 1, calculate z, then calculate p = 1 / (1 + e^(-z)) to two decimal places, and decide the class using a threshold of 0.5.
- 2 A model predicts p = 0.82 for an email being spam. Using a threshold of 0.5, what class is predicted? If the threshold is changed to 0.9, what class is predicted?
- 3 A medical test model has many false negatives when detecting a serious disease. Explain whether the decision threshold should likely be raised or lowered, and justify your reasoning.