Logistic regression is a machine learning model used to make yes or no predictions from data. It is often used for classification, such as deciding whether an email is spam, whether a student may pass a test, or whether a tumor is likely benign or malignant. Even though its name includes regression, its output is a probability that gets turned into a class label.
It matters because many real world AI systems need clear decisions based on uncertain information.
The model starts by combining input features into a single score using weights and a bias. That score is passed through an S-shaped sigmoid function, which squeezes any number into a value between 0 and 1. The result can be interpreted as the probability of belonging to the positive class, such as probability of spam.
A decision threshold, often 0.5, then converts the probability into a binary decision.
Understanding AI & Machine Learning: Logistic Regression Explained
During training, the model learns from examples with known outcomes. For each example, it makes a probability prediction, then compares that prediction with the real label. A confident wrong prediction is treated as a larger mistake than an uncertain wrong prediction.
Cross entropy loss measures this kind of mistake. The learning process changes each weight by a small amount in the direction that reduces the total loss. This is usually done with gradient descent.
It repeats over many rounds until improvements become small. A positive weight means that larger feature values tend to raise the predicted chance of class 1.
A negative weight means that larger values tend to lower it. The bias gives the model a starting tendency before any feature information is included.
The weights can be useful for interpretation, but students need to read them carefully. Their size depends on the scale of each feature. For example, age measured in years and income measured in pounds can have very different number ranges.
A larger weight does not automatically mean a more important feature. Scaling numerical features can make training more stable and make weight comparisons fairer. Text labels must be converted into numerical features before training.
A category such as payment type can become separate zero or one columns. Missing values need attention too. A blank field is not always the same as a value of zero, so careless handling can create misleading patterns.
The threshold is a choice made for the situation, not a law of the model. In spam filtering, marking a safe email as spam may be annoying, while allowing dangerous mail through may be worse in another setting. A medical screening tool may use a lower threshold to catch more possible cases, then send people for further tests.
Lowering the threshold usually finds more positive cases, but it can create more false positives. Raising it usually reduces false positives, but it can miss real positives.
A confusion matrix helps show these tradeoffs by counting true positives, true negatives, false positives, and false negatives. Accuracy alone can hide problems when one class is much more common than the other.
Logistic regression draws a straight decision boundary in the feature space. This works well when the classes can be separated by a roughly linear pattern. It may struggle when outcomes depend on complex shapes or strong interactions.
For instance, a risk may rise only when both temperature and humidity are high. Extra interaction features can help represent such cases. The model can still produce a probability for a new example, but that number is trustworthy only if the training data is relevant and representative.
Biased historical data can lead to biased predictions. Students should check where the data came from, test the model on unseen examples, and remember that a probability supports a decision rather than proving that decision is correct.
Key Facts
- Logistic regression is mainly used for binary classification, such as class 0 versus class 1.
- Linear score: z = w1x1 + w2x2 + ... + b
- Sigmoid function: p = 1 / (1 + e^(-z))
- The output p is a probability between 0 and 1.
- Decision rule with threshold 0.5: if p >= 0.5, predict class 1; if p < 0.5, predict class 0.
- Training adjusts the weights and bias to reduce prediction error, often using a loss function called cross entropy.
Vocabulary
- Feature
- A feature is an input variable the model uses to make a prediction, such as hours studied or number of words in an email.
- Weight
- A weight is a number that shows how strongly a feature affects the model's prediction.
- Bias
- A bias is a constant added to the model's score so the decision boundary can shift to better fit the data.
- Sigmoid Function
- The sigmoid function converts any input score into a probability between 0 and 1.
- Threshold
- A threshold is the cutoff value used to turn a probability into a predicted class.
Common Mistakes to Avoid
- Calling logistic regression a model for predicting any number. It is usually used for classification because its sigmoid output is a probability between 0 and 1.
- Treating the raw score z as the final probability. The score must pass through the sigmoid function before it can be interpreted as a probability.
- Assuming 0.5 is always the best threshold. Different problems may need a higher or lower threshold depending on the cost of false positives and false negatives.
- Ignoring feature scaling when features have very different sizes. Large scale features can make training harder and can cause some weights to dominate unfairly.
Practice Questions
- 1 A logistic regression model uses z = 2x - 3. If x = 1, calculate z, then calculate p = 1 / (1 + e^(-z)) to two decimal places, and decide the class using a threshold of 0.5.
- 2 A model predicts p = 0.82 for an email being spam. Using a threshold of 0.5, what class is predicted? If the threshold is changed to 0.9, what class is predicted?
- 3 A medical test model has many false negatives when detecting a serious disease. Explain whether the decision threshold should likely be raised or lowered, and justify your reasoning.