Algorithms often make predictions by learning patterns from past data, but those patterns can include human bias, measurement bias, or unequal representation. When biased training data enters a model, the model can produce biased predictions even if the code seems neutral. This matters because algorithms are used in hiring, lending, health care, policing, school admissions, and online recommendations.
Statistics helps us detect, measure, and reduce unfair patterns before they cause real harm.
Algorithmic bias can enter at many stages, including data collection, labeling, feature selection, model training, and deployment. A model may appear accurate overall while performing poorly for a smaller group, so group-level evaluation is essential. Fairer models often require better data, careful choice of target variables, bias audits, and tradeoff analysis between accuracy and fairness.
The goal is not to make a model perfect, but to make its assumptions, errors, and impacts visible and testable.
Key Facts
- Training data bias occurs when the sample used to train a model does not represent the population where the model will be used.
- Prediction error = actual value - predicted value, and error rates should be compared across groups.
- Accuracy = correct predictions / total predictions, but high overall accuracy can hide poor performance for a subgroup.
- False positive rate = false positives / actual negatives, and unequal false positive rates can signal unfair treatment.
- False negative rate = false negatives / actual positives, and unequal false negative rates can deny opportunities or services.
- A common fairness goal is demographic parity, where P(predicted positive | group A) = P(predicted positive | group B), though it may conflict with other goals.
Vocabulary
- Algorithmic bias
- Algorithmic bias is systematic unfairness in a model's outputs that disadvantages or favors certain people or groups.
- Training data
- Training data is the set of examples a model uses to learn patterns and make future predictions.
- Feature
- A feature is an input variable, such as age, income, test score, or location, that a model uses to make a prediction.
- Proxy variable
- A proxy variable is a feature that indirectly represents a sensitive trait, such as zip code acting as a proxy for race or income.
- Fairness metric
- A fairness metric is a numerical measure used to compare model outcomes or error rates across different groups.
Common Mistakes to Avoid
- Assuming removing sensitive variables eliminates bias. This is wrong because other features, such as school, zip code, or browsing history, can act as proxy variables.
- Checking only overall accuracy. This is wrong because a model can be accurate for a large majority group while making many errors for a smaller group.
- Treating historical data as objective truth. This is wrong because past decisions may reflect unequal access, discrimination, or biased measurement.
- Using one fairness metric as the final answer. This is wrong because fairness metrics can conflict, so the correct choice depends on the context, harms, and goals of the system.
Practice Questions
- 1 A hiring model is tested on 1,000 applicants and makes 820 correct predictions. What is its accuracy as a percentage?
- 2 For Group A, a loan model produces 30 false positives out of 300 actual negatives. For Group B, it produces 45 false positives out of 150 actual negatives. Compute the false positive rate for each group and identify which group has the higher rate.
- 3 A school admissions model does not use family income as a feature, but it uses neighborhood, high school attended, and number of unpaid internships. Explain how bias could still enter the model and name one way to test for it.