Algorithms often make predictions by learning patterns from past data, but those patterns can include human bias, measurement bias, or unequal representation. When biased training data enters a model, the model can produce biased predictions even if the code seems neutral. This matters because algorithms are used in hiring, lending, health care, policing, school admissions, and online recommendations.
Statistics helps us detect, measure, and reduce unfair patterns before they cause real harm.
Algorithmic bias can enter at many stages, including data collection, labeling, feature selection, model training, and deployment. A model may appear accurate overall while performing poorly for a smaller group, so group-level evaluation is essential. Fairer models often require better data, careful choice of target variables, bias audits, and tradeoff analysis between accuracy and fairness.
The goal is not to make a model perfect, but to make its assumptions, errors, and impacts visible and testable.
Understanding Statistics: Bias in Algorithms
A model does not discover fairness on its own. It learns from examples that humans created, recorded, or judged. Consider a system trained to predict who should receive a job interview.
If past hiring decisions favored applicants from certain schools or neighborhoods, those decisions become labels in the data. The model may learn that these background details predict hiring, even when they say little about a person's ability.
This is especially difficult because a target such as successful employee may be measured using promotion records, manager ratings, or length of employment. Each measure can reflect earlier unequal treatment rather than true job performance.
Some inputs act as proxies for protected traits. A proxy is a variable that seems unrelated but carries similar information. Postal code can reflect race, wealth, access to transport, or local school funding.
A gap in employment history may reflect illness, caring responsibilities, or unequal access to work. Removing an obvious field such as gender does not guarantee a neutral model because other fields may recreate much of the same pattern.
Students should notice the difference between a variable that is useful for prediction and a variable that is justified for a decision. A strong statistical relationship does not prove that using it is fair or appropriate.
Testing needs more than one score for the whole system. A confusion table separates correct decisions from different kinds of mistakes. For a loan model, a false positive could mean approving a loan that is not repaid.
A false negative could mean rejecting someone who would have repaid. Both errors matter, but their costs fall on different people. Results should be broken down by relevant groups and checked with enough data in each group.
Small groups produce uncertain estimates, so a single percentage can be misleading. Confidence intervals, repeated testing, and review of individual cases help show whether an apparent gap is stable or caused by random variation.
Fairness goals can conflict because groups may face different starting conditions in the data. Making approval rates equal across groups can change error rates. Making one error rate equal can change another.
There is no purely mathematical setting that settles every value judgment. People must decide which harms matter most, who has power to appeal, and whether the system should make that decision at all. Good practice includes documenting the purpose of the model, checking data before training, monitoring outcomes after release, and giving people a way to challenge harmful decisions.
Algorithms can change the world they measure. If a policing system sends more patrols to one area, it may record more incidents there, then use those records to justify sending even more patrols. This feedback loop can make an old pattern look like new evidence.
Key Facts
- Training data bias occurs when the sample used to train a model does not represent the population where the model will be used.
- Prediction error = actual value - predicted value, and error rates should be compared across groups.
- Accuracy = correct predictions / total predictions, but high overall accuracy can hide poor performance for a subgroup.
- False positive rate = false positives / actual negatives, and unequal false positive rates can signal unfair treatment.
- False negative rate = false negatives / actual positives, and unequal false negative rates can deny opportunities or services.
- A common fairness goal is demographic parity, where P(predicted positive | group A) = P(predicted positive | group B), though it may conflict with other goals.
Vocabulary
- Algorithmic bias
- Algorithmic bias is systematic unfairness in a model's outputs that disadvantages or favors certain people or groups.
- Training data
- Training data is the set of examples a model uses to learn patterns and make future predictions.
- Feature
- A feature is an input variable, such as age, income, test score, or location, that a model uses to make a prediction.
- Proxy variable
- A proxy variable is a feature that indirectly represents a sensitive trait, such as zip code acting as a proxy for race or income.
- Fairness metric
- A fairness metric is a numerical measure used to compare model outcomes or error rates across different groups.
Common Mistakes to Avoid
- Assuming removing sensitive variables eliminates bias. This is wrong because other features, such as school, zip code, or browsing history, can act as proxy variables.
- Checking only overall accuracy. This is wrong because a model can be accurate for a large majority group while making many errors for a smaller group.
- Treating historical data as objective truth. This is wrong because past decisions may reflect unequal access, discrimination, or biased measurement.
- Using one fairness metric as the final answer. This is wrong because fairness metrics can conflict, so the correct choice depends on the context, harms, and goals of the system.
Practice Questions
- 1 A hiring model is tested on 1,000 applicants and makes 820 correct predictions. What is its accuracy as a percentage?
- 2 For Group A, a loan model produces 30 false positives out of 300 actual negatives. For Group B, it produces 45 false positives out of 150 actual negatives. Compute the false positive rate for each group and identify which group has the higher rate.
- 3 A school admissions model does not use family income as a feature, but it uses neighborhood, high school attended, and number of unpaid internships. Explain how bias could still enter the model and name one way to test for it.