Computer Science: AI Bias and Fairness Case Studies
Analyzing real-world impacts of bias, fairness, and accountability in AI systems
Computer Science: AI Bias and Fairness Case Studies
Analyzing real-world impacts of bias, fairness, and accountability in AI systems
Computer Science - Grade 9-12
- 1
A school uses an AI tool to flag students who may need academic support. The tool was trained mostly on data from large suburban schools, but it is now used in rural, urban, and online schools. Identify two possible fairness concerns and explain why they matter.
Think about whether the training data represents all student groups who will be affected.
One fairness concern is that the model may not work as accurately for students from schools that were not well represented in the training data. Another concern is that the model may confuse differences in school resources or attendance patterns with student ability. These concerns matter because students could be incorrectly denied support or unfairly labeled as at risk. - 2
A hiring algorithm recommends applicants for interviews. In a test set, 80 out of 100 qualified Group A applicants are recommended, while 60 out of 100 qualified Group B applicants are recommended. Calculate the true positive rate for each group and explain what the difference suggests.
True positive rate means the percent of actually qualified people who were correctly selected.
The true positive rate for Group A is 80 percent because 80 out of 100 qualified applicants were recommended. The true positive rate for Group B is 60 percent because 60 out of 100 qualified applicants were recommended. The difference suggests that the system is less likely to correctly recommend qualified applicants from Group B, which may be a fairness problem. - 3
A facial recognition system is tested on four groups. The error rates are: Group 1, 1 percent; Group 2, 3 percent; Group 3, 12 percent; Group 4, 18 percent. What fairness issue does this reveal, and what should developers investigate next?
This reveals unequal error rates across groups. The system makes far more mistakes for Groups 3 and 4 than for Groups 1 and 2. Developers should investigate whether the training data, lighting conditions, camera quality, labeling process, or model design caused these unequal results. - 4
A bank uses an AI model to approve loans. The model does not use race as an input, but it uses ZIP code, income, school attended, and employment history. Explain how bias could still enter the system.
A proxy variable is a variable that indirectly reveals information about another variable.
Bias could still enter the system because variables like ZIP code, school attended, and employment history can act as proxies for protected characteristics such as race or socioeconomic status. Even if race is not directly included, the model may learn patterns from historical inequality and repeat unfair decisions. - 5
A hospital AI predicts which patients should receive extra care coordination. The model uses past health-care spending as a measure of need. Explain why this choice could create bias.
Using past health-care spending as a measure of need can create bias because spending does not always equal medical need. Some groups may have received less care in the past because of access barriers, cost, discrimination, or lack of insurance. The model could underestimate their needs and continue the unfair pattern. - 6
In a content moderation system, posts written in Standard English are incorrectly removed 2 percent of the time, while posts written in a dialect are incorrectly removed 14 percent of the time. Identify the type of error being described and explain why it is harmful.
A false positive happens when the system flags something as a problem even though it is not a problem.
The error being described is a false positive because acceptable posts are incorrectly removed. It is harmful because users who write in the dialect may be unfairly silenced, punished, or discouraged from participating online. - 7
A city uses an AI system to decide where to send more police patrols. The training data comes from historical arrest records. Explain one reason this data may be biased and one possible consequence.
Historical arrest records may be biased because they reflect where police were sent in the past, not necessarily where all crime occurred. A possible consequence is a feedback loop where the AI sends more patrols to the same neighborhoods, leading to more arrests there and reinforcing the original pattern. - 8
A company tests two speech recognition models. Model X has an overall word error rate of 8 percent. Model Y has an overall word error rate of 10 percent. However, Model X has much higher errors for speakers with certain accents. Which model might be fairer, and what additional information is needed?
An overall average can hide large differences between groups.
Model Y might be fairer if its errors are more evenly distributed across accent groups, even though its overall error rate is slightly higher. Additional information is needed about the word error rate for each accent group, not just the overall average. - 9
An AI resume screener was trained on resumes from employees who were hired over the past 15 years. Most past hires were men. The system begins ranking resumes with women's colleges or women-focused organizations lower. Explain what likely happened.
The system likely learned biased patterns from historical hiring data. Because past hiring favored men, the model treated features associated with women as negative signals, even if those features were unrelated to job performance. This is an example of historical bias being reproduced by an AI system. - 10
A fairness audit compares approval rates for a scholarship AI. Group A has 50 approvals out of 200 applicants. Group B has 30 approvals out of 200 applicants. Calculate the approval rate for each group and state whether the results raise a fairness concern.
Approval rate equals approvals divided by total applicants.
Group A has an approval rate of 25 percent because 50 divided by 200 equals 0.25. Group B has an approval rate of 15 percent because 30 divided by 200 equals 0.15. The results raise a fairness concern because Group B is approved at a lower rate, although more investigation is needed to understand why. - 11
A developer says, "Our AI is fair because the code treats every user the same way." Explain why this claim is incomplete.
The claim is incomplete because fairness depends on more than whether the code follows the same steps for every user. The training data, labels, input features, evaluation metrics, and real-world context can still create unequal outcomes. Equal treatment by code can still lead to unfair impact. - 12
A photo-tagging AI labels pictures of adults in business clothing as "manager" more often for men than for women, even when the images are similar. Name one possible source of the bias and one way to reduce it.
Look for patterns in the labels that the model may have learned from examples.
One possible source of the bias is training data that showed men labeled as managers more often than women. One way to reduce it is to rebalance or relabel the training data so that job labels are not strongly tied to gender when the visual evidence does not support that connection. - 13
An AI system for college admissions uses extracurricular activities as one input. Explain how this could disadvantage some applicants, even if the model is accurate on past data.
This could disadvantage applicants who had fewer opportunities for extracurricular activities because of work, family responsibilities, transportation limits, cost, disability, or school resources. If the model learns from past admissions patterns without considering unequal access, it may reward opportunity rather than potential. - 14
A team wants to improve fairness in an AI model. List three actions they could take before releasing the model to the public.
Think about data, testing, human review, and accountability.
The team could test model performance separately for different demographic groups, review training data for underrepresentation or biased labels, and involve affected communities or domain experts in the design process. They could also document limitations and create an appeal process for people affected by the model. - 15
A judge is given a risk score from an AI tool that predicts whether a defendant might miss a court date. The tool's explanation only says "high risk" without showing important factors or confidence. Explain two reasons this lack of transparency is a problem.
The lack of transparency is a problem because the judge and defendant cannot understand which factors influenced the score or check whether the factors are fair and relevant. It is also hard to challenge mistakes, measure bias, or hold anyone accountable if the system gives only a label with no explanation.