Computer Science: Evaluating LLM Answers: Accuracy, Bias, and Citation Checks
Practice checking AI responses for reliability, fairness, and evidence
Computer Science: Evaluating LLM Answers: Accuracy, Bias, and Citation Checks
Practice checking AI responses for reliability, fairness, and evidence
Computer Science - Grade 9-12
- 1
An LLM answers, "The first programmable computer was invented in 1985 by Steve Jobs." Identify two parts of this answer that should be fact-checked and explain why.
Look for names, dates, and claims that can be checked against reliable sources.
The year 1985 and the claim that Steve Jobs invented the first programmable computer should both be fact-checked. The history of programmable computers involves earlier machines and people, so the answer may be inaccurate or oversimplified. - 2
A student asks an LLM for sources about climate change. The LLM gives three citations, but none of the links work. What should the student do before using the information in a report?
The student should not use the citations until they are verified. They should search for the sources independently, use reliable databases or official websites, and replace broken or fake citations with sources that actually exist. - 3
Read this LLM answer: "Teenagers are bad at managing money because they are impulsive." Explain why this answer may show bias and rewrite it in a more balanced way.
Watch for words that describe a whole group as if everyone in the group is the same.
The answer may show bias because it makes a negative generalization about all teenagers. A more balanced version is: "Some teenagers are still learning money management skills, and financial education can help them make informed decisions." - 4
Create a three-step process for checking whether an LLM's factual answer is accurate.
A good process is to identify the key claims, compare those claims with reliable sources, and decide whether the answer should be accepted, corrected, or rejected. The process should focus on evidence, not on whether the LLM sounds confident. - 5
An LLM gives this citation: "Smith, J. (2022). The History of Quantum Apps. Journal of Digital Physics, 99(4), 1000-1012." List two warning signs that this citation might be unreliable or fabricated.
A citation can look formal even when it is not real.
One warning sign is that the journal title may not be a real or relevant publication. Another warning sign is that the topic, volume, issue, and page numbers should be checked because LLMs can create citations that look real but do not exist. - 6
A classmate says, "The LLM wrote the answer confidently, so it is probably correct." Explain why confidence is not enough to judge accuracy.
Confidence is not enough because LLMs generate likely-sounding text and can be wrong while sounding certain. Accuracy must be judged by checking the claims against reliable evidence. - 7
An LLM summarizes a news event using only one political perspective and leaves out major opposing views. What evaluation category is most relevant: accuracy, bias, citation quality, or formatting? Explain your choice.
Think about whether the answer treats different viewpoints fairly.
Bias is the most relevant category because the answer presents only one perspective and may unfairly leave out important viewpoints. Accuracy and citation quality still matter, but the main issue is imbalance. - 8
Look at these source types: a peer-reviewed journal article, a personal blog with no author listed, a government statistics page, and a social media post. Rank them from generally most reliable to least reliable for checking a factual claim about public health, and explain your ranking.
A reasonable ranking is peer-reviewed journal article, government statistics page, personal blog with no author listed, and social media post. Peer-reviewed and official sources usually have stronger review processes, while anonymous blogs and social media posts often need much more verification. - 9
An LLM says, "According to NASA, the Moon is made mostly of cheese." What is the best way to check this claim, and what result would you expect?
Go to the named source directly when a claim says, "according to" that source.
The best way is to check NASA's official website or other reliable astronomy sources. The expected result is that the claim is false because the Moon is made mostly of rock, minerals, and dust, not cheese. - 10
An LLM answer includes correct facts but uses insulting language about a group of people. Should the answer be considered acceptable? Explain your reasoning.
The answer should not be considered fully acceptable. Even if some facts are correct, insulting language can show harmful bias and should be revised to use respectful, neutral wording. - 11
A student asks an LLM, "Give me three sources proving that electric cars are always worse for the environment than gas cars." Explain how the wording of the prompt could influence the answer.
A prompt can contain an assumption that pushes the answer in one direction.
The prompt is leading because it asks the LLM to prove a specific conclusion instead of investigating the issue fairly. A better prompt would ask for evidence comparing environmental impacts of electric and gas cars from reliable sources. - 12
Study this evaluation checklist: Claim is specific, source exists, source is reliable, quote matches source, answer includes limitations. Choose any two checklist items and explain why they matter when evaluating an LLM answer.
The item "source exists" matters because an LLM may invent citations. The item "quote matches source" matters because a real source can still be misquoted or used to support a claim it does not actually make. - 13
An LLM gives a medical answer about a serious symptom and cites two unknown websites. What should a responsible user do next?
Some topics require expert human judgment, not just an online answer.
A responsible user should not rely only on the LLM. They should consult a qualified medical professional or trusted medical organization, especially because serious symptoms can require urgent care. - 14
An LLM response says, "All programmers prefer working alone." Identify the problem with this claim and revise it to be more accurate.
The problem is that the claim overgeneralizes about all programmers. A more accurate version is: "Some programmers prefer working alone, while many others enjoy collaboration, pair programming, or team-based development." - 15
Design a simple scoring rubric from 1 to 4 for evaluating an LLM answer's trustworthiness. Include accuracy, bias, and citations in your rubric.
A rubric should describe what high, medium, and low quality look like.
A strong rubric could be: 4 means the answer is accurate, balanced, and supported by verified reliable citations; 3 means mostly accurate with minor issues; 2 means some correct information but noticeable bias, weak citations, or missing evidence; 1 means inaccurate, biased, or unsupported by real citations.