Bias in surveys and samples happens when the data collected does not fairly represent the population you want to study. This matters because even a large amount of data can lead to wrong conclusions if the sample is distorted. Biased data can affect science, business decisions, public policy, and everyday claims reported in the media.
Learning to spot bias helps students judge whether a result is trustworthy.
Bias can enter at several stages of a study, including who gets selected, who chooses to respond, how questions are worded, and how data is recorded. A sample should give every relevant group a fair chance to be included, or at least account for differences carefully. If one group is overrepresented or underrepresented, the sample mean, proportion, or trend may not match the true population value.
Good survey design uses random sampling, clear questions, and careful follow-up to reduce these problems.
Understanding Bias in Surveys and Samples
A survey has two kinds of uncertainty. Random sampling variation occurs because a sample is only one small part of a population. It can make a result a little high or low by chance.
Bias is more serious because it tends to pull results in one direction repeatedly. Increasing the number of people in a biased sample usually makes the wrong estimate more precise, not more accurate. Imagine estimating the average height of all students by measuring only basketball players.
Measuring one thousand players does not fix the missing shorter students. Students should learn to separate sample size from sample quality. A large sample is useful only when the way it was collected is sound.
Undercoverage begins before anyone answers a question. It occurs when the list used to choose people leaves out part of the population. An online survey about local transport may miss residents without reliable internet access.
A school survey sent only through a student club may miss students who do not join clubs. Some missing groups may have different experiences, which changes the final result. Random selection from an incomplete list cannot repair this problem.
Researchers first need a good sampling frame, meaning a list or method that reaches the population of interest. They may divide the population into groups such as year level, area, or age, then randomly select from each group.
This is called stratified sampling. It helps make sure small but important groups are not lost.
Response bias comes from the answers people give, even when the selected sample is fair. People may forget details, misunderstand a question, feel embarrassed, or give an answer that seems socially acceptable. Questions about sleep, study time, exercise, spending, or rule breaking can be affected by this pressure.
Privacy can reduce the problem. Anonymous forms often produce more honest answers than a teacher asking students aloud. The setting matters too.
A survey taken just after a difficult exam may produce different ratings of school stress than one taken during a quiet week. Nonresponse creates a related issue. If busy students ignore a survey about homework while students with more free time answer it, the responses may not describe the whole school.
Wording can quietly guide people toward an answer. A loaded word such as wasteful, dangerous, or unfair can create an emotional reaction before a person has considered the issue. A question that asks whether a student supports improving lunches and reducing prices combines two separate ideas.
Someone may support one idea but not the other. This is called a double barreled question. Good questions use neutral language, ask one thing at a time, and provide response choices that fit the situation.
The order of questions can matter as well. Asking about a bad experience first may influence a later overall rating.
When reading survey results, check who was invited, who actually responded, how the questions were phrased, and when the survey occurred. These details often matter more than a headline percentage.
Key Facts
- Bias is a systematic error that pushes results away from the true population value.
- A sample is representative when its important characteristics are similar to those of the population.
- Selection bias happens when some members of the population are more likely to be chosen than others.
- Voluntary response bias happens when people choose themselves to participate, often attracting strong opinions.
- Nonresponse bias happens when selected people do not respond and the responders differ from nonresponders.
- Sample proportion formula: , where is the number with the trait and is the sample size.
Vocabulary
- Population
- The full group of people or objects that a study wants to describe.
- Sample
- A smaller group taken from the population and actually measured or surveyed.
- Bias
- A consistent error in data collection or analysis that makes results systematically inaccurate.
- Random sample
- A sample chosen by chance so that members of the population have a fair opportunity to be selected.
- Nonresponse bias
- Bias caused when the people who do not answer differ in important ways from the people who do answer.
Common Mistakes to Avoid
- Assuming a large sample is automatically unbiased, because size alone cannot fix a sample that was collected in a flawed way. A huge biased sample can still give a very wrong answer.
- Using a voluntary online poll to represent the whole population, because people with strong opinions are more likely to respond. This makes the sample different from the target group.
- Ignoring who was left out of the sampling process, because excluded groups can shift the results. A sample cannot represent people who had no real chance to be selected.
- Writing leading or confusing survey questions, because wording can push respondents toward certain answers. This changes the measured response instead of revealing true opinions.
Practice Questions
- 1 A school has 1200 students, but a survey about lunch quality is given only to students in the cafeteria during first lunch. Explain whether this sample is likely biased and identify the type of bias.
- 2 In a town survey, 250 people are contacted and 150 respond. Of those who respond, 96 support a new park. Calculate p_hat for support among respondents.
- 3 A researcher wants to estimate average weekly study time for all college students but surveys only students in the library on Sunday night. Explain why the estimate may be biased and whether it is likely too high or too low.