Click image to open full size
Bias in Surveys and Samples
Selection, Response & Wording Bias
Related Tools
Related Labs
Related Worksheets
Related Cheat Sheets
Bias in surveys and samples happens when the data collected does not fairly represent the population you want to study. This matters because even a large amount of data can lead to wrong conclusions if the sample is distorted. Biased data can affect science, business decisions, public policy, and everyday claims reported in the media. Learning to spot bias helps students judge whether a result is trustworthy.
Bias can enter at several stages of a study, including who gets selected, who chooses to respond, how questions are worded, and how data is recorded. A sample should give every relevant group a fair chance to be included, or at least account for differences carefully. If one group is overrepresented or underrepresented, the sample mean, proportion, or trend may not match the true population value. Good survey design uses random sampling, clear questions, and careful follow-up to reduce these problems.
Key Facts
- Bias is a systematic error that pushes results away from the true population value.
- A sample is representative when its important characteristics are similar to those of the population.
- Selection bias happens when some members of the population are more likely to be chosen than others.
- Voluntary response bias happens when people choose themselves to participate, often attracting strong opinions.
- Nonresponse bias happens when selected people do not respond and the responders differ from nonresponders.
- Sample proportion formula: , where is the number with the trait and is the sample size.
Vocabulary
- Population
- The full group of people or objects that a study wants to describe.
- Sample
- A smaller group taken from the population and actually measured or surveyed.
- Bias
- A consistent error in data collection or analysis that makes results systematically inaccurate.
- Random sample
- A sample chosen by chance so that members of the population have a fair opportunity to be selected.
- Nonresponse bias
- Bias caused when the people who do not answer differ in important ways from the people who do answer.
Common Mistakes to Avoid
- Assuming a large sample is automatically unbiased, because size alone cannot fix a sample that was collected in a flawed way. A huge biased sample can still give a very wrong answer.
- Using a voluntary online poll to represent the whole population, because people with strong opinions are more likely to respond. This makes the sample different from the target group.
- Ignoring who was left out of the sampling process, because excluded groups can shift the results. A sample cannot represent people who had no real chance to be selected.
- Writing leading or confusing survey questions, because wording can push respondents toward certain answers. This changes the measured response instead of revealing true opinions.
Practice Questions
- 1 A school has 1200 students, but a survey about lunch quality is given only to students in the cafeteria during first lunch. Explain whether this sample is likely biased and identify the type of bias.
- 2 In a town survey, 250 people are contacted and 150 respond. Of those who respond, 96 support a new park. Calculate p_hat for support among respondents.
- 3 A researcher wants to estimate average weekly study time for all college students but surveys only students in the library on Sunday night. Explain why the estimate may be biased and whether it is likely too high or too low.