Sampling bias happens when the people, objects, or events chosen for a study do not represent the full population of interest. It matters because even a large sample can give a wrong answer if it is collected in a biased way. Good statistical conclusions depend more on how the sample is chosen than on sample size alone.
Recognizing common bias types helps researchers design fairer surveys, experiments, and observational studies.
Bias can enter through the sampling frame, the recruitment method, who chooses to respond, or which cases remain visible in the data. Selection bias, undercoverage, nonresponse bias, voluntary response bias, and survivorship bias each distort a sample in a different way. These problems affect estimates such as proportions, means, and comparisons between groups.
Careful random sampling, follow-up with nonresponders, and clear population definitions reduce the risk of misleading results.
Key Facts
- Sampling bias occurs when a sample is systematically different from the population it is meant to represent.
- A random sample gives every member of the population a known chance of selection.
- Sample proportion: p-hat = x/n, where x is the number with the trait and n is the sample size.
- Bias of an estimator: bias = E(estimator) - true value.
- Increasing n reduces random sampling error, but it does not automatically remove sampling bias.
- Common sampling bias types include selection bias, undercoverage bias, nonresponse bias, voluntary response bias, and survivorship bias.
Vocabulary
- Population
- The entire group of people, objects, or events that a study wants to learn about.
- Sample
- A smaller group selected from the population to collect data from.
- Sampling frame
- The list or method used to identify members of the population who can be selected for the sample.
- Undercoverage bias
- A bias that occurs when some groups in the population are left out or are less likely to be included in the sample.
- Nonresponse bias
- A bias that occurs when people who do not respond differ in important ways from people who do respond.
Common Mistakes to Avoid
- Assuming a large sample is automatically unbiased. A large biased sample can still give a very precise but wrong estimate if the same groups are overrepresented or excluded.
- Confusing voluntary response with random sampling. People who choose to respond often have stronger opinions or different experiences than the overall population.
- Ignoring the sampling frame. If the frame leaves out people without phones, internet access, addresses, or membership in a list, the resulting sample can suffer from undercoverage.
- Treating nonresponse as harmless missing data. If nonresponders differ from responders on the topic being studied, estimates such as means and proportions can be distorted.
Practice Questions
- 1 A school has 1200 students, but a survey about cafeteria food is emailed only to the 800 students who joined the school app. If 160 students respond and 112 say they like the food, what is the sample proportion p-hat, and what type of sampling bias might occur?
- 2 A city survey calls 1000 randomly selected phone numbers about public transportation. Only 420 people answer, and 252 of them support increasing bus service. What percent of respondents support the plan, and what bias could occur if nonresponders have different commuting habits?
- 3 A magazine asks readers to vote online on whether homework should be banned. Explain why the result may not represent all students, even if 50,000 people respond.