When scientists test a hypothesis, they often use a significance level such as alpha = 0.05 to decide whether an observed pattern is unlikely under a null hypothesis. The multiple comparisons problem appears when many tests are run on the same dataset or in the same study. Even if every null hypothesis is true, some tests can look significant just by random chance.
This matters because false positives can lead researchers to report patterns that do not really exist.
Key Facts
- A p-value is the probability of getting results at least as extreme as the observed results, assuming the null hypothesis is true.
- For one test with alpha = 0.05, the false positive rate is 5 percent if the null hypothesis is true.
- For m independent tests, the chance of at least one false positive is 1 - (1 - alpha)^m.
- Family-wise error rate is the probability of making at least one Type I error across a family of tests.
- Bonferroni correction uses alpha_adjusted = alpha_family / m for each individual test.
- If alpha_family = 0.05 and m = 20, then alpha_adjusted = 0.05 / 20 = 0.0025.
Vocabulary
- Multiple comparisons problem
- The problem that running many statistical tests increases the chance of finding at least one false significant result.
- Null hypothesis
- A default claim that there is no real effect, no difference, or no relationship in the population.
- False positive
- A result that is declared significant even though the null hypothesis is actually true.
- Family-wise error rate
- The probability of making one or more Type I errors across a group of related hypothesis tests.
- Bonferroni correction
- A method that controls the family-wise error rate by dividing the desired overall alpha level by the number of tests.
Common Mistakes to Avoid
- Treating each p-value as if it came from the only test performed, which ignores the increased chance of false positives across many tests.
- Using alpha = 0.05 for every comparison without adjustment, which can make the overall error rate much larger than 5 percent.
- Applying Bonferroni correction without defining the family of tests, which can make the correction either too weak or unnecessarily strict.
- Interpreting a corrected nonsignificant result as proof that there is no effect, which is wrong because the study may simply lack enough power to detect the effect.
Practice Questions
- 1 A researcher runs 10 independent tests with alpha = 0.05 for each test. What is the probability of at least one false positive if all null hypotheses are true? Use 1 - (1 - alpha)^m.
- 2 A study makes 25 comparisons and wants a family-wise error rate of 0.05. What alpha level should be used for each test under the Bonferroni correction?
- 3 A dataset produces 3 significant results after 60 different tests were tried, but the researcher reports only those 3 tests. Explain why this can be misleading and how a correction changes the interpretation.