When scientists test a hypothesis, they often use a significance level such as alpha = 0.05 to decide whether an observed pattern is unlikely under a null hypothesis. The multiple comparisons problem appears when many tests are run on the same dataset or in the same study. Even if every null hypothesis is true, some tests can look significant just by random chance.
This matters because false positives can lead researchers to report patterns that do not really exist.
Understanding Statistics: The Multiple Comparisons Problem
The problem grows because researchers usually have many choices. A health study may compare several treatments, measure several outcomes, and examine results in different age groups. A psychology experiment may record reaction time, memory score, mood, and sleep.
Each comparison creates another opportunity for random variation to resemble a real effect. Looking at subgroups after seeing the data adds even more tests.
Trying several statistical methods or stopping data collection when a result crosses a cutoff can create a similar risk. The important point is that the full set of choices matters, not only the one result reported at the end.
A group of related tests is often called a family. Defining that family requires scientific judgment. It might mean every outcome in one experiment.
It might mean all pairwise comparisons among several groups. If a researcher measures ten outcomes but highlights only the smallest p-value, readers need to know that ten outcomes were examined. The tests are not always independent.
For example, height and weight are related, so tests involving them tend to move together. This can change the exact probability of false positives.
It does not remove the basic concern. More opportunities to search can still produce misleading findings.
One way to control the risk is to use a stricter cutoff for each test. The Bonferroni method divides the allowed family error rate among the planned tests. It is simple and works even when tests are related, but it can be very conservative.
A stricter cutoff lowers false positives while making real effects harder to detect. This is a tradeoff between Type I errors, which are false alarms, and Type II errors, which are missed effects. Other methods, such as the Holm procedure, adjust cutoffs in stages and can retain more power.
In studies with huge numbers of tests, researchers may control the false discovery rate instead. That approach limits the expected share of reported discoveries that are false, rather than demanding almost no false discovery in the whole family.
Students meet this issue whenever results are sorted, filtered, or compared. A class survey can produce apparent patterns between music preference, homework time, screen use, and test scores simply because many comparisons were tried. Sports statistics, medical research, online A B tests, and genetic studies face the same challenge.
Good practice begins before data are collected. Researchers can state which outcome is primary, plan the comparisons, choose a correction, and report every test they ran.
Findings discovered after exploring data are still useful, but they should be described as exploratory and checked with new data. Replication matters because a result that survives a new study is less likely to be a lucky pattern.
Key Facts
- A p-value is the probability of getting results at least as extreme as the observed results, assuming the null hypothesis is true.
- For one test with alpha = 0.05, the false positive rate is 5 percent if the null hypothesis is true.
- For m independent tests, the chance of at least one false positive is 1 - (1 - alpha)^m.
- Family-wise error rate is the probability of making at least one Type I error across a family of tests.
- Bonferroni correction uses alpha_adjusted = alpha_family / m for each individual test.
- If alpha_family = 0.05 and m = 20, then alpha_adjusted = 0.05 / 20 = 0.0025.
Vocabulary
- Multiple comparisons problem
- The problem that running many statistical tests increases the chance of finding at least one false significant result.
- Null hypothesis
- A default claim that there is no real effect, no difference, or no relationship in the population.
- False positive
- A result that is declared significant even though the null hypothesis is actually true.
- Family-wise error rate
- The probability of making one or more Type I errors across a group of related hypothesis tests.
- Bonferroni correction
- A method that controls the family-wise error rate by dividing the desired overall alpha level by the number of tests.
Common Mistakes to Avoid
- Treating each p-value as if it came from the only test performed, which ignores the increased chance of false positives across many tests.
- Using alpha = 0.05 for every comparison without adjustment, which can make the overall error rate much larger than 5 percent.
- Applying Bonferroni correction without defining the family of tests, which can make the correction either too weak or unnecessarily strict.
- Interpreting a corrected nonsignificant result as proof that there is no effect, which is wrong because the study may simply lack enough power to detect the effect.
Practice Questions
- 1 A researcher runs 10 independent tests with alpha = 0.05 for each test. What is the probability of at least one false positive if all null hypotheses are true? Use 1 - (1 - alpha)^m.
- 2 A study makes 25 comparisons and wants a family-wise error rate of 0.05. What alpha level should be used for each test under the Bonferroni correction?
- 3 A dataset produces 3 significant results after 60 different tests were tried, but the researcher reports only those 3 tests. Explain why this can be misleading and how a correction changes the interpretation.