P-Hacking & Multiple Comparisons Lab
Test one hypothesis at the five percent level and a false positive is rare. Test a hundred at once and several false positives are almost guaranteed. Generate a batch of tests, see the false discoveries appear, and watch each correction method tame them.
Guided Experiment: How many false positives appear when you test 100 nulls, and how does each correction change that?
Predict how many of 100 pure null tests will come out significant at alpha 0.05 with no correction, then predict what Bonferroni and Benjamini-Hochberg will do to that count.
Write your hypothesis in the Lab Report panel, then click Next.
Controls
Each new study draws a fresh batch of p-values from the same setup, so the false positives land on different tests every time, just like rerunning a real experiment from scratch.
Test grid
One cell per test, colored by its outcome under the current correction. Switch the method to watch the false positives appear and vanish.
Results under No correction
Discoveries (rejected)
11
out of 100 tests
True positives
8
real effects found
False positives
3
nulls flagged by chance
Missed real effects
2
false negatives
False discovery proportion
27.3%
false among discoveries
Power
80%
of real effects found
Why a correction is needed
With 100 tests and no correction, the chance of at least one false positive among the 90 null tests is about 99%. That is 1 minus (1 minus 0.05) to the power 100. The more tests you run, the closer this climbs to certainty.
This study ran 100 independent tests, 10 of which had a real effect. Under No correction you made 11 discoveries: 8 real and 3 false.
Sorted p-values and thresholds
Each dot is a test, sorted smallest p-value first. Filled dots are rejected under the current method. Compare the three thresholds directly.
Data Table
(0 rows)| # | Tests (m) | Real effects | Effect strength | Alpha | Correction | Discoveries | False positives | FDR (%) | Power (%) |
|---|
Reference Guide
The Multiple-Comparisons Problem
A single hypothesis test at alpha 0.05 has a five percent chance of a false positive when the null is true. Run many tests and those small chances add up fast.
- Each null test flagged as significant is a false positive.
- With 100 nulls at alpha 0.05 you expect about 5 false positives.
- Reporting only the significant ones makes noise look like signal.
The fix is not to test fewer ideas. It is to adjust the threshold for how many tests you ran.
The Family-Wise Error Rate
The family-wise error rate is the chance of at least one false positive across the whole set of tests. For m independent nulls with no correction it grows quickly.
FWER = 1 − (1 − alpha) to the power m
At alpha 0.05, m = 20 gives about 64 percent.
At m = 100 it is above 99 percent.
Raise the number of tests in the lab and watch the amber banner climb toward certainty.
Bonferroni and the FWER
The Bonferroni correction divides alpha by the number of tests. A test only counts as significant if its p-value clears that stricter bar.
Reject if p < alpha / m
This caps the chance of any false positive at alpha.
Bonferroni controls the family-wise error rate but is conservative. It can miss real effects, which lowers statistical power.
Benjamini-Hochberg and the FDR
The false discovery rate is the expected fraction of your discoveries that are false. Benjamini-Hochberg controls it instead of the FWER.
- Sort the p-values from smallest to largest.
- Find the largest rank k with p(k) ≤ (k / m) × alpha.
- Reject the k smallest p-values.
BH keeps far more power than Bonferroni while still limiting false discoveries, which makes it the standard in fields that run thousands of tests.
P-Hacking and the Replication Crisis
P-hacking means running many analyses and reporting only the ones that cross alpha, without telling anyone how many comparisons were tried. The published result looks impressive, but it is often a false positive dressed up as a finding.
- Selective reporting. Hiding the tests that came out null.
- Flexible analysis. Trying many outcomes until one is significant.
- No correction. Treating each test as if it were the only one.
These practices inflate false positives, which is a major reason so many published results fail to replicate. Pre-registering hypotheses and applying a correction for the number of tests are the standard defenses.
