A p-value is often used to decide whether a study result looks surprising under a null hypothesis, but it can be misused when researchers try many analyses and report only the one that looks significant. This behavior is called p-hacking, and it increases the chance of false positives. The replication crisis refers to the discovery that many published findings, especially in fields with complex human behavior, fail to appear again when independent researchers repeat the study.
Understanding this problem matters because science depends on results that are not just exciting, but reliable.
Key Facts
- A p-value is P(data at least this extreme | null hypothesis is true), not the probability that the null hypothesis is true.
- A common significance cutoff is p < 0.05, meaning results this extreme occur less than 5 percent of the time if the null hypothesis is true.
- If 20 independent tests are run with α = 0.05, the chance of at least one false positive is 1 - (0.95)^20 = 0.642.
- Multiple comparisons correction makes significance harder to claim, such as Bonferroni α_adjusted = α / number of tests.
- Replication means repeating a study with new data to test whether the original finding appears again under similar conditions.
- Pre-registration records the hypothesis, sample size, variables, and analysis plan before seeing the results.
Vocabulary
- p-hacking
- p-hacking is the practice of trying many analyses, variables, or data exclusions until a statistically significant result is found.
- p-value
- A p-value is the probability of getting results at least as extreme as the observed results if the null hypothesis is true.
- False positive
- A false positive is a result that appears statistically significant even though the tested effect is not real.
- Replication
- Replication is the process of repeating a study with new data to see whether the same finding can be observed again.
- Pre-registration
- Pre-registration is the public recording of a study plan before data analysis to reduce hidden flexibility in research decisions.
Common Mistakes to Avoid
- Treating p < 0.05 as proof of truth is wrong because statistical significance does not guarantee that an effect is real or important.
- Running many tests and reporting only the smallest p-value is wrong because each extra test raises the chance of finding a false positive by luck.
- Changing the hypothesis after seeing the data is wrong because it makes an exploratory pattern look like a planned prediction.
- Ignoring failed replications is wrong because a scientific claim should become stronger or weaker based on the full pattern of evidence, not only the first positive study.
Practice Questions
- 1 A researcher tests 10 independent hypotheses using α = 0.05 for each test. What is the probability of getting at least one false positive if all null hypotheses are true? Use 1 - (0.95)^10.
- 2 A study tests 8 outcomes and wants a familywise error rate of α = 0.05 using the Bonferroni correction. What adjusted significance cutoff should be used for each test?
- 3 A scientist finds p = 0.03 only after removing two outliers, switching from one outcome measure to another, and testing several subgroups. Explain why this result may be less trustworthy than a pre-registered result with the same p-value.