Statistical significance and practical significance answer different questions about data. Statistical significance asks whether an observed result is unlikely to have happened by random chance under a null hypothesis. Practical significance asks whether the size of the result is large enough to matter in real life.
This distinction matters because a result can be statistically convincing but too small to be useful.
Sample size plays a major role in this difference. With a very large sample, even a tiny effect can produce a small p-value, while a small sample may miss an effect that could be important. Effect size, confidence intervals, and real-world context help decide whether a result is meaningful.
Good statistical reporting includes both the evidence against the null hypothesis and the size and importance of the effect.
Key Facts
- Statistical significance is often judged by p < alpha, where alpha is commonly 0.05.
- A p-value is the probability of getting results at least as extreme as the observed results if the null hypothesis is true.
- Practical significance depends on effect size, cost, risk, benefit, and context.
- Effect size for a difference in means can be measured by Cohen's d = (mean1 - mean2) / pooled standard deviation.
- A large sample size can make a very small effect statistically significant.
- Report both statistical evidence and real-world size, such as p-value, confidence interval, and effect size.
Vocabulary
- Statistical significance
- Statistical significance means the observed result is unlikely to occur by random chance if the null hypothesis is true.
- Practical significance
- Practical significance means the result is large enough or important enough to matter in a real-world setting.
- P-value
- A p-value is the probability of observing a result as extreme as the sample result, assuming the null hypothesis is true.
- Effect size
- Effect size is a numerical measure of how large a difference or relationship is.
- Sample size
- Sample size is the number of observations or participants included in a study.
Common Mistakes to Avoid
- Treating p < 0.05 as proof the result is important. A small p-value shows evidence against the null hypothesis, but it does not show that the effect is large or useful.
- Ignoring sample size when interpreting significance. Very large samples can detect tiny effects, while small samples may fail to detect meaningful effects.
- Reporting only the p-value. This is incomplete because readers also need an effect size, confidence interval, and context to judge real-world meaning.
- Confusing statistical significance with causation. A significant association does not prove one variable caused another unless the study design supports a causal claim.
Practice Questions
- 1 A study with 10,000 people finds that a new study app raises average test scores by 0.4 points on a 100-point exam, with p = 0.01. Is the result statistically significant at alpha = 0.05, and is it likely to be practically significant for most students? Explain briefly.
- 2 Two teaching methods are compared. Method A has a mean score of 82, Method B has a mean score of 78, and the pooled standard deviation is 8. Calculate Cohen's d using d = (meanA - meanB) / pooled standard deviation. Interpret whether the effect is small, moderate, or large if 0.2 is small, 0.5 is moderate, and 0.8 is large.
- 3 A medical treatment lowers average recovery time by 1 day, but it costs much more and has uncomfortable side effects. Explain why a statistically significant result might still not be practically significant in this situation.