A p-value is a probability used in hypothesis testing to measure how unusual your data would be if the null hypothesis were true. It helps decide whether an observed result is surprising enough to question a default claim, such as no effect or no difference. In science, medicine, engineering, and social science, p-values help separate random variation from evidence of a real pattern.
They do not prove a hypothesis, but they quantify how compatible the data are with a specific model.
Understanding Statistics: The p-Value in Depth
Before a p-value can be calculated, researchers choose a test statistic. This is one number that summarizes the gap between what was observed and what the null model predicts. A difference between sample means, a correlation, or a proportion can serve this role.
The test statistic is compared with a sampling distribution. That distribution describes values expected from many repeated samples when the null model holds. A result far from the center of this distribution gives a smaller p-value.
A result near the center gives a larger one. The calculation depends on the kind of data, the sample size, and the assumptions of the chosen test.
The direction of a test matters. A one-sided test counts unusual results in one specified direction, such as a new battery lasting longer than an old one. A two-sided test counts unusual results in either direction, such as a battery lasting either longer or shorter.
The direction must be chosen before the data are examined. Choosing a one-sided test after seeing a convenient direction makes the result look stronger than it really is. This is one reason a p-value is connected to the full study plan, not just to the final data table.
The significance level is a rule set in advance for making decisions. A common level is five percent. If researchers repeatedly used this rule when there were truly no effects, about five out of every hundred tests could still lead to a false rejection of the null hypothesis.
This is called a Type One error. Lowering the significance level reduces this risk, though it can make real effects harder to detect.
That ability to detect a real effect is called power. Larger samples usually give more power because random variation has less influence on the estimate.
Students often meet p-values in reports about medicines, teaching methods, polls, sports data, or psychology experiments. A small p-value does not tell whether an effect is large enough to matter in real life. A tiny improvement can produce a small p-value if the sample is very large.
For this reason, good reports include an effect size and a confidence interval. These show the estimated size of the difference and the range of plausible values. It is important to check whether observations were independent, whether the sample was selected fairly, and whether many tests were tried.
Running many analyses until one gives a small p-value increases the chance of a misleading result. A careful conclusion states what the data provide evidence against, while admitting uncertainty and limits.
Key Facts
- A p-value is P(data as extreme or more extreme than observed | H0 is true).
- A small p-value means the observed result would be unlikely under the null hypothesis.
- Reject H0 when p-value ≤ alpha, where alpha is the chosen significance level.
- For a right-tailed z-test, p-value = P(Z ≥ zobs).
- For a two-tailed z-test, p-value = 2P(Z ≥ |zobs|).
- A p-value is not P(H0 is true | data), and it is not the probability that the result happened by chance.
Vocabulary
- Null hypothesis
- The null hypothesis, written H0, is the default claim being tested, often stating that there is no effect, no difference, or no association.
- Alternative hypothesis
- The alternative hypothesis, written Ha or H1, is the claim that the test looks for evidence to support.
- p-value
- The p-value is the probability of getting a result at least as extreme as the observed result, assuming the null hypothesis is true.
- Significance level
- The significance level, written alpha, is the cutoff probability chosen before the test for deciding when to reject the null hypothesis.
- Test statistic
- A test statistic is a standardized number, such as z or t, that measures how far the observed data are from what the null hypothesis predicts.
Common Mistakes to Avoid
- Saying the p-value is the probability that H0 is true is wrong because the p-value assumes H0 is true and then asks how unusual the data are under that assumption.
- Treating p = 0.049 and p = 0.051 as completely different is wrong because the evidence changes smoothly, while the alpha cutoff is a decision rule.
- Using alpha after seeing the p-value is wrong because alpha should be chosen before the analysis to avoid biased decision making.
- Thinking a small p-value proves a large or important effect is wrong because statistical significance depends on sample size as well as effect size.
Practice Questions
- 1 A right-tailed z-test gives zobs = 1.96. Using P(Z ≥ 1.96) = 0.025, what is the p-value, and would you reject H0 at alpha = 0.05?
- 2 A two-tailed z-test gives zobs = -2.30. Using P(Z ≥ 2.30) = 0.0107, calculate the p-value and decide whether the result is significant at alpha = 0.01.
- 3 A study reports p = 0.03 for a new treatment compared with a placebo. Explain what this p-value means, what it does not mean, and how the conclusion depends on alpha.