The significance level alpha, written as α, is the cutoff a researcher chooses before running a hypothesis test. It represents the probability of rejecting the null hypothesis when the null hypothesis is actually true. In a bell-curve test diagram, α is shown as the shaded rejection region in the tail or tails of the distribution.
This idea matters because it sets the standard for how strong the evidence must be before a result is called statistically significant.
A common choice is α = 0.05, meaning the researcher accepts a 5% chance of a Type I error if the null hypothesis is true. Lowering α makes false positives less likely, but it also makes it harder to detect real effects, which can increase the chance of a Type II error. Researchers choose α based on the consequences of mistakes, the field's standards, the study design, and whether the test is one-tailed or two-tailed.
Alpha is not the probability that the null hypothesis is true, and it does not measure the size or importance of an effect.
Understanding Statistics: The Significance Level Alpha
A hypothesis test works by imagining what results would look like if the null model were correct over many repeated samples. The p value measures how unusual the observed result would be within that imagined set of repeats. A small p value means the data would be rare under the null model.
It does not prove that the null model is false. Random samples can produce rare outcomes by chance. Alpha provides a rule for acting when that happens.
Before seeing the data, the researcher decides how rare a result must be before rejecting the null model. Setting this rule in advance protects against changing the standard after an interesting result appears.
The chosen alpha determines a critical value for the test statistic. For example, a two sided test using an alpha of five percent places the cutoff far from the center of the null distribution on both sides. An observed statistic beyond either cutoff leads to rejection.
A one sided test puts the full allowed error rate on one side, so its cutoff is less extreme in the predicted direction. This makes one sided tests more sensitive to an effect in that direction.
It cannot fairly be changed into a two sided test after the data are known. The test direction must match the research claim and be chosen beforehand.
Alpha becomes especially important when a study performs many tests. If a class compares test scores across twenty different groups, some comparisons may look significant just through random variation. Using the same five percent cutoff for every comparison increases the chance of at least one false positive somewhere in the set.
Researchers may use a stricter alpha or an adjustment method to control this problem. This issue appears in medical studies that test many outcomes, in genetics where thousands of genes are examined, and in business reports that search through many customer measures. Looking repeatedly at data while collecting it can create a similar problem unless stopping rules were planned in advance.
A smaller alpha is sensible when a false positive could cause serious harm. A new drug, a safety device, or a criminal evidence procedure may need very strong evidence before a claim is accepted. But strict cutoffs do not create better data by themselves.
They can cause real effects to be missed when the sample is too small or measurements are noisy. Researchers improve their ability to detect meaningful effects by planning a large enough sample, reducing measurement error, and selecting a realistic effect size for the study plan. Students should read significance results alongside the estimated effect size and confidence interval.
A tiny effect can pass a significance test in a huge sample, while an important effect can fail to pass in a small study. Replication in new data remains important.
Key Facts
- α = P(reject H0 | H0 is true)
- If α = 0.05, the rejection region contains 5% of the null distribution.
- A result is statistically significant when p ≤ α.
- Type I error probability = α.
- Power = 1 - β, where β is the probability of a Type II error.
- For a two-tailed test with α = 0.05, each tail has area 0.025.
Vocabulary
- Significance level
- The chosen cutoff probability for deciding when evidence against the null hypothesis is strong enough to reject it.
- Alpha
- Alpha, written α, is the probability of making a Type I error in a hypothesis test.
- Null hypothesis
- The null hypothesis, written H0, is the default claim that there is no effect, no difference, or no change.
- P-value
- A p-value is the probability of getting a result at least as extreme as the observed result, assuming the null hypothesis is true.
- Type II error
- A Type II error occurs when a test fails to reject a false null hypothesis.
Common Mistakes to Avoid
- Saying α is the probability that H0 is true is wrong because α is calculated assuming H0 is true, not as the probability of H0 itself.
- Choosing α after seeing the p-value is wrong because it changes the rules of the test and can make results look more convincing than they are.
- Thinking p = 0.05 proves an effect exists is wrong because statistical significance only indicates evidence against H0 under the model, not absolute proof.
- Using α = 0.05 for every study without thinking is wrong because high-risk decisions may need a smaller α, while exploratory work may use a different standard.
Practice Questions
- 1 A researcher uses α = 0.05 in a one-tailed right-tail test. What percent of the null distribution is in the rejection region?
- 2 In a two-tailed test with α = 0.01, how much area is placed in each tail of the rejection region?
- 3 A medical screening test could cause harmful treatment if it gives a false positive. Should the researcher consider using a larger or smaller α than 0.05? Explain the reasoning in terms of Type I and Type II errors.