The significance level alpha, written as α, is the cutoff a researcher chooses before running a hypothesis test. It represents the probability of rejecting the null hypothesis when the null hypothesis is actually true. In a bell-curve test diagram, α is shown as the shaded rejection region in the tail or tails of the distribution.
This idea matters because it sets the standard for how strong the evidence must be before a result is called statistically significant.
A common choice is α = 0.05, meaning the researcher accepts a 5% chance of a Type I error if the null hypothesis is true. Lowering α makes false positives less likely, but it also makes it harder to detect real effects, which can increase the chance of a Type II error. Researchers choose α based on the consequences of mistakes, the field's standards, the study design, and whether the test is one-tailed or two-tailed.
Alpha is not the probability that the null hypothesis is true, and it does not measure the size or importance of an effect.
Key Facts
- α = P(reject H0 | H0 is true)
- If α = 0.05, the rejection region contains 5% of the null distribution.
- A result is statistically significant when p ≤ α.
- Type I error probability = α.
- Power = 1 - β, where β is the probability of a Type II error.
- For a two-tailed test with α = 0.05, each tail has area 0.025.
Vocabulary
- Significance level
- The chosen cutoff probability for deciding when evidence against the null hypothesis is strong enough to reject it.
- Alpha
- Alpha, written α, is the probability of making a Type I error in a hypothesis test.
- Null hypothesis
- The null hypothesis, written H0, is the default claim that there is no effect, no difference, or no change.
- P-value
- A p-value is the probability of getting a result at least as extreme as the observed result, assuming the null hypothesis is true.
- Type II error
- A Type II error occurs when a test fails to reject a false null hypothesis.
Common Mistakes to Avoid
- Saying α is the probability that H0 is true is wrong because α is calculated assuming H0 is true, not as the probability of H0 itself.
- Choosing α after seeing the p-value is wrong because it changes the rules of the test and can make results look more convincing than they are.
- Thinking p = 0.05 proves an effect exists is wrong because statistical significance only indicates evidence against H0 under the model, not absolute proof.
- Using α = 0.05 for every study without thinking is wrong because high-risk decisions may need a smaller α, while exploratory work may use a different standard.
Practice Questions
- 1 A researcher uses α = 0.05 in a one-tailed right-tail test. What percent of the null distribution is in the rejection region?
- 2 In a two-tailed test with α = 0.01, how much area is placed in each tail of the rejection region?
- 3 A medical screening test could cause harmful treatment if it gives a false positive. Should the researcher consider using a larger or smaller α than 0.05? Explain the reasoning in terms of Type I and Type II errors.