Permutation tests are a flexible way to decide whether an observed difference between groups is surprising. Instead of assuming a normal distribution or using a formula from a textbook table, they build a reference distribution directly from the data. The central idea is to ask what results would look like if the group labels did not matter.
This makes permutation tests useful in experiments, biology, psychology, medicine, and any setting with small or unusual data sets.
A permutation test starts by calculating a test statistic, such as the difference in group means. Then the data values are repeatedly shuffled among the group labels, keeping the original group sizes fixed, and the statistic is recalculated each time. These shuffled statistics form a null distribution, which represents results expected if there is no real group effect.
The p-value is the proportion of shuffled statistics at least as extreme as the observed statistic.
Key Facts
- Null hypothesis: group labels are exchangeable, meaning the labels could be reassigned without changing the data-generating process.
- Observed statistic: Tobs = mean(group A) - mean(group B), or another chosen measure such as a median difference or correlation.
- Permutation step: shuffle labels, keep data values fixed, keep original sample sizes fixed, and recompute T.
- Estimated p-value: p = number of permuted statistics at least as extreme as Tobs divided by number of permutations.
- For a two-sided mean difference test: extreme means |Tperm| >= |Tobs|.
- Permutation tests shine when sample sizes are small, distributions are nonnormal, or a custom test statistic is needed.
Vocabulary
- Permutation test
- A statistical test that estimates how surprising an observed statistic is by repeatedly shuffling labels and recalculating the statistic.
- Null distribution
- The distribution of test statistics produced under the assumption that there is no real effect or difference.
- Test statistic
- A single number calculated from the data that measures the effect being tested, such as a difference in means.
- P-value
- The proportion of null results that are as extreme as or more extreme than the observed result.
- Exchangeability
- The condition that data labels can be swapped under the null hypothesis without changing the meaning of the experiment.
Common Mistakes to Avoid
- Shuffling the data values instead of the group labels is wrong because a permutation test keeps the observed values fixed and only reassigns labels under the null model.
- Changing the group sizes during shuffling is wrong because the null comparison must match the original study design and sample sizes.
- Using a one-sided p-value when the question is two-sided is wrong because effects in the opposite direction are also considered extreme for a two-sided test.
- Interpreting the p-value as the probability that the null hypothesis is true is wrong because the p-value measures how unusual the data are assuming the null hypothesis is true.
Practice Questions
- 1 Group A has values 5, 7, 8 and Group B has values 2, 4, 6. Compute the observed difference in means, Tobs = mean(A) - mean(B).
- 2 In a permutation test with 2000 shuffled labelings, 86 shuffled statistics were at least as extreme as the observed statistic. Estimate the p-value.
- 3 A researcher compares two teaching methods using students from two separate schools, with one method used only at School 1 and the other only at School 2. Explain why a simple permutation test that shuffles student labels between methods may not be valid.