Bootstrap and permutation tests use repeated resampling to answer statistical questions when formulas are difficult or assumptions are uncertain. This cheat sheet helps students compare two powerful simulation methods used in modern statistics. It is especially useful for understanding confidence intervals, significance tests, and randomization-based reasoning.
Students need these tools to connect data, variability, and evidence in a clear way.
Bootstrap methods estimate sampling variability by resampling with replacement from the observed data. Permutation tests test a null hypothesis by rearranging labels or outcomes to simulate what random chance could produce. The key ideas are the observed statistic, the resampling distribution, the confidence interval, and the p-value.
Correct interpretation depends on matching the method to the question and keeping the null hypothesis clear.
Key Facts
- A bootstrap sample is created by sampling observations with replacement from the original sample of size .
- A bootstrap statistic, such as or , is calculated from each bootstrap sample.
- A bootstrap standard error can be estimated by the standard deviation of the bootstrap statistics, .
- A percentile bootstrap confidence interval uses the middle values of the bootstrap distribution, such as the and percentiles for a interval.
- A permutation test simulates the null hypothesis by randomly rearranging labels, groups, or paired signs while keeping the original data values fixed.
- A p-value is the proportion of simulated statistics at least as extreme as the observed statistic, .
- Bootstrap methods are mainly used for estimation, while permutation tests are mainly used for testing hypotheses.
- For a two-sided test, extremeness is measured in both directions, often using .
Vocabulary
- Bootstrap sample
- A resample made by selecting observations with replacement from an original sample of size .
- Bootstrap distribution
- The distribution of a statistic calculated from many bootstrap samples.
- Permutation test
- A significance test that simulates the null hypothesis by randomly rearranging data labels or outcomes.
- Observed statistic
- The statistic calculated from the original data, such as or .
- P-value
- The probability, assuming the null hypothesis is true, of getting a statistic as extreme as or more extreme than the observed statistic.
- Confidence interval
- An interval of plausible values for a population parameter, often written as .
Common Mistakes to Avoid
- Sampling without replacement for a bootstrap sample is wrong because bootstrap samples must mimic repeated sampling by using replacement.
- Using a permutation test to build a confidence interval is usually wrong because permutation tests are designed to test a null hypothesis, not directly estimate uncertainty.
- Forgetting to center the permutation distribution at the null value is wrong because the test must simulate what would happen if were true.
- Counting only values greater than the observed statistic in a two-sided test is wrong because two-sided p-values include results at least as extreme in both directions.
- Interpreting a confidence interval as a probability that the fixed parameter is inside one computed interval is wrong because the method captures the parameter in about of repeated samples.
Practice Questions
- 1 A sample has values. How many observations are selected to form one bootstrap sample, and is replacement used?
- 2 In a permutation test, out of simulated statistics are at least as extreme as the observed statistic. Find the p-value using .
- 3 A bootstrap distribution for a mean has a percentile of and a percentile of . State the approximate confidence interval.
- 4 Explain why a bootstrap procedure keeps the data values but resamples with replacement, while a permutation test keeps the data values but rearranges labels under the null hypothesis.
Understanding Bootstrap & Permutation Tests Reference
A resampling method treats the sample as a small stand in for the larger population. This idea works best when the original observations were collected fairly and independently. If a survey missed an important group, repeating its values thousands of times cannot repair that bias.
Bootstrap samples reveal how much a statistic might change from sample to sample. Some original observations appear more than once in a resample, while others do not appear at all. That variation creates a distribution of possible estimates.
A narrow distribution suggests a more precise estimate. A wide distribution suggests that the data leave considerable uncertainty.
The statistic must match the research goal. For numerical data, students may estimate a mean, median, difference in means, or regression slope. For category data, they may estimate a proportion or difference in proportions.
The median is a useful example because its sampling behavior can be awkward to calculate with a standard formula, especially for small samples or skewed data. A bootstrap simulation handles the same basic task by repeatedly recalculating the median.
The resulting interval gives a range of plausible population values, based on the sample and the resampling process. It does not prove that a fixed percentage of individual data values fall inside the interval.
A permutation test starts from a different claim. It assumes that the group labels make no real difference under the null hypothesis. If a study compares a treatment group with a control group, the observed scores stay fixed while the labels are shuffled.
Each shuffle creates a result that could occur if assignment to groups had no effect. The observed difference is then compared with this chance distribution. A small p-value means that results at least this far from the null value were rare in the simulation.
It is not the probability that the null hypothesis is true. It is not a measure of how large or important an effect is.
Students should decide whether the task is estimation or testing before choosing a method. They should state the parameter of interest, define the statistic, and keep the direction of a difference consistent throughout the work. For paired data, such as before and after measurements on the same people, the meaningful values are the within-person differences.
Shuffling whole observations as if the groups were independent would break the study design. The number of simulations matters too. A few hundred runs can give a rough picture, but several thousand make interval endpoints and small p-values more stable.
Graphs of resampling distributions are worth reading carefully. Look for the center, spread, shape, and any unusual results. These features show what the data can support and where the conclusions remain uncertain.