Bootstrap and permutation tests use repeated resampling to answer statistical questions when formulas are difficult or assumptions are uncertain. This cheat sheet helps students compare two powerful simulation methods used in modern statistics. It is especially useful for understanding confidence intervals, significance tests, and randomization-based reasoning.
Students need these tools to connect data, variability, and evidence in a clear way.
Bootstrap methods estimate sampling variability by resampling with replacement from the observed data. Permutation tests test a null hypothesis by rearranging labels or outcomes to simulate what random chance could produce. The key ideas are the observed statistic, the resampling distribution, the confidence interval, and the p-value.
Correct interpretation depends on matching the method to the question and keeping the null hypothesis clear.
Key Facts
- A bootstrap sample is created by sampling observations with replacement from the original sample of size .
- A bootstrap statistic, such as or , is calculated from each bootstrap sample.
- A bootstrap standard error can be estimated by the standard deviation of the bootstrap statistics, .
- A percentile bootstrap confidence interval uses the middle values of the bootstrap distribution, such as the and percentiles for a interval.
- A permutation test simulates the null hypothesis by randomly rearranging labels, groups, or paired signs while keeping the original data values fixed.
- A p-value is the proportion of simulated statistics at least as extreme as the observed statistic, .
- Bootstrap methods are mainly used for estimation, while permutation tests are mainly used for testing hypotheses.
- For a two-sided test, extremeness is measured in both directions, often using .
Vocabulary
- Bootstrap sample
- A resample made by selecting observations with replacement from an original sample of size .
- Bootstrap distribution
- The distribution of a statistic calculated from many bootstrap samples.
- Permutation test
- A significance test that simulates the null hypothesis by randomly rearranging data labels or outcomes.
- Observed statistic
- The statistic calculated from the original data, such as or .
- P-value
- The probability, assuming the null hypothesis is true, of getting a statistic as extreme as or more extreme than the observed statistic.
- Confidence interval
- An interval of plausible values for a population parameter, often written as .
Common Mistakes to Avoid
- Sampling without replacement for a bootstrap sample is wrong because bootstrap samples must mimic repeated sampling by using replacement.
- Using a permutation test to build a confidence interval is usually wrong because permutation tests are designed to test a null hypothesis, not directly estimate uncertainty.
- Forgetting to center the permutation distribution at the null value is wrong because the test must simulate what would happen if were true.
- Counting only values greater than the observed statistic in a two-sided test is wrong because two-sided p-values include results at least as extreme in both directions.
- Interpreting a confidence interval as a probability that the fixed parameter is inside one computed interval is wrong because the method captures the parameter in about of repeated samples.
Practice Questions
- 1 A sample has values. How many observations are selected to form one bootstrap sample, and is replacement used?
- 2 In a permutation test, out of simulated statistics are at least as extreme as the observed statistic. Find the p-value using .
- 3 A bootstrap distribution for a mean has a percentile of and a percentile of . State the approximate confidence interval.
- 4 Explain why a bootstrap procedure keeps the data values but resamples with replacement, while a permutation test keeps the data values but rearranges labels under the null hypothesis.