This cheat sheet covers how p-values, significance levels, and error types work in hypothesis testing. Students need these ideas to decide whether sample data give strong enough evidence against a null hypothesis. It helps connect the decision rule to the real-world meaning of rejecting or failing to reject a claim.
It also shows how errors and power describe the reliability of a statistical test.
The core idea is to compare the p-value to the significance level . If , reject ; if , fail to reject . A Type I error means rejecting a true , with probability , while a Type II error means failing to reject a false , with probability .
Power is the probability of correctly rejecting a false null hypothesis, given by .
Key Facts
- The null hypothesis is the starting claim, often a statement of no effect, no difference, or equality.
- The alternative hypothesis is the claim supported when the data provide strong evidence against .
- A p-value is the probability, assuming is true, of getting a test statistic at least as extreme as the observed one.
- The decision rule is reject if and fail to reject if .
- A Type I error occurs when a true is rejected, and its probability is .
- A Type II error occurs when a false is not rejected, and its probability is .
- The power of a test is the probability of rejecting a false , so .
- Lowering makes Type I errors less likely but can make Type II errors more likely if the sample size stays the same.
Vocabulary
- Null hypothesis
- The null hypothesis is the default claim tested by the data, usually stating no change, no difference, or no effect.
- Alternative hypothesis
- The alternative hypothesis is the competing claim that the test seeks evidence to support.
- P-value
- A p-value is the probability of observing results at least as extreme as the sample result, assuming is true.
- Significance level
- The significance level is the cutoff probability for deciding whether the evidence is strong enough to reject .
- Type I error
- A Type I error happens when the test rejects even though is actually true.
- Power
- Power is the probability that a test correctly rejects a false null hypothesis, equal to .
Common Mistakes to Avoid
- Saying the p-value is the probability that is true is wrong because a p-value assumes is true and measures how unusual the data are under that assumption.
- Rejecting when is wrong because the data are not statistically significant at that chosen significance level.
- Claiming that failing to reject proves is true is wrong because the test may simply lack enough evidence or power to detect an effect.
- Confusing Type I and Type II errors is wrong because Type I means rejecting a true , while Type II means failing to reject a false .
- Thinking a smaller p-value proves a larger effect is wrong because the p-value depends on sample size, variability, and effect size.
Practice Questions
- 1 A hypothesis test gives with . Should you reject or fail to reject ?
- 2 A test has . What is the power of the test?
- 3 A medical screening test uses . What is the probability of a Type I error if the null hypothesis is true?
- 4 Explain why lowering can reduce the chance of a Type I error but may increase the chance of a Type II error.
Understanding P-Values & Type I/Type II Errors
A p-value describes how unusual the sample result would be if the starting claim were correct. It is not the chance that the starting claim is true. It is not the chance that the result happened by luck alone.
This distinction matters because a hypothesis test begins by temporarily treating the null claim as a model for repeated samples. If the observed result would be rare under that model, the data conflict with it. A result can be statistically significant yet still show only a tiny difference.
For example, a large survey may detect a one minute change in average screen time. That change may be real, but its practical importance needs separate judgment.
The two error types have different costs in different settings. A Type I error can mean approving a medicine that does not truly work, flagging an innocent student for cheating, or claiming that a factory process changed when it did not. A Type II error can mean missing a useful medicine, failing to detect cheating, or overlooking a faulty machine.
Scientists and decision makers choose a significance level partly by considering these consequences. In medical safety studies, a false positive claim can cause harm.
In disease screening, missing a serious illness can be worse. There is no single significance level that fits every situation.
Power depends strongly on sample size, the size of the real effect, and the amount of natural variation in the data. Larger samples usually give more precise estimates, making real differences easier to detect. A large effect is easier to find than a small effect.
Noisy measurements make both tasks harder. Researchers often plan a study before collecting data so it has enough power for an effect that would matter in practice.
A non-significant result from a small study is not strong proof that there is no effect. It may simply mean the study had too little information to separate a real pattern from random variation.
When learning hypothesis tests, focus on the full chain of reasoning rather than memorizing a cutoff rule. State what the null claim means in the real situation. Identify the population, the sample, and the measurement being studied.
Check whether the sampling method supports the assumptions of the test, especially randomness and independence. Choose a one-sided or two-sided alternative before seeing the results. Changing the direction after looking at data can make evidence seem stronger than it is.
When many tests are run, some small p-values will occur by chance, so repeated testing needs extra caution. Good statistical conclusions report the estimated effect and its uncertainty, then explain what the decision means in context.