P-Values & Type I/Type II Errors Cheat Sheet

Key Facts

• The null hypothesis $H_0$ is the starting claim, often a statement of no effect, no difference, or equality.
• The alternative hypothesis $H_a$ is the claim supported when the data provide strong evidence against $H_0$.
• A p-value is the probability, assuming $H_0$ is true, of getting a test statistic at least as extreme as the observed one.
• The decision rule is reject $H_0$ if $p \leq \alpha$ and fail to reject $H_0$ if $p > \alpha$.
• A Type I error occurs when a true $H_0$ is rejected, and its probability is $\alpha$.
• A Type II error occurs when a false $H_0$ is not rejected, and its probability is $\beta$.
• The power of a test is the probability of rejecting a false $H_0$, so $\text{power} = 1 - \beta$.
• Lowering $\alpha$ makes Type I errors less likely but can make Type II errors more likely if the sample size stays the same.

Vocabulary

Null hypothesis

The null hypothesis $H_0$ is the default claim tested by the data, usually stating no change, no difference, or no effect.

Alternative hypothesis

The alternative hypothesis $H_a$ is the competing claim that the test seeks evidence to support.

P-value

A p-value is the probability of observing results at least as extreme as the sample result, assuming $H_0$ is true.

Significance level

The significance level $\alpha$ is the cutoff probability for deciding whether the evidence is strong enough to reject $H_0$.

Type I error

A Type I error happens when the test rejects $H_0$ even though $H_0$ is actually true.

Power

Power is the probability that a test correctly rejects a false null hypothesis, equal to $1 - \beta$.