Experimental Design and Causal Inference Primer Cheat Sheet

Experimental design and causal inference help students understand when data can support a cause and effect claim. This cheat sheet covers randomized experiments, observational studies, treatment effects, confounding, blocking, and basic causal diagrams. Students need these tools to separate association from causation and to judge whether a study design answers the question being asked.

The goal is to connect statistical comparisons with the assumptions that make causal conclusions valid.

The most important ideas are potential outcomes, random assignment, control groups, and adjustment for confounders. A treatment effect is often written as a difference in outcomes, such as $\tau = Y(1) - Y(0)$ , even though only one potential outcome is observed for each unit. Randomization makes treatment assignment independent of potential outcomes, written $T \perp \{Y(0),Y(1)\}$ .

In observational studies, causal claims require careful assumptions, such as no unmeasured confounding and proper adjustment for variables that affect both treatment and outcome.

Key Facts

The individual causal effect is $\tau_i = Y_i(1) - Y_i(0)$ , but only one of $Y_i(1)$ or $Y_i(0)$ is observed for each unit.
The average treatment effect is $ATE = E[Y(1) - Y(0)]$ , which summarizes the mean causal effect in a target population.
In a randomized experiment, treatment assignment satisfies $T \perp \{Y(0),Y(1)\}$ , so treated and control groups are comparable in expectation.
A simple difference in means estimator is $\hat{\tau} = \bar{Y}_T - \bar{Y}_C$ , where $\bar{Y}_T$ is the treated mean and $\bar{Y}_C$ is the control mean.
Confounding occurs when a variable $Z$ affects both treatment $T$ and outcome $Y$ , creating a backdoor path such as $T \leftarrow Z \rightarrow Y$ .
Blocking or stratification improves precision by comparing treatment groups within levels of an important variable, then combining stratum estimates.
A standard error for a difference in independent means is $SE(\bar{Y}_T - \bar{Y}_C) = \sqrt{\frac{s_T^2}{n_T} + \frac{s_C^2}{n_C}}$ .
A common approximate confidence interval for a treatment effect is $\hat{\tau} \pm z^{*}SE(\hat{\tau})$ , where $z^{*}$ depends on the confidence level.

Vocabulary

Random assignment: A design method where units are assigned to treatment conditions by chance so that groups are comparable before treatment.
Potential outcome: The outcome a unit would have under a specific treatment condition, such as $Y_i(1)$ under treatment or $Y_i(0)$ under control.
Average treatment effect: The expected difference between potential outcomes in a population, written $ATE = E[Y(1) - Y(0)]$ .
Confounder: A variable that influences both the treatment and the outcome, which can make an association look causal when it is not.
Blocking: A design strategy that groups similar units before random assignment to reduce variation and improve precision.
Causal diagram: A graph using arrows to represent assumed causal relationships among variables.

Common Mistakes to Avoid

Treating correlation as causation is wrong because an association between $T$ and $Y$ may be explained by a confounder $Z$ rather than a causal effect.
Adjusting for a collider is wrong because conditioning on a variable caused by both $T$ and $Y$ can create a false association between them.
Ignoring random assignment failures is wrong because noncompliance, attrition, or missing data can break the comparability that randomization was meant to create.
Comparing raw group means in an observational study is wrong when treatment groups differ on variables that also affect the outcome.
Using a tiny sample without considering power is wrong because a study may have a large $SE(\hat{\tau})$ and fail to detect meaningful effects.

Practice Questions

1 In an experiment, the treated group has $n_T = 50$ , $\bar{Y}_T = 82$ , and the control group has $n_C = 50$ , $\bar{Y}_C = 76$ . Compute $\hat{\tau} = \bar{Y}_T - \bar{Y}_C$ .
2 Given $s_T = 12$ , $s_C = 10$ , $n_T = 40$ , and $n_C = 40$ , compute $SE(\bar{Y}_T - \bar{Y}_C) = \sqrt{\frac{s_T^2}{n_T} + \frac{s_C^2}{n_C}}$ .
3 A blocked experiment has stratum effects $\hat{\tau}_1 = 4$ and $\hat{\tau}_2 = 10$ with stratum weights $w_1 = 0.60$ and $w_2 = 0.40$ . Compute $\hat{\tau}_{blocked} = w_1\hat{\tau}_1 + w_2\hat{\tau}_2$ .
4 A study finds that students who attend extra tutoring have higher test scores, but tutoring was voluntary. Explain why this evidence alone may not justify the causal claim that tutoring caused higher scores.

Understanding Experimental Design and Causal Inference Primer

A study needs a clear unit, treatment, outcome, and target population. The unit might be a student, a patient, a school, or a web user. These choices affect the design.

If a school program is assigned to whole classrooms, students in the same classroom are not independent in the usual way. Their shared teacher and environment can make their results similar. Researchers must plan analysis around the level where assignment happened.

They should define the outcome before collecting data. Changing the outcome after seeing results can make a weak pattern look more convincing than it is.

Random assignment only works when it is carried out carefully. The assignment process should be hidden from people who enroll participants, so they cannot steer certain people into a preferred group. Blinding can reduce bias too.

A participant who knows they received a new treatment may report different symptoms. A researcher who knows group labels may measure outcomes differently without meaning to. Real experiments often face noncompliance, where some people do not take the assigned treatment, and attrition, where some people leave the study.

These problems can destroy the balance created by random assignment. Students should check how many people were assigned, treated, measured, and included in the final analysis.

Causal diagrams help identify variables that should not be handled the same way. A confounder is measured before treatment and helps explain both who receives treatment and what outcome occurs. Adjusting for a genuine confounder can make groups more comparable.

However, adjusting for every available variable is not safe. A mediator lies on the path from treatment to outcome. Controlling for it can remove part of the effect being studied.

A collider is caused by two variables. Conditioning on a collider can create a false relationship between those variables. For example, selecting only people admitted to a hospital may distort relationships between illness risks that influence admission.

Power is the chance that a study can detect an effect of a chosen size when that effect is real. Small samples give noisy estimates, so even useful treatments may appear ineffective. Larger samples usually improve precision, but they cannot fix biased recruitment or poor measurement.

Power depends on sample size, outcome variability, the size of effect worth detecting, and the chosen error rate. Researchers should plan sample size before the study rather than treating a non-significant result as proof of no effect. Finally, consider whether the study population matches the people who will use the result.

A well-run experiment in one clinic, age group, or country may not apply unchanged elsewhere. Internal validity concerns whether the study supports its causal claim. External validity concerns where that claim can reasonably travel.

Sign in to save

Sign in to save

Experimental Design and Causal Inference Primer Cheat Sheet

Related Tools

Related Labs

Related Worksheets

Related Infographics

Study as Flashcards

Key Facts

Vocabulary

Common Mistakes to Avoid

Practice Questions

Understanding Experimental Design and Causal Inference Primer