A/B testing is a statistical method for comparing two versions of something, such as a webpage, app feature, message, or product design. Users are randomly assigned to Version A or Version B, and a chosen outcome metric is measured for each group. The goal is to decide whether an observed difference is likely due to a real effect or just random variation.
This matters because it helps teams make evidence-based decisions instead of relying on opinion or lucky patterns in small samples.
The core of an A/B test is a hypothesis test. A null hypothesis usually says that Version A and Version B have the same true performance, while an alternative hypothesis says that one version performs differently or better. Random assignment makes the groups comparable, and statistical calculations estimate how surprising the observed difference would be if there were no true effect.
Good A/B testing also requires planning sample size, avoiding repeated peeking, and focusing on metrics that match the real goal.
Key Facts
- Random assignment helps make Version A and Version B comparable before the treatment is applied.
- Null hypothesis: H0: pA = pB, meaning the true conversion rates are equal.
- Alternative hypothesis: HA: pA != pB for a two-sided test, or HA: pB > pA for a one-sided test.
- Sample proportion: p_hat = x/n, where x is the number of successes and n is the number of users.
- Difference in conversion rates: d = p_hatB - p_hatA.
- A p-value is the probability of getting results at least as extreme as the observed result if the null hypothesis is true.
Vocabulary
- A/B test
- A controlled experiment that compares two versions by randomly assigning participants and measuring an outcome.
- Random assignment
- The process of placing participants into groups by chance so that the groups are similar on average.
- Conversion rate
- The proportion of users who complete a desired action, such as clicking, signing up, or buying.
- P-value
- The probability of seeing a result as extreme as the observed one if the null hypothesis is true.
- Statistical significance
- A result is statistically significant when it is unlikely enough under the null hypothesis to reject that hypothesis at a chosen significance level.
Common Mistakes to Avoid
- Peeking at results repeatedly and stopping when the p-value looks small. This inflates the chance of a false positive unless the test design accounts for repeated checks.
- Changing the main metric after seeing the data. This makes the test biased because the decision rule was not fixed before the experiment began.
- Assuming a statistically significant result is automatically important. A tiny effect can be statistically significant with a large sample but still have little practical value.
- Ignoring random assignment problems such as putting mobile users mostly in one group. This is wrong because group differences can be caused by user mix instead of the tested version.
Practice Questions
- 1 Version A has 40 conversions out of 1000 users, and Version B has 55 conversions out of 1000 users. Find p_hatA, p_hatB, and the difference p_hatB - p_hatA.
- 2 An A/B test uses a significance level of alpha = 0.05 and produces a p-value of 0.018. Should the null hypothesis be rejected, and what does that decision mean in context?
- 3 A team checks the A/B test every hour and plans to stop as soon as Version B becomes significant. Explain why this can lead to a misleading conclusion and what a better plan would be.