A/B testing is a statistical method for comparing two versions of something, such as a webpage, app feature, message, or product design. Users are randomly assigned to Version A or Version B, and a chosen outcome metric is measured for each group. The goal is to decide whether an observed difference is likely due to a real effect or just random variation.
This matters because it helps teams make evidence-based decisions instead of relying on opinion or lucky patterns in small samples.
The core of an A/B test is a hypothesis test. A null hypothesis usually says that Version A and Version B have the same true performance, while an alternative hypothesis says that one version performs differently or better. Random assignment makes the groups comparable, and statistical calculations estimate how surprising the observed difference would be if there were no true effect.
Good A/B testing also requires planning sample size, avoiding repeated peeking, and focusing on metrics that match the real goal.
Understanding Statistics: A/B Testing
A fair test starts before anyone sees a result. Teams choose one main success measure, such as the share of visitors who sign up or the average time needed to finish a task. They should decide what counts as a useful improvement.
A change from ten percent to ten point one percent may be statistically detectable with a huge audience, yet it may not be worth the cost or effort. This is why practical importance matters alongside statistical evidence.
A confidence interval is useful here because it gives a range of plausible values for the true difference. A wide interval means the estimate is still uncertain.
Sample size affects what a test can reveal. Small groups produce noisy results because a few extra purchases or clicks can change the rate sharply. Larger groups reduce this random wobble.
Before running a test, analysts estimate how many users are needed to have a good chance of detecting a meaningful effect. This chance is called statistical power. Power rises when the sample is larger, when the expected effect is bigger, or when the data are less variable.
Ending a test too early can make a temporary pattern look convincing. Checking results every day and stopping at the first small p value increases the risk of a false positive.
The two groups must receive the same experience except for the planned change. If Version B is shown mostly on weekends while Version A was shown mostly on weekdays, time may explain the result. If one version loads more slowly, that can affect behaviour even when the wording or design is the intended change.
Users should normally remain in their assigned group throughout the test. Otherwise, a person could see both versions and their later actions may be influenced by the first one.
It is important to check whether the groups have similar sizes and similar starting characteristics. A large imbalance can signal a technical problem with assignment or tracking.
A low p value does not prove that a new version is correct or guaranteed to work forever. It means the data would be unusual under the assumption of no real difference. False positives still happen, especially when a team tests many button colours, headlines, layouts, and audience groups.
With enough comparisons, one option can appear successful by chance. A planned analysis limits this problem. Teams can state their primary measure in advance and treat extra findings as ideas for later tests.
They should inspect data quality too. Missing purchases, duplicate users, bot traffic, or a broken event tracker can create a result that statistics cannot fix.
Students meet A/B testing in online stores, streaming apps, school surveys, public health messages, and game design. A website may test a simpler checkout page, while a school might compare two reminder messages to see which leads to more completed forms. The method has ethical limits.
A test should not hide important information, unfairly burden one group, or expose people to harm just to collect data. When learning this topic, separate three ideas carefully. Statistical significance concerns chance variation.
Effect size describes how large the difference is. Causation depends on a well-run experiment with trustworthy measurement. Strong conclusions need all three.
Key Facts
- Random assignment helps make Version A and Version B comparable before the treatment is applied.
- Null hypothesis: H0: pA = pB, meaning the true conversion rates are equal.
- Alternative hypothesis: HA: pA != pB for a two-sided test, or HA: pB > pA for a one-sided test.
- Sample proportion: p_hat = x/n, where x is the number of successes and n is the number of users.
- Difference in conversion rates: d = p_hatB - p_hatA.
- A p-value is the probability of getting results at least as extreme as the observed result if the null hypothesis is true.
Vocabulary
- A/B test
- A controlled experiment that compares two versions by randomly assigning participants and measuring an outcome.
- Random assignment
- The process of placing participants into groups by chance so that the groups are similar on average.
- Conversion rate
- The proportion of users who complete a desired action, such as clicking, signing up, or buying.
- P-value
- The probability of seeing a result as extreme as the observed one if the null hypothesis is true.
- Statistical significance
- A result is statistically significant when it is unlikely enough under the null hypothesis to reject that hypothesis at a chosen significance level.
Common Mistakes to Avoid
- Peeking at results repeatedly and stopping when the p-value looks small. This inflates the chance of a false positive unless the test design accounts for repeated checks.
- Changing the main metric after seeing the data. This makes the test biased because the decision rule was not fixed before the experiment began.
- Assuming a statistically significant result is automatically important. A tiny effect can be statistically significant with a large sample but still have little practical value.
- Ignoring random assignment problems such as putting mobile users mostly in one group. This is wrong because group differences can be caused by user mix instead of the tested version.
Practice Questions
- 1 Version A has 40 conversions out of 1000 users, and Version B has 55 conversions out of 1000 users. Find p_hatA, p_hatB, and the difference p_hatB - p_hatA.
- 2 An A/B test uses a significance level of alpha = 0.05 and produces a p-value of 0.018. Should the null hypothesis be rejected, and what does that decision mean in context?
- 3 A team checks the A/B test every hour and plans to stop as soon as Version B becomes significant. Explain why this can lead to a misleading conclusion and what a better plan would be.