A/B Testing Statistical Foundations for Practitioners

Why Statistical Rigor Matters in Testing

A 2024 analysis by Statsig found that 38% of declared A/B test winners were false positives -- the observed effect was indistinguishable from random noise. These false wins lead to product changes that have no real impact, consuming engineering resources and potentially degrading user experience. The root cause is almost always inadequate sample sizes, premature test stopping, or multiple comparison errors.

Statistical testing provides a framework for distinguishing signal from noise. Without it, you are pattern-matching against randomness. A conversion rate that increases from 3.0% to 3.3% might represent a genuine improvement or a lucky streak. Only proper statistical analysis can determine which explanation is more plausible given the sample size and observed variance.

The cost of false negatives is equally real but less visible. A valid improvement that gets killed because the test was underpowered -- too few users to detect a real but small effect -- represents lost revenue that never appears on any report. Proper power analysis before launching tests ensures you can detect effects of the size that would be business-meaningful.

Sample Size Calculation and Power Analysis

Sample size depends on four factors: baseline conversion rate, minimum detectable effect (MDE), significance level (alpha), and statistical power. The MDE is the smallest improvement that would be worth implementing. For a checkout page with 5% conversion, a 0.5% absolute increase might represent millions in revenue. For a feature toggle on a settings page, a 5% relative lift might not justify the engineering cost.

Power -- typically set at 80% -- represents the probability of detecting a real effect when it exists. At 80% power, you will correctly identify a true winner 8 out of 10 times. Increasing power to 90% or 95% requires substantially larger samples. The tradeoff is test duration versus detection reliability. Evan Miller's sample size calculator and tools like Optimizely's Stats Engine make these calculations accessible without manual computation.

Most practitioners underestimate required sample sizes. Detecting a 1% relative improvement in a 5% conversion rate with 80% power and 95% confidence requires approximately 780,000 observations per variation. At 1,000 daily visitors per variation, that test runs for over two years. This math forces pragmatic decisions about which improvements are realistically detectable given your traffic volume.

Significance Testing and Interpretation

Frequentist significance testing asks: if there were no real difference between variations, how likely is the observed result? A p-value of 0.03 means there is a 3% probability of seeing this result by chance alone. This is not the same as a 97% probability that the treatment is better -- a distinction that trips up even experienced practitioners.

Bayesian A/B testing offers an alternative framework that directly answers the question practitioners care about: what is the probability that variation B is better than A? Bayesian methods provide probability distributions over the true effect size, enabling statements like there is an 92% probability that B improves conversion by 0.5% to 2.1%. Google's Bayesian A/B testing framework and VWO's SmartStats use this approach.

Confidence intervals provide more information than p-values alone. An interval of [0.1%, 2.5%] tells you both that the effect is likely positive and the plausible range of improvement. An interval of [-0.3%, 3.1%] tells you the effect might be zero even if the point estimate is positive. Always report and interpret confidence intervals alongside significance decisions.

Common Pitfalls That Invalidate Results

Peeking -- checking results repeatedly and stopping when significance is reached -- inflates false positive rates dramatically. With daily checks over a 30-day test, the actual false positive rate can exceed 30% even with a 5% significance threshold. Sequential testing methods (like alpha spending functions or always-valid confidence intervals) account for multiple looks at the data without inflating error rates.

The multiple comparisons problem arises when testing several metrics or segments simultaneously. Testing 20 metrics guarantees at least one will appear significant by chance at the 5% level. The Bonferroni correction (dividing alpha by the number of comparisons) is the simplest fix, though it is conservative. The Benjamini-Hochberg procedure provides better power while controlling the false discovery rate.

Sample ratio mismatch -- when the actual traffic split differs from the intended split -- indicates a systematic bias in assignment. If you intended 50/50 but observe 48/52, something is wrong with the randomization, and results cannot be trusted regardless of statistical significance. Check sample ratios before interpreting any test result. Consistent mismatches point to technical bugs in the assignment mechanism.

Building a Testing Culture That Learns

Effective testing programs focus on learning rate rather than win rate. A team that runs 50 tests per quarter and learns something from each -- including the 60% that show no significant effect -- builds compounding knowledge about their users. A team that runs 10 tests and celebrates the 3 winners accumulates much less insight over the same period.

Document every test with hypothesis, design, results, and interpretation regardless of outcome. This repository prevents repeated testing of the same ideas and enables meta-analysis across tests. Over time, patterns emerge: certain types of changes consistently produce results (reducing friction), while others rarely do (changing button colors). These patterns guide future hypotheses toward higher-probability areas.

Invest in testing infrastructure that reduces the cost per experiment. Feature flags, automated analysis pipelines, and standardized reporting templates lower the barrier to running tests. When a product manager can launch a test in 30 minutes rather than a sprint, the organization runs more experiments and accumulates knowledge faster. Eppo's 2024 experimentation report found that companies with mature testing infrastructure ran 4x more experiments per quarter than those without.