A/B Testing Infrastructure Setup Guide

Architecture of an Experimentation Platform

A/B testing infrastructure has four components: assignment (which users see which variant), tracking (recording user behavior), analysis (computing statistical results), and management (creating and monitoring experiments). Open source tools like GrowthBook, Unleash, and Eppo provide these components as integrated platforms. Build-versus-buy depends on experiment volume: teams running fewer than 10 experiments per quarter should use a managed platform rather than building custom infrastructure.

The assignment layer must be deterministic and consistent. A user assigned to variant B must see variant B every time they visit, across sessions and devices. Hash-based assignment using a combination of user ID and experiment ID produces deterministic assignment without storing state. This approach, used by Facebook's PlanOut framework, ensures that assignment is reproducible and that experiments do not interfere with each other when multiple experiments run simultaneously.

The tracking layer must capture events with experiment context -- which variant the user was assigned to when the event occurred. This requires instrumenting the event pipeline to include experiment assignments as metadata. If tracking and assignment are decoupled, analysis produces incorrect results because events cannot be reliably attributed to the correct variant. Netflix's engineering blog documents how they solved this coupling challenge at scale.

Sample Size and Statistical Power

The most common A/B testing mistake is ending experiments too early. Statistical significance requires sufficient sample size, which depends on three factors: baseline conversion rate, minimum detectable effect, and desired statistical power. For a page with a 5% conversion rate, detecting a 10% relative improvement (from 5% to 5.5%) with 80% power and 95% confidence requires approximately 31,000 users per variant. Most online sample size calculators implement this formula.

Running experiments on small populations produces unreliable results. A test with 200 users per variant might show a 30% improvement that disappears when run on 10,000 users. This is not a fluke -- it is the expected behavior of small samples. Evan Miller's sample size calculator and the power analysis functions in R or Python's statsmodels library help teams determine the minimum sample size before starting an experiment.

Sequential testing methods allow early stopping without inflating false positive rates. Traditional fixed-horizon testing requires waiting until the predetermined sample size is reached. Sequential methods like the always-valid p-value or Bayesian approaches allow checking results at any time while maintaining statistical validity. Optimizely and Eppo both implement sequential testing, which is particularly useful for teams that cannot wait weeks for results on high-traffic features.

Avoiding Common Experimentation Pitfalls

Peeking at results before reaching the required sample size inflates the false positive rate from the nominal 5% to as high as 30%, depending on how frequently results are checked. This phenomenon, called the peeking problem or alpha inflation, is well documented in the statistics literature. Either commit to a fixed sample size and do not check early, or use sequential testing methods designed for continuous monitoring.

Simpson's paradox can reverse experiment results when segments are analyzed separately versus together. An experiment might show an overall positive effect while being negative for every individual user segment because the variant changed the segment mix. Always analyze results both in aggregate and by key segments (device type, user tenure, geography) to detect this pattern. Twyman's Law states that any figure that looks interesting or different is usually wrong -- investigate surprising results thoroughly before acting on them.

Novelty effects and primacy effects distort short-running experiments. A new design might perform better initially because users are curious, then regress to baseline as the novelty wears off. Conversely, an interface change might perform worse initially because users are disrupted, then improve as they adapt. Run experiments for at least two full business cycles (typically two weeks) to account for these temporal effects, even if sample size is reached earlier.

Building an Experiment Review Process

Every experiment should have a documented hypothesis, primary metric, guardrail metrics, and a pre-registered analysis plan. The hypothesis states the expected causal relationship: 'Simplifying the checkout form from 5 fields to 3 will increase completion rate because users abandon complex forms.' The primary metric is checkout completion rate. Guardrail metrics include revenue per checkout and customer support ticket rate -- metrics that should not degrade even if the primary metric improves.

Pre-registration prevents p-hacking -- the practice of analyzing data multiple ways until a statistically significant result appears. By documenting the analysis plan before seeing results, the team commits to a specific metric, segment, and decision criteria. If the pre-registered analysis shows no effect, the experiment is negative, regardless of whether post-hoc analysis of a specific segment shows significance.

Create an experiment review board that meets weekly to approve new experiments, review running experiments, and make ship/no-ship decisions on completed experiments. Microsoft's ExP platform processes thousands of experiments through this kind of governance structure. The review board ensures that experiments are well-designed, that results are interpreted correctly, and that decisions are made consistently.

From Experiment Results to Product Decisions

A statistically significant result does not automatically mean the variant should ship. Consider the practical significance -- is the effect large enough to matter? A statistically significant 0.1% improvement in conversion rate on a page with 1,000 monthly visitors generates one additional conversion per month. The engineering cost of maintaining the new variant may exceed the value of that improvement.

Negative results are as valuable as positive ones because they prevent the team from investing in the wrong direction. Document negative results thoroughly: what was tested, what was expected, what was observed, and what the team learned. Booking.com, which runs over 1,000 experiments simultaneously, reports that the majority of experiments show no significant effect. This is normal and expected -- if most experiments succeeded, the team would not be testing ambitious enough ideas.

Build an experiment knowledge base that accumulates learnings over time. After a year of experimentation, the knowledge base reveals patterns: which types of changes consistently improve metrics, which user segments respond most strongly to changes, and which parts of the product are resistant to optimization. This institutional knowledge compounds, making each subsequent experiment more likely to produce actionable results.