Why low traffic breaks most A/B tests
An A/B test compares two versions and asks whether the difference is real or noise. To answer, it needs enough observations to separate signal from chance, and startups rarely have them. With a few hundred conversions a month, the math says you can only spot large differences — so the popular advice to test everything sets you up to run tests that were never going to conclude. Picking the few tests worth running is the point of an experimentation roadmap.
Statistical power and MDE in plain words
Statistical power is the probability a test detects a real effect when one exists — 80% is the usual target. The minimum detectable effect (MDE) is the smallest improvement a test can catch at that power, given your baseline rate and sample size. The two move together: fewer visitors push the MDE up. At startup scale the MDE is often a 20% to 50% relative swing, not the 2% a button color might deliver. If your change is smaller than your MDE, the test cannot see it within any realistic window.
Why only big effects are detectable
Required sample size scales roughly with the inverse square of the effect you want to detect, so halving the MDE roughly quadruples the visitors you need. Optimizely's own worked example puts numbers on it: at a 10% baseline, detecting a 3% lift needs roughly fifteen times the visitors that a 10% lift does. The effects a small team can measure honestly are therefore the coarse ones: a new pricing page, a reworked onboarding flow, a different core offer — not micro-copy. A CRO specialist spends most of their time deciding which changes are big enough to be worth a test.
Sequential and Bayesian testing help, a little
Sequential methods let you stop early when a result is decisive, with the math built to survive repeated checks. Evan Miller's sequential design can cut observations by 25% to 50% when the true effect is large — and it exists precisely so you are not peeking at a fixed-sample test. Bayesian approaches report the probability one variant beats another rather than a p-value, which is easier to reason about mid-flight. Both reduce the sample-size problem. Neither removes it: if the true effect is tiny, every method still needs more data than you have.
Test bigger changes, and guard against peeking
Two rules keep low-traffic testing honest. First, test bigger changes: pick swings large enough to clear your MDE, and run one clean A/B rather than a multivariate grid that splits your traffic further. Second, fix the sample size and stopping rule before you start, and do not call a winner the first day it looks significant — early peeking at a fixed-sample test is how noise gets shipped as a win. If you run automated split steps inside a journey, the same discipline applies: pre-register the metric and the horizon. A holdout group — a slice you deliberately keep on the old experience — is a simple way to sanity-check that an effect is real. Small teams run few tests, so pick the ones worth running and run them properly.