Find your row. It tells you the smallest improvement a four-week test on that page could reliably see, and so which kind of evidence your store can afford.
Count the visitors who reach the page you want to change, not the site’s total. Then read across.
| Visitors to the page in 4 weeks | Smallest lift a 4-week test can reliably detect, at 2% conversion | At 3% conversion | Weeks to see a 10% lift, at 2% |
|---|---|---|---|
| 10,000 | 40% | 32% | 63 |
| 25,000 | 25% | 20% | 26 |
| 50,000 | 18% | 14% | 13 |
| 100,000 | 13% | 10% | 7 |
| 250,000 | 8% | 6% | 3 |
| 1,000,000 | 4% | 3% | 1 |
DerivedTwo versions with traffic split evenly, a two-sided 5% significance level and 80% power, using the standard rule of thumb for sample size in chapter 3. Lifts are relative: a 20% lift takes 2.0% to 2.4%. Weeks are rounded up to whole weeks.
Now hold that against what real changes do. At Bing, one of the most tested products in the world, a change to how ad headlines were displayed raised revenue by 12%, worth over $100 million a year in the US; the book that tells the story says simple changes that big happen there “only once every few years” Published. If your row says the smallest lift you can see is 18% or more, a clean A/B test can’t reliably see most of the changes you’re likely to make. It will still show you winners. They’ll mostly be noise.
Your traffic decides what you can learn from a test before your ideas do.
This is one chapter of The Honest Test, which is free and readable in full on a single page with no form in front of it.