Testing tools will happily start any test. None of them leads with the question that decides whether it’s worth starting: how many visitors does it take to see the lift you care about? The answer comes from a rule of thumb that statisticians have used for decades.

The rule

For a test with two versions and traffic split evenly, the number of visitors each version needs is about:

This is Lehr’s rule, and Kohavi and his co-authors give it for exactly this use: it provides a two-sided 5% significance level and 80% power, which means that if the lift is real and that size, the test will detect it four times in five. Replace the 16 with 21 for 90% power Published.

Work it through and a shortcut appears. To see a 10% lift you need about 1,600 orders in each version, and to see a 5% lift about 6,300, almost regardless of your conversion rate Derived. The statistician Martin Goodson, then at the testing company Qubit, gave the same rule of thumb in 2014: 1,600 conversions per group for a 10% lift, 6,000 for 5% Reported. So the fastest way to know if you can test a page is to count its orders. A product page that produces 400 orders a month needs about eight months to see a 10% lift.

Count the orders, not the visitors. About 1,600 per version to see a 10% lift.

Kohavi’s own summary is blunt: A/B tests are useful for effects of reasonable size when you have “at least, thousands of active users, preferably tens of thousands” Published. Compare that with what some tools will accept. VWO, a popular testing tool, says its engine needs at least 25 conversions per version and 1,500 visitors in total before it will call a result Reported. That’s enough to show a winner. It’s nowhere near enough to know if the winner is real.

Revenue needs more than orders

Conversion rate is the cheapest metric to test, because every visitor either orders or doesn’t. Revenue per visitor is far noisier, because a few large orders swing it. In Kohavi’s worked example, detecting a 5% change in revenue per user took 3.3 times as many users as detecting a 5% change in conversion rate Published. In studies of retail advertising, a customer’s sales commonly varied by ten times their average Published. So test on orders per visitor, and check that order value didn’t fall. Changes that aim straight at order value, like bundles, price points and free-shipping thresholds, need revenue or contribution as the metric, and that needs a big store. Chapter 11 has the alternatives.

Test where the traffic is

The visitors that count are the ones who see the change. A new product page layout is seen by product page visitors, not by the whole site. A checkout change is seen only by people who reach checkout, a small fraction of your traffic but with a high conversion rate, which helps. Run the numbers for the page, and prefer changes that apply across many pages at once, such as a template, where the traffic adds up.

Look at the fourth number in the example. A test stopped at four weeks, on a page that needed twenty-one, has about a one-in-four chance of detecting a real 10% lift. Put that power into the tool in the next chapter and see what it does to the winners.

Do this

• Run the planner for your top three pages at a 10% lift. Write the weeks needed next to each page on your test roadmap. Any test planned for a page that needs more than your longest acceptable run (four weeks is a good default; past eight, expiring cookies start to mix the groups) either becomes a bolder change or leaves the roadmap.