THE LADDER OF EVIDENCE

If the traffic table told you your store can’t test most changes, you still have to change things. The choice isn’t between a perfect experiment and guessing. There’s a ladder of methods between them, each cheaper and weaker than the one above.

1. An A/B test to its planned sizeRandom split, planned size, one primary metric, read with the checks in chapter 5. The strongest evidence, and the most expensive in traffic. Use it for bold changes on your highest-traffic pages.
2. A holdout rolloutShip the change to most visitors and keep a random slice, say 10%, on the old version for a longer period. If the change helps, only the held-back slice misses out while you learn. If it hurts, most visitors bear it, so watch the guardrails weekly. A 90/10 split needs almost three times the visitors of a 50/50 test to reach the same power, so it runs for months, not weeks. Assign the holdout by customer ID where you can, because browser cookies expire over months.
3. On-off weeksSwitch the change on and off a whole week at a time, for six to eight weeks, deciding which week of each pair is on by a coin flip rather than strict alternation. It’s weaker than randomizing visitors: a sale or a press mention can land in one kind of week, returning shoppers see both versions, and you have only a handful of weeks to compare. Compare week by week rather than pooling visitors, and log everything else that happened. Never use it for prices.
4. Before and after, with guardsCompare four whole weeks before the change with four after, with the guards below. The weakest rung that still deserves the name evidence.
5. Research, then a fixFor problems that research shows plainly, like a broken button or a missing delivery date, fix them and move on. Chapter 8.

A weaker method used carefully beats a stronger one used badly.

The guards for a before-and-after

A plain before-and-after comparison fools you whenever anything else changed at the same time: the season, the traffic mix, a promotion, a competitor’s sale. The guards don’t make it an experiment. They make it harder to fool yourself.

• Write down in advance the metric, the weeks you’ll compare and the size of change you’d act on.
• Use whole weeks, and avoid weeks with promotions, launches or holidays on either side.
• Find a comparison that the change didn’t touch. If you changed the mobile product page, desktop is your comparison: did mobile improve more than desktop over the same weeks? If you changed one product’s page, compare with similar products whose pages you left alone.
• Check the traffic mix. If paid social’s share of traffic doubled in the after period, conversion moved for that reason. Compare channel by channel.
• Check last year. If the same weeks rose by a similar amount last year, the season did it.
• Know your noise. Run the same comparison on three or four past periods when nothing changed. If the gap between the two moved by 10% in ordinary months, a 9% result tells you nothing.

Say you rebuild the mobile product page. Over four weeks before and four after, mobile conversion goes from 1.6% to 1.8%, a 12.5% lift. Desktop, untouched, went from 2.8% to 2.9% over the same weeks, about 3.6%. Mobile rose about 9% more than desktop (1.125 ÷ 1.036), and that’s your best estimate of what the rebuild did. It’s weaker than a test, and much better than the 12.5% you’d have reported without the comparison. But check it against noise: at 50,000 mobile visitors a month, a result like this could plausibly be anywhere from a small loss to a 25% gain. That’s what the last guard is for.

Test only bold changes

If the page gets about 50,000 visitors or more in four weeks, you can still A/B test, as long as the change is big enough to clear the smallest lift you can see. That rules out button colors and word tweaks and rules in whole new pages, a different offer, a different way of presenting price or delivery. A bold change that loses teaches you more than a timid one you can’t read.

Do this

• For each change on your roadmap, write which rung you’ll use to judge it, and for before-and-after changes, the comparison and the weeks. Changes that can’t name a rung are opinions; label them as such in the log.