Check the split, check the size, read the range, then ask whether the story holds. In that order, every time.
A test result is only as good as the machinery that produced it. Two checks decide whether there’s a result at all. Two more decide how much to believe it.
If you asked for 50/50 and got 52/48 on a big test, something is broken. Aleksander Fabijan and colleagues found this problem, called a sample ratio mismatch, in about 6% of experiments at Microsoft and about 10% of analyses at LinkedIn. It “in most cases completely invalidates experiment results” Published, because whatever pushed visitors out of one version usually pushed out a particular kind of visitor. Microsoft flags a mismatch when a standard chi-square test on the counts gives a p-value below 0.0005 Published. On a store, the usual causes are a version that redirects to a new page and loses people during the extra load, bot filtering that treats the versions differently, and a script that fails to fire in one version.
Did the test reach the size planned before launch, and run whole weeks? If not, it isn’t finished, whatever the dashboard says. Never stop early because it looks like a win, unless your tool uses a sequential method designed for it and you know that it does. Stopping early because the split is broken or a guardrail is in trouble is fine.
Read the confidence interval, not the single number. A lift of 11% with a range from 1% to 22% says the change probably helps, by an amount you don’t know well. Plan on the low end. A range that includes zero says you can’t tell, which isn’t the same as saying there’s no difference.
Does the result make sense, and does it hold within the test? Look at it week by week. A lift that’s big in week one and gone by week three may be novelty: Kohavi and colleagues describe returning users who “investigate the new feature, click everywhere, and thus introduce a ‘novelty’ bias that dies quickly” Published. Most store visitors are new, so novelty matters less on a store than on a product people use daily, but check anyway. Check guardrails: order value, returns, contribution. And if the result is surprising, run it again. The 2022 paper recommends a stricter threshold, 0.01 or 0.005, before believing surprising results Published.
Check the split before the result. A broken split means there is no result.
From my workOne more check belongs before the test starts: is the baseline clean? On one program, a broken product block was dragging down the numbers of the very thing we planned to redesign. Fixing the bug and testing the redesign at the same time would have credited the redesign with the fix. The rule I hold to: fix first, in both versions, then give the page two to three weeks so you know its new conversion rate before you size the test. Never let a repair ride along in only one version, and never plan a test on a baseline you haven’t measured.
Before you trust a testing setup, test it against itself. Split traffic between two identical versions for two weeks. You should see an even split, and usually no winner. A single A/A winner happens one time in twenty by chance, so rerun it. If it wins again, or if the split fails the check, fix the setup before running anything that matters.
This is one chapter of The Honest Test, which is free and readable in full on a single page with no form in front of it.