Bugs, slowness and hidden information don’t need an experiment. Testing them spends your scarcest resource proving what you already know.
Some changes are uncertain: nobody knows whether a new layout will help. Others aren’t: a button that doesn’t work on some phones is costing you orders, and nobody needs a test to know it. Treating the second kind like the first is one of the most expensive habits in conversion work.
Speed is where the evidence is most often overstated, so sort it. The strongest example is Vodafone’s: a true A/B test in Italy, with half the traffic sent to an optimized landing page. Its main content loaded 31% sooner, by the measure Google calls Largest Contentful Paint, and sales rose 8% Reported. The widely quoted Deloitte study for Google, which found that a 0.1-second improvement went with 8.4% more retail conversions, is observational: it watched speed vary naturally across 37 brands Reported. And the old line that 53% of mobile visits are abandoned after three seconds comes from 2016 Google data with thin methods; it’s too dated and loosely sourced to plan around. Speed matters. Measure your own before and after.
If five of five people trip on it, fix it. Save your traffic for questions you can’t answer by watching.
A few “obvious” fixes have real downsides and deserve a test or a holdout: anything that changes price or what the customer pays for shipping, anything that removes information some shoppers rely on, and anything that trades conversion for margin. Everything else, fix.
And fix before you test anything else on the same page. A test run on top of a broken baseline credits the new version with the repair. Log each fix with its date, so the before-and-after in chapter 6 has something to read.
This is one chapter of The Honest Test, which is free and readable in full on a single page with no form in front of it.