Every test starts with someone who believes in the idea. Ron Kohavi, who ran experimentation at Microsoft and later at Airbnb, spent two decades counting how often that belief is right. The answer is the most useful number in conversion work.

The rates

PublishedKohavi, Crook and Longbotham, 2009; Kohavi and colleagues, 2012, 2014 and 2022, the last citing Stefan Thomke for Booking.com and Jim Manzi for Google. Full references in Appendix C.

These are products tuned by thousands of engineers, so the easy wins are long gone, and a small DTC store can reasonably expect a higher hit rate on its first ideas. But the pattern holds: most ideas won’t work. At Microsoft, the two-thirds of ideas that didn’t improve their metric either made no measurable difference or made things worse. Nobody could reliably tell which ideas would work before testing them.

Nobody can pick the winners

The clearest example comes from Bing. An engineer suggested showing more of an ad’s text in its headline. It was a small change, rated as low priority, and it sat in the backlog for more than six months. When someone finally tested it, revenue rose 12%, worth more than $100 million a year in the US Published. Kohavi and Stefan Thomke later called it the best revenue-generating idea in Bing’s history. Nobody had thought it would matter.

Your confidence in an idea tells you almost nothing about whether it will work. That’s why you test, and why the test has to be one you can believe.

What this means for a small store

• Expect most tests to lose. If more than half your tests are “winning,” something is wrong with how you’re reading them. Chapter 4 covers what.
• A loss is worth something. Every change you didn’t ship because a test caught it is a small disaster avoided. Two-thirds of Microsoft’s ideas would have shipped on opinion alone.
• Where ideas come from matters more than how many you test. If you can only run a handful of real tests a year, the hit rate of your ideas decides whether you find anything. In my experience, ideas drawn from watching customers struggle win more often than ideas copied from a competitor’s homepage. Chapter 7 is about finding them.

Do this

• List your last ten tests or major site changes and mark each as won, lost or no difference. If more than half won, keep that list open while you read the next three chapters.