The Shelf

Trustworthy Online Controlled ExperimentsRon Kohavi, Diane Tang and Ya Xu, 2020The manual. Everything in part one is here in more depth, from people who ran testing at Microsoft, Google and LinkedIn.
Experimentation WorksStefan Thomke, 2020How companies like Booking.com built a culture of testing, and what it asks of leaders.
Rocket Surgery Made EasySteve Krug, 2009A short, practical guide to running your own usability sessions, the method in chapter 7.
Don’t Make Me Think, RevisitedSteve Krug, third edition, 2014The classic on why people get stuck on websites, and still the fastest way to see your own site as a stranger does.
UncontrolledJim Manzi, 2012On why controlled experiments beat intuition in business, and how rarely ideas work.
Statistical Rules of ThumbGerald van Belle, second edition 2008Where the sample-size rule in chapter 3 lives, with dozens more like it.

And the research: Kohavi, Deng and Vermeer (2022) on false winners; Fabijan and colleagues (2019) on broken splits; Johari and colleagues (2022) on peeking; Deng, Xu, Kohavi and Walker (2013) on variance reduction; Nielsen and Landauer (1993) and Faulkner (2003) on how many users to test; Lewis, Singh and Fay (2006) and Shehu, Papies and Neslin (2020) on free shipping; Iyengar and Lepper (2000), Scheibehenne and colleagues (2010) and Chernev and colleagues (2015) on choice. Full references are in Appendix C.

If this is your store

See where your first-time buyers stall.

The free Klaviyo audit reads your own account and scores it. Start with your store’s address. A read-only key gets the full report.

Get the free audit →

Written by Andrew Lauchner, a growth and retention operator for consumer brands. The paid work is one ninety-day Sprint.

This is one chapter of The Honest Test, which is free and readable in full on a single page with no form in front of it.

See where your first-time buyers stall.