Experimentation and statistics (chapters 2 to 5, 13, 14)

• Ron Kohavi, Diane Tang and Ya Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapter 1 excerpt.
• Ron Kohavi, Alex Deng and Lukas Vermeer. “A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments.” KDD 2022. Full text.
• Ron Kohavi, Thomas Crook and Roger Longbotham. “Online Experimentation at Microsoft.” Third Workshop on Data Mining Case Studies, 2009.
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne. “Controlled experiments on the web: survey and practical guide.” Data Mining and Knowledge Discovery 18(1), 2009.
• Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu. “Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained.” KDD 2012.
• Ron Kohavi, Alex Deng, Roger Longbotham and Ya Xu. “Seven Rules of Thumb for Web Site Experimenters.” KDD 2014.
• Ron Kohavi and Stefan Thomke. “The Surprising Power of Online Experiments.” Harvard Business Review, September–October 2017.
• Stefan Thomke. “Building a Culture of Experimentation.” Harvard Business Review, March–April 2020.
• Robert Lehr. “Sixteen S-squared over D-squared: a relation for crude sample size estimates.” Statistics in Medicine 11(8), 1992. Gerald van Belle, Statistical Rules of Thumb, second edition, Wiley, 2008.
• Aleksander Fabijan, Jayant Gupchup, Somit Gupta, Jeff Omhover, Wen Qin, Lukas Vermeer and Pavel Dmitriev. “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments.” KDD 2019. Microsoft Research, “Diagnosing Sample Ratio Mismatch in A/B Testing,” 2020.
• Ramesh Johari, Pete Koomen, Leonid Pekelis and David Walsh. “Always Valid Inference: Continuous Monitoring of A/B Tests.” Operations Research 70(3), 2022.
• Evan Miller. “How Not To Run an A/B Test.” 2010.
• Martin Goodson. “Most Winning A/B Test Results Are Illusory.” Qubit, 2014.
• Leonid Pekelis. “Statistics for the Internet Age: The Story Behind Optimizely’s New Stats Engine.” Optimizely, January 2015. VWO, how VWO calculates a winning variation and Enhanced SmartStats, December 2024.
• David Robinson. “Is Bayesian A/B Testing Immune to Peeking? Not Exactly.” 2015.
• Alex Deng, Ya Xu, Ron Kohavi and Toby Walker. “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data.” WSDM 2013.
• Randall A. Lewis and Justin M. Rao. “The Unfavorable Economics of Measuring the Returns to Advertising.” Quarterly Journal of Economics 130(4), 2015.
• Dan Siroker. “How Obama Raised $60 Million by Running a Simple Experiment.” 2010, republished by Optimizely.

Research methods (chapters 6, 7)

• Jakob Nielsen and Thomas K. Landauer. “A mathematical model of the finding of usability problems.” INTERCHI 1993. Jakob Nielsen, “Why You Only Need to Test with 5 Users,” 2000.
• Laura Faulkner. “Beyond the five-user assumption.” Behavior Research Methods, Instruments, and Computers 35(3), 2003.
• Nielsen Norman Group. How to conduct a heuristic evaluation; A/B testing 101, 2024.
• CXL. On qualitative customer surveys.

The page, mobile and speed (chapters 8 to 10, 12)

• Baymard Institute. Cart abandonment rate and reasons, updated 2025; current state of checkout UX, 2025; current state of product page UX, March 2026; mobile UX, July 2026; ecommerce search, 2026.
• Contentsquare. 2026 Digital Experience Benchmark. Littledata, Shopify conversion rate benchmark, 2023.
• Tom Collinger, Edward Malthouse, Ewa Maslowska and Yorgos Askalidis. “How Online Reviews Influence Sales.” Spiegel Research Center, Northwestern University, 2017.
• Sheena Iyengar and Mark Lepper. “When Choice Is Demotivating.” Journal of Personality and Social Psychology 79(6), 2000. Benjamin Scheibehenne, Rainer Greifeneder and Peter Todd, “Can There Ever Be Too Many Options?” Journal of Consumer Research 37(3), 2010. Alexander Chernev, Ulf Böckenholt and Joseph Goodman, “Choice overload: A conceptual review and meta-analysis.” Journal of Consumer Psychology 25(2), 2015.
• Google. Core Web Vitals; INP becomes a Core Web Vital, 2024; Vodafone case study, 2021; Milliseconds Make Millions, 2020; avoid intrusive interstitials.
• Shopify Help Center. Improving web performance.
• Omnisend. Pop-up statistics, February 2026. Octane AI, site claims, read September 2026. Econsultancy on site search, 2013; Adobe on site search, 2015.

Shipping and price (chapter 11)

• Michael Lewis, Vishal Singh and Scott Fay. “An Empirical Study of the Impact of Nonlinear Shipping and Handling Fees on Purchase Incidence and Expenditure Decisions.” Marketing Science 25(1), 2006.
• Edlira Shehu, Dominik Papies and Scott A. Neslin. “Free Shipping Promotions and Product Returns.” Journal of Marketing Research 57(4), 2020.
• Amazon. Statement regarding random price testing, September 27, 2000.
• New York General Business Law § 349-a. Federal Trade Commission, surveillance pricing study, January 2025.
• Intelligems. Price testing documentation, read September 2026.