In December 2007, Dan Siroker, then working on Barack Obama’s presidential campaign and later a co-founder of the testing company Optimizely, ran a test on the campaign’s splash page, the page that asked visitors to sign up with their email address before entering the site. He wrote it up in 2010 on Optimizely’s blog, and the story has been told in marketing talks ever since.

What was tested

Two parts of the page changed at once. Four versions of the button text (“Sign Up,” “Learn More,” “Join Us Now” and “Sign Up Now”) were crossed with six pieces of media: three photos and three videos. That’s 24 combinations, shown to 310,382 visitors. The winner, a “Learn More” button with a family photo, raised the sign-up rate from 8.26% to 11.6%, a lift of 40.6%. And every video did worse than every photo Reported.

Why it was a good test

• The traffic fit the question. Split 24 ways, each combination saw about 12,900 visitors and about 1,070 sign-ups, enough to see a difference of about 12% between two combinations, or about 16% once you allow for comparing 23 combinations with the original Derived. The winning lift was more than twice that.
• The changes were bold. Not shades of a button, but different media and different asks. Bold changes are the only kind a limited amount of traffic can read.
• It measured the outcome that mattered. Sign-ups, the thing the page existed for, not clicks on a video or time on page.
• It was on the busiest page. Most visitors met the splash page first, so the test borrowed most of the site’s traffic.

What it didn’t measure

The post’s title says the test raised $60 million. It didn’t measure that. Siroker estimated that about 10 million people signed up over the campaign, assumed the lift held the whole time, and calculated 2.88 million extra email addresses. He then assumed 10% of sign-ups volunteered, for 288,000 extra volunteers, and that each address was worth an average of $21 in donations, for $60 million Reported. Each step is reasonable. None was measured by the test, and the post reports no significance test or range for the lift itself. With 24 combinations, the best one also benefits from the winner’s curse in chapter 4: picking the top of 24 flatters it.

Keep the measurement and the extrapolation in separate sentences. The first is evidence. The second is a forecast.

That’s not a criticism of the test, which was well designed and clearly worth running. It’s a lesson in how results travel. A measured 40.6% lift on sign-ups became “$60 million” in the headline, and the headline is what people remember. In your own reports, write the measured number first, with its range, and put the projection after it with each assumption named.

Do this

• Find your splash page: the page most of your visitors pass through. Usually it’s the home page or your best-selling product page. Plan the one bold test it can carry this quarter with the planner in chapter 3, and write the report’s first sentence before the test starts: “We measured ___ on ___, with a range of ___.”