A test that won without a written reason teaches you to repeat a result you don’t understand. A test that lost with one teaches you something about your customers you’ll use for years. The brief is where the reason goes.

The brief

Fill it in before anything launches. The Whole Machine has a shorter version; this one adds the decisions that part one of this guide showed are easy to get wrong.

1. Because we sawThe evidence from research: sessions, survey answers, the funnel. “Three of five people couldn’t find the delivery date.”
2. We believeThe change, and who sees it.
3. Primary metricOne. Usually orders per visitor, or contribution per visitor for anything touching price or shipping.
4. GuardrailsWhat mustn’t get worse: order value, returns, contribution, page speed.
5. Smallest lift worth acting onAnd the planner’s answer: visitors per version, weeks, and the power you’ll have.
6. The rungA/B test, holdout, on-off weeks or before-and-after, from chapter 6.
7. Decided in advanceWhat you’ll do if it wins, if it loses, and if it can’t tell. Writing the third answer down is what stops a team from rerunning the same test until it wins.

Decide what you’ll do with every possible result before you see any of them.

The log

One row per test and per major site change, in a shared sheet anyone can search: the date, the page, the brief, the rung, the planned and actual size, the result with its range, and what changed because of it. Before anyone proposes a test, they search the log. A year in, the log is the most valuable document in your conversion program, because it’s the only record of what your changes did, as opposed to what someone remembers they did.

How many tests

A page that needs four weeks per test can carry at most thirteen tests a year, one after another, and fewer once you allow for launches and sales. Tests on different pages can run at the same time, as long as each assigns visitors at random on its own and the two changes can’t clash (two tests both changing the delivery message, say). Counted against the traffic table, most DTC stores I see have room for somewhere between two and twenty real tests a year. Spend them on bold changes drawn from research, not on small ideas drawn from opinion.

From my workFor experiences a testing tool doesn’t reach, like email flows and post-purchase pages, I stamp every customer with a random digit from 0 to 9 when they’re first recorded, and never change it. Holding out digit 0 gives a standing 10% control group, with no setup per test. It measures everything you’ve shipped to the other 90% together, so read it as the combined effect of your program. To read one change on its own, give it its own random split.

Do this

• Start the log today with your last five tests and your last five major site changes. Fill in what you can. The blanks show you which parts of the brief your team has been skipping.