The test brief

BECAUSE WE SAW     evidence from research (sessions, survey, funnel):
WE BELIEVE         the change, and who sees it:
PRIMARY METRIC     one:
GUARDRAILS         order value / returns / contribution / speed:
SMALLEST LIFT      worth acting on:        %
PLANNED SIZE       visitors per version:        weeks:        power:
RUNG               A/B / holdout / on-off weeks / before-and-after
IF IT WINS         we will:
IF IT LOSES        we will:
IF IT CAN'T TELL   we will:
OWNER              name, and the date it will be read:

The session script

BEFORE    "We're testing the site, not you. Please think out loud.
           There are no wrong answers, and I won't be able to help."
TASK 1    "You're shopping for [need]. Find something you'd buy."
TASK 2    "Buy it. Stop when you'd have to pay."
TASK 3    "Before buying, find out when it would arrive and how
           returns work."
AFTER     "What was the most confusing moment?"
          "What almost made you leave?"
NOTE      Every hesitation over five seconds, every tap on
          something that isn't a link, every question asked aloud.

The post-purchase question

ON THE CONFIRMATION PAGE, OR THE FIRST EMAIL AFTER PURCHASE
One question, free text:
  "What almost stopped you from buying today?"
Optional second question, multiple choice:
  "Where did you first hear about us?"
Read every answer weekly. Group them. Count the groups.

The test log

DATE | PAGE | CHANGE | RUNG | BRIEF LINK | PLANNED SIZE
     | ACTUAL SIZE | SPLIT CHECK | RESULT (WITH RANGE)
     | DECISION | WHAT WE LEARNED | OWNER

The before-and-after record

CHANGE              what, where, live date:
METRIC              one, decided before the change:
BEFORE WEEKS        whole weeks, no promotions:
AFTER WEEKS         same length, no promotions:
COMPARISON          what the change didn't touch (other device, similar
                    products):
TRAFFIC MIX CHECK   share by channel, before vs after:
LAST YEAR           same weeks' change last year:
RESULT              change relative to the comparison (ratio), and the
                    same figure for past periods with no change: