Front

TEN POSITIONS

What this guide argues, and what would prove each claim wrong.

A position says what would prove it wrong. Test each on your own store.

  1. Most ideas don’t move the number they target, even at the best companies.Wrong if more than half of your fully powered tests produced a win that held up when you tested it again.
    Most Ideas Don’t Work
  2. Traffic decides what you can test, before any idea does.Wrong if your store can detect a 5% lift on its product pages in a month.
    How Much Traffic a Test Needs
  3. A winner from an underpowered test is more likely to be noise than a winner from a well-powered one, and its lift is exaggerated either way.Wrong if your winners’ lifts hold at full size when you run them again.
    Why Winners Lie
  4. Stopping a test when it looks good multiplies false winners.Wrong if, across many A/A tests checked daily and stopped at the first significant result, no more than one in twenty finds a winner.
    Why Winners Lie
  5. Check the split before the result.Wrong if none of your past tests had a traffic split off by more than chance allows.
    Before You Believe a Result
  6. When you can’t test, research tells you what to change and a careful before-and-after tells you whether it helped.Wrong if five customer sessions on your product page find nothing you didn’t already know.
    The Ladder of Evidence
  7. Bugs, slowness and missing information get fixed, not tested.Wrong if fixing known checkout bugs has, on balance, lowered your checkout completion rate.
    Fix It, Don’t Test It
  8. Most product pages lose sales by hiding answers: the total cost, the delivery date, the returns, the reviews.Wrong if your post-purchase survey’s “what almost stopped you” answers are about something else.
    The Product Page
  9. Judge conversion changes on contribution, not conversion rate.Wrong if your free-shipping threshold or discount test raised conversion and margin every time.
    Shipping, Offers and Price Tests
  10. A test that wasn’t written down before it ran teaches nothing, even when it wins.Wrong if your team can say what it learned from its last ten tests without looking them up.
    The Brief and the Log

This is one chapter of The Honest Test, which is free and readable in full on a single page with no form in front of it.