FIX ONE STRETCH AT A TIME

By now you’ll have more fixes than time, and every one has an advocate. The ranking turns the argument into arithmetic, and the holdout tells you afterward whether the arithmetic was right.

The ranking

For each candidate fix, estimate five things:

• Reach: customers entering the stage each month.
• Lift: the change in the stage’s conversion you expect, in points.
• Value: what one extra customer passing the stage is worth. From the chapter 4 defaults, an extra customer reaching first use is worth about $24 (20% of them go on to a third order worth $120) and one reaching the second order about $69 Derived.
• Confidence: how likely the fix is to work at all.
• Cost: everything for the first year: build, tools, discounts, time.

Expected value for a year is reach × 12 × lift × value × confidence. Divide by cost and rank. Because it’s in money, not points, it can be checked against what happened.

Setting confidence on a fixed scale

Confidence is where rankings get rigged, because the advocate sets it. Use a fixed scale: 80% for a fix that already worked in a holdout elsewhere in your own business; 50% for one with good outside evidence; 20% to 30% for a reasonable hunch. Never go above 80%. If one fix only ranks first at 90%, it doesn’t rank first.

With the defaults, the delivery-day how-to email ranks first at 6.4 times its cost, about $25,400 of expected value in its first year. The reorder reminder is close behind at 6.2 times, and it has the larger net value ($31,260 against $21,402), so the ranking can’t separate them. The offer returns 3.6 times. The faster carrier returns less than a fifth of its cost in the first year, because a one-point gain at a stage that already passes 98% of customers is worth little further down the line Derived.

One stretch, with a holdout

Ship one fix per stage at a time, and keep a random share of customers on the old journey. Two fixes in one stage can’t be told apart, and a fix read against last month mostly measures the calendar. The Honest Test covers holdouts and reading them; The Noise Floor covers how big a change has to be before it’s real.

Size it first. To see first-use share move from 61% to 64%, you need roughly 4,200 customers in each group Derived, from the rule of thumb 16 × p(1 − p) ÷ d² at 80% power and 5% significance. With 4,900 delivered customers a month split evenly, both groups fill in under two months; a 10% holdout would take more than eight. For small effects, split evenly.

Do this

• List every fix anyone has proposed for your stall and put the top four into the tool, with confidence set on the fixed scale. Ship the first one with a holdout sized as above, and write down the result that would make you stop.