Your cross-sell flow reports every purchase by someone who got the message, within an attribution window. Some bought because of it. If the Amazon estimate for recommendation clicks in chapter 7 carries over, many would have bought anyway. The only way to know the split is to hold some customers back.

How a holdout works

Take everyone who qualifies for the flow, say everyone who has placed a second core order, and randomly assign a fixed share, often 10%, to get nothing from it. They still get newsletters and core flows. After 60 to 90 days, compare the groups on three numbers, counted from the day each customer qualified, opened or not:

• Share who bought the new category, the number the flow was built to move.
• Total contribution per customer, the number that matters. A customer who buys the new category instead of their usual core reorder adds nothing.
• Unsubscribes and complaints, the cost you don’t see in revenue.

Say 40,000 customers qualify and 4,000 are held out. In 90 days, 6.0% of the customers who got the flow buy the new category, against 4.5% of the holdout. Attribution would credit the flow with about 2,160 buyers, 6.0% of 36,000. The holdout says it caused about 540, 1.5 points on 36,000: a quarter of what the dashboard shows Derived. At these sizes the difference is about four times its standard error, well clear of noise Derived. With a smaller list, work out first whether the test can see the lift you care about; the sample-size math is in The Honest Test.

Attributed revenue is what the flow was near. Incremental revenue is what it did.

What to test

• Timing. After the first, second or third core order. In field experiments at two business-to-business firms, V. Kumar, Rajkumar Venkatesan and Werner Reinartz timed sales calls to each customer’s predicted purchase timing and needs, and the incremental profit came to more than $1 million Published.
• No discount against a discount. Expect the discount to win on buyers and often lose on contribution, for the reasons in chapter 8.
• Who gets it. Exclude unprofitable cross-buyers, and read heavy and light core buyers separately: heavy buyers cross-buy more with or without the nudge.

Bundles and cart add-ons are the same question at checkout: they need a holdout too, and they’re covered in The First Offer.

Keep a standing holdout

A small permanent holdout, 5% of new customers who never get cross-category messages, lets you check every quarter whether the whole cross-sell program still adds contribution. It costs a little revenue, and it’s the one cross-sell number nobody can argue with.

Do this

• Add a 10% random holdout to your main cross-category flow this week, split at the moment customers qualify. Put a date 90 days out in the calendar to read it with the query in Appendix A.