Your flow report credits each order to the message that sat closest to it. Use it to rank flows; it proves nothing about growth. Proof takes a holdout. A holdout only works under three rules nobody writes down: write the random number once, let nothing overwrite it, and read it across the whole program. Then comes the limit: on a small file, a holdout can only confirm a large effect.

Attribution answers a bookkeeping question: which touch sat closest to the order. The business question is whether the order would have happened without that touch. A customer running low on moisturizer walks toward checkout on a schedule set by the bathroom cabinet. The replenishment email in front of them collects credit for an order already on its way. Attributed revenue is a program grading its own homework.

The error grows as targeting improves, because a flow aimed at likely buyers claims orders they would have placed anyway. In accounts I've audited, the channels' attributed totals often add up to more than the store sold. That makes attributed revenue unfit for a funding decision. It still works for an operating one. There a rough ranking is enough, and a wrong call costs one rewrite.

Attributed revenue tells you which flow to rewrite next; only a holdout big enough to read tells you whether the program grew the business.

Rank on revenue per recipient

Divide each flow's attributed revenue by the people it reached. Totals reward whatever you sent to the most people. A big list can hide a weak flow for years. At a promotions-led brand, a welcome-tail flow had sent 27,326 emails and looked busy. Those sends produced $237, or $0.009 per recipient. Per-recipient ranking puts that flow at the bottom of the list; a total hides it.

Don't let that ranking settle a test. One large order can swing revenue per recipient in a small arm and crown the wrong variant. Score a test on placed-order rate. Report revenue per recipient beside it, with the largest orders capped so a single big order can't pick the winner.

The Holdout Digit

A holdout is a randomly chosen group that gets no marketing at all. It turns your program into a randomized experiment. Because assignment is random, both groups get the same product mix, the same season and the same paid spend. Any gap in purchase rate between them is what the program caused, whichever tool claims the click.

Assign the group with the Holdout Digit. It's a random number from 0 to 99, written once to every profile at creation and never recomputed. Buckets 0–4 are your holdout. Store it in a profile property called Holdout Bucket and hold it to three rules.

1. Keep it stableWrite the digit once, when the profile is created. Put the holdout range in the runbook. Changing the range mid-read spoils every day of data before the change.
2. Guard the writeNothing overwrites it. Write only when the field is empty, or derive the digit from a hash of an immutable ID, so nothing is written and nothing can be reshuffled. A sync that recomputes it reshuffles your control group and leaves no trace.
3. Read it pooledRead the holdout across the whole program, not per flow. Split a small control group five ways and you get five unreadable answers instead of one readable one. The budget covers the program. Nobody funds a winback flow.

Apply it everywhere

Define the holdout once, as one segment (Holdout Bucket 0–4), and apply it everywhere: a filter in every marketing flow, an exclusion on every campaign, and a synced exclusion in your SMS platform and paid audiences. Klaviyo won't apply it for you.

Then build the leak check: a segment of holdout profiles who received any marketing email or text in the last 30 days. It should be empty. Read it every Monday. One forgotten exclusion gives the holdout some marketing, and the read then understates what the program did.

Never withhold a message the law requires: renewal reminders, price-change notices, order and safety notices.

Size it before you pick a percentage

Work out what your file can detect first, then choose the percentage. Read purchase rate, not revenue. A purchase rate counts each buyer once, while a revenue average moves with one big order.

Take an invented file: 10,000 contactable profiles, a 10% holdout and an 8% ninety-day purchase rate. At 80% power and a two-sided 95% test, the smallest difference you can reliably see is about 2.5 points. That's a lift of roughly a third, which is bigger than most programs' true effect.

So on a small file, hold out 15–20% if you can afford it, read at 180 days, and say in writing that the holdout can confirm a large program, not measure a modest one. Five percent starts to work near 100,000 contactable profiles.

Write the read date down before launch and agree it with whoever will read the result. Checking early and often turns a null result into a false positive, because every extra look gives noise another chance to cross the line.

What it costs

A 5–10% holdout costs a few percent of flow revenue for six months. Skipping it means you can never say the program earned anything, including to whoever decides whether to keep paying for it. Bring the read to The Budget Ask. Finance can check a randomized comparison the way it checks sales.

There's no holdout result from my own work in this book. That is the gap this chapter exists to close, in your program and in mine.

Do this

• Write the Holdout Digit to every profile this week.
• Build the leak-check segment and put it on Monday's scorecard.