Registered before launch · published at day 90
The holdout goes on the record before anyone knows the answer.
Most results in this business are reconstructed after the fact, from whichever window looks best. Here, each Sprint’s test is written down before launch: who is held back, for how long and what counts as a win. The result follows at day 90.
The first founding Sprints register here on the day they start. Founding brands publish their result, whatever it shows.
Before launch
What gets registered.
Who is held back
First-time buyers assigned at random, the share held back, and a rule that the assignment is written once and never overwritten.
Start and read dates
The day the program goes live and the day the result is read, fixed in advance so nobody gets to pick the flattering week.
What counts as a win
The primary number, usually the second-order rate among first-time buyers, and exactly how it is counted.
The smallest lift it can see
With the sample the brand has, the smallest difference the test could detect. Anything below it is reported as noise.
At day 90
What gets published.
If the lift is small, it goes up small. If there is no lift, it goes up as no lift.
- 01The baseline rate, and the rate in each group.
- 02The difference, stated as relative and in points.
- 03How sure we can be, and the size of each group.
- 04What else changed during the window.
Price moves, launches, a big promotion: anything that touched the result.
- 05What happens next.
The register
Every registered holdout.
| # | Brand | Registered | Split | Window | Primary measure | Result |
|---|---|---|---|---|---|---|
| F1 | Founding place 1 | Open. Registers on the day its Sprint starts. | ||||
| F2 | Founding place 2 | Open. | ||||
| F3 | Founding place 3 | Open. | ||||
Brands beyond the founding three can keep their name private; their design and result are still registered here, anonymized.
Why this exists
Because my best number can’t prove itself.
Magic Spoon’s repeat purchase rose 38% over about six months. It is a before-and-after with no control group, and I publish it that way. This register is how every result after it is different.
+38%Repeat purchase over about six months · Magic Spoon
Before and after · no control group · relative or points not yet published