Skill repeats and luck doesn’t. So the test for skill is whether a ranking holds from one period to the next.
Michael Mauboussin’s The Success Equation (2012) starts from a simple model: every result is part skill and part luck. You can’t see the parts in one result. You can see them across results, because skill carries forward and luck doesn’t.
In a 2012 Harvard Business Review article, Mauboussin set out what makes a statistic useful. One test is that it’s persistent: the same action produces a similar outcome from one time to the next Published. Apply that to anything you’re judging on results: channels, creatives, agencies, creators, people.
Say you ran ten creators last quarter and ten again this quarter, the same ten. Rank them by cost per order in each quarter and compare the two rankings. If the top three last quarter are near the top again, results are persistent and the ranking reflects something real. If last quarter’s top three are scattered through this quarter’s list, the ranking was mostly luck. Spreadsheets do this in one line: the correlation between the two columns of ranks.
That correlation does double duty. It’s also roughly the share of a winner’s lead you should expect next period, which is the same shrinkage as chapter 5‘s tool, measured a different way Derived.
A ranking that doesn’t repeat isn’t a ranking of skill.
Mauboussin’s paradox of skill: as people get better at an activity, the gaps between the best, the average and the worst become “much narrower” Published. When everyone is skilled, luck decides more of the ranking. I’d expect paid social to work this way. When every brand in a category uses the same ad platform, the same creative formats and similar agencies, the difference in skill between them shrinks, and the week-to-week ranking of who’s winning becomes noisier. The edge moves to the places competitors can’t copy quickly: the product, the offer, the retention program.
Mauboussin put the sports version plainly: a player with a great statistical year was almost always “skillful and lucky” Reported. The same holds for a founder who had one great year, or an agency whose case study features its best client. Both parts were there. Only one of them is for hire.
Results over one quarter are a poor measure of a person whose results depend on campaigns, markets and budgets they don’t control. Judge on two things instead: results over several periods, and the process measures they do control, like whether tests were set up properly, whether launches shipped on time, whether the scorecard was kept. Process measures tend to persist, which is why they’re fairer.
This is one chapter of The Noise Floor, which is free and readable in full on a single page with no form in front of it.