Quant
The multiple-testing trap and the deflated Sharpe
Someone shows you a backtest with a Sharpe ratio of 2.0. Is it good? You cannot answer — and neither can they — without one more number: how many strategies did you test to find this one? Because the best of many tries is not the best strategy. It's the luckiest one. And luck scales with how hard you looked.
The best of many is not the best
Run one worthless strategy and it scores around zero, give or take. Run two hundred worthless strategies on the same data and take the winner, and the winner will look impressive — not because any of them worked, but because the maximum of two hundred noisy draws is far from the average of them. Selecting the max is the overfitting. You didn't need to fit a single parameter.
That's the whole trap in one picture. The question was never "is 2.0 a high Sharpe?" It's "is 2.0 higher than what my search process would have produced from nothing?" Here it isn't — so the honest conclusion is that you have found precisely no evidence of an edge.
Your trial count is much bigger than you think
Most people, asked how many strategies they tested, say "one — this one." The real count includes:
- Every parameter value in every grid search. Two parameters with twenty values each is 400 trials, not one.
- Every idea you tried and abandoned because it didn't work. Those were tests. They count.
- Every time you re-ran with a different date range, universe, or bar size after seeing a bad result.
- Every filter you added because it "cleaned up" the equity curve.
- The strategies you read about online — which were themselves selected from thousands by other people, and reached you because they backtested well.
A realistic count for an afternoon of research is in the hundreds. Nobody logs this, which is exactly why the literature is full of edges that never repeat.
Flip enough coins and one will come up heads ten times running. It isn't a special coin. If you tried 200 strategies and kept the best, the number you're celebrating is the size of your search, not the size of your edge.
Deflating the number
The fix is to compare your Sharpe against the distribution of the maximum you'd expect from your number of trials, rather than against zero. That's the idea behind the deflated Sharpe ratio (Bailey and López de Prado), which adjusts an observed Sharpe for the number of trials, the length of the sample, and the skew and fat tails of the returns — all of which flatter the raw figure.
You don't have to implement the formula to get the benefit. The intuition is the usable part: the bar rises with the number of things you tried, roughly with the logarithm of the trial count. A Sharpe of 2.0 from one pre-specified hypothesis is interesting. The same 2.0 from a 400-cell grid search is nothing at all.
Practical defence
- Count and log every experiment. If the trial count is unknown, no deflation is possible and no claim is meaningful. This is the single highest-leverage habit in research.
- Decide the rule before you look. Write down what you're testing and what would falsify it, then run it. One pre-registered test beats a thousand exploratory ones.
- Keep a holdout you touch once. The moment you look twice, it's training data — you've started fitting to it with your own decisions.
- Prefer few, theory-motivated tests. An idea with an economic reason behind it needs less evidence than one found by grid search, because you weren't searching.
- Look for plateaus, not peaks. A peak is what a lucky trial looks like; a plateau is much harder for noise to fake.
This is the discipline the rest of the toolkit is built on. Walk-forward and Monte Carlo tell you whether a strategy is fragile; the trial count tells you whether it was ever a finding in the first place. And it's the sharpest form of the random walk's lesson: randomness produces gorgeous results for free, and the more you search, the more of them it hands you.