Quant
Survivorship bias — backtesting on a graveyard
Before your code has a chance to be wrong, your data already is. The most expensive backtest bugs aren't in the strategy logic at all — they're baked into the file you downloaded. And the oldest of them is survivorship bias: a dataset that quietly forgot everything that died.
The bombers that came back
The classic version comes from the Second World War. Analysts studied returning bombers, mapped where they were riddled with bullet holes, and proposed armouring those areas. The statistician Abraham Wald pointed out the flaw: they were only looking at the planes that made it home. The holes showed where a bomber could be shot and still fly. The armour belonged where the returning planes had no holes — because the planes hit there never came back to be counted.
Your ticker list is a hangar full of returning bombers.
What it does to a backtest
Say you test "buy the dip on large-cap stocks" using today's index constituents. Every name in that list is there because it succeeded. The companies that dipped and never recovered — the ones that would have turned your dip-buy into a permanent loss — aren't in the list to be bought. Your strategy gets to buy dips with the benefit of knowing, in advance, that the company survives.
The same trap runs through every asset class. Crypto backtests on today's exchange pairs miss the thousands of coins that went to zero and got delisted. Fund performance studies flatter the industry because closed funds stop reporting. Any dataset assembled by asking "what exists now?" and walking backwards has this baked in.
It isn't a rounding error, either — it's one of the largest and best-documented biases in published backtests, and it works in exactly the direction that makes you want to trade.
Your data only contains the companies that survived. A strategy tested on it is picking from a list of known winners — which is why it looks brilliant and why it won't work on the future, where the losers haven't been removed yet.
Its close cousin: point-in-time data
Survivorship is one case of a bigger question: what did I actually know on that date? The honest answer is usually less than your file suggests.
- Index membership changes. A stock added to the index in 2022 was not in it in 2016 — but today's list says it was.
- Fundamentals get restated. The earnings figure sitting in your database for Q1 is the corrected one, published months later. The number available at the time was different, sometimes very different.
- Prices get adjusted retroactively for splits and dividends. Useful for returns, quietly wrong if your logic keys off actual traded price levels.
Each of these hands your strategy a small piece of the future. That's look-ahead bias, and survivorship is simply its most structural form — the future leaked in before you wrote a line of code.
What to actually do
The fix starts with a blunt test: search your dataset for a name that went to zero. If you can't find one — not one bankruptcy, not one delisting, not one dead pair — you are not testing on history. You're testing on a highlight reel.
From there: use a data provider that explicitly includes delisted securities and states its point-in-time policy (this is precisely what you pay for). Reconstruct the universe as of each date rather than filtering with today's list. And when you can't get clean data, at least know the direction of the lie — survivorship always flatters, so treat the result as an optimistic ceiling, never an estimate.
It's the same discipline as the rest of honest testing: your default assumption is that the impressive number is an artefact, and the burden of proof is on the edge. A backtest can only ever be as truthful as the history you feed it — and against the random walk, a dataset that has already deleted its own failures is not history at all.