INKWIRE(7) argument errors INKWIRE(7)
survivorship
Survivorship Bias
You only studied the ones that made it, then asked what they have in common.
NAME
Definition
Survivorship bias is reasoning from a sample that was selected by the very outcome you are trying to explain. The failures were removed before you looked, so whatever is common among the survivors looks like a cause of survival.
The classic case is wartime aircraft armour. Abraham Wald, working at the Statistical Research Group in 1943, was asked where to add armour based on the bullet holes in returning bombers. The holes clustered on wings and tail. His answer was to armour the places with no holes, because the planes hit there never came back to be counted.
The bias is not limited to machines. Investment track records, published research, startup advice and gym programmes are all filtered by survival before anyone analyses them.
SIGNALS
-
S1
The sample was filtered by the outcome
Ask how the cases got into your sample. If entering required already having succeeded, every shared feature is contaminated.
-
S2
Cemetery silence
Advice that only quotes practitioners who are still practising is usually missing the larger group who did the same thing and stopped.
-
S3
Common features are promoted to causes
"Every successful founder I interviewed reads forty books a year" describes the filter, not a mechanism. The question is what the non-survivors read.
Three checks, in order
- Write down the selection rule that produced your sample, then name one group the rule excluded.
- Look for the missing denominator: how many started, and how many of them did the same thing you are crediting.
- Ask whether the feature you found is more common in survivors than in the base population, not just common among survivors.
Where the link breaks
| The sample | Who was removed |
|---|---|
| Bombers that returned with bullet holes | Bombers shot down over the target, which are never in the hangar |
| Mutual funds that beat the index for ten years | Funds that closed or merged after bad years and vanish from the league table |
| Published studies showing an effect | Studies that found nothing, which sit in filing cabinets and are never submitted |
EXAMPLES
Of the ten largest companies in the index, nine were founded by people under thirty; therefore starting young is the decisive advantage.
Why it fails
The claim names the winners and never mentions the far larger number of people who started young and produced nothing. Age at founding is common among the successful because young people are common among all founders — the ratio, not the raw count, is what carries information.
Kindred entries
- [F] base-rate A vivid new detail outweighs the much larger number it should be measured against.
- [F] post-hoc B happened after A, therefore A made B happen.
- index — /t/fallacy/
[+] Notes and objections 3
- Why is this one so hard to notice?
- Because the missing cases are missing. Nothing in the room feels absent; there is no blank space where the sunk bombers should be. You have to infer the shape of the gap from the shape of what remains.
- Is it still survivorship bias if the sample is large?
- Yes, and size makes it worse. A large biased sample looks authoritative; a study of 5,000 funded startups is still not a study of startups, because the unfunded ones were never in the frame.
- How do I apply this to advice I read?
- Ask what a loser of the same strategy would have had to do identically. If the losing group's behaviour is indistinguishable from the winning group's, the advice is describing a filter, not a cause.
Scope — Reference material for reading arguments, not professional advice.