INKWIRE(7) argument errors INKWIRE(7)
base-rate
Base Rate Neglect
A vivid new detail outweighs the much larger number it should be measured against.
NAME
Definition
Base rate neglect is treating a diagnostic fact as if it settled a question, while ignoring how rare or common the thing being diagnosed actually is. The specific evidence is real; the error is reading it without its denominator.
The standard illustration is a medical test. A disease affects 1 in 1,000 people. The test is 99 per cent sensitive and has a 5 per cent false-positive rate. You test positive. The chance you have the disease is not 95 per cent — it is about 2 per cent, because out of 1,000 people about 50 will be false positives and only 1 will be a true positive.
The same arithmetic governs hiring signals, fraud alerts, content moderation flags and security warnings. The rarer the condition, the more a positive result is made of false positives.
SIGNALS
-
S1
A single diagnostic fact is offered as decisive
"The test was positive" or "the profile matches" is treated as the answer, with no mention of how many people with that result do not have the condition.
-
S2
Sensitivity is quoted, prevalence is not
Statistics about how well the detector performs appear; statistics about how common the target is do not.
-
S3
The story is about a case, not a rate
One compelling individual case substitutes for the population number it would need to be compared against.
Three checks, in order
- Ask for the prevalence: out of everyone tested, what fraction actually has the condition?
- Build the 1,000-person table. Fill in true positives, false positives, true negatives and false negatives before reading any percentage.
- Check the direction of rarity: the rarer the condition, the lower the share of positives that are real — this is counter-intuitive and worth doing by hand once.
Where the link breaks
| The intuitive reading | What the denominator says |
|---|---|
| A 99 per cent accurate test means a positive result is 99 per cent certain | Prevalence decides; at 1 in 1,000 the same test gives about 2 per cent |
| Most flagged accounts must be guilty | At a low base rate, most flags are errors even with a good detector |
| The profile matches the typical offender | Typical offenders are also typical of almost everyone else when the offence is rare |
EXAMPLES
The detector flagged 200 accounts and is 98 per cent accurate, so roughly 196 of them are fraudulent.
Why it fails
If fraud is present in 0.5 per cent of accounts, then in 100,000 accounts there are 500 real frauds and 99,500 clean ones. At 98 per cent accuracy the detector catches about 490 frauds and, more importantly, misflags about 2 per cent of 99,500 — roughly 1,990 innocent accounts. Of the 2,480 flags, only about 20 per cent are real.
Kindred entries
- [F] survivorship You only studied the ones that made it, then asked what they have in common.
- [F] texas-sharpshooter Shoot the barn, then draw the target around the tightest cluster of holes.
- index — /t/fallacy/
[+] Notes and objections 3
- Why is the true answer so much lower than intuition?
- Because the healthy group is enormous. A small error rate applied to a huge group produces a large number of errors, and those errors can outnumber the genuine cases entirely when the condition is rare.
- Does a better test fix it?
- Only up to a point. To make a positive result more likely true than false when prevalence is 1 in 1,000 you need a false-positive rate below roughly 0.1 per cent. Screening for rare conditions is dominated by specificity, not sensitivity.
- Is this the same as the prosecutor's fallacy?
- They are close relatives. The prosecutor's fallacy swaps the probability of the evidence given innocence for the probability of innocence given the evidence; base rate neglect is the broader habit of leaving the prevalence out of the calculation altogether.
Scope — Reference material for reading arguments, not professional advice.