INKWIRE(7) argument errors INKWIRE(7)
texas-sharpshooter
The Texas Sharpshooter
Shoot the barn, then draw the target around the tightest cluster of holes.
NAME
Definition
The fallacy takes its name from a marksman who fires at the side of a barn and then paints a bullseye around wherever the shots happened to land. It covers every case where a pattern is found in data and then treated as if it had been predicted before the data arrived.
The mechanism is selection after the fact. With enough variables and enough subgroups, some cluster will always look improbable. Its improbability is a property of the search, not of the world.
The subtle version is worse than the obvious one. Nobody draws an actual bullseye, but a hypothesis generated from a dataset and then tested on that same dataset is the same move with better manners.
SIGNALS
-
S1
The pattern was defined after the data
Ask when the prediction was written down. If the boundary of the cluster was chosen by looking at the cluster, nothing was predicted.
-
S2
The comparison group is missing
"All eleven of these cases share feature F" means nothing without the count of non-cases that also share F.
-
S3
Multiple comparisons go uncounted
If twenty subgroups were examined and one showed an effect at the 5 per cent level, that is roughly what chance alone produces — and the other nineteen are not reported.
Three checks, in order
- Ask how many hypotheses were tested. A single reported result out of an unstated number of attempts is not a result.
- Ask whether the finding holds in a fresh sample. This is the only real cure: a pattern found in one dataset and confirmed in another is no longer a painted target.
- Look for the pre-registration or the original plan. If the analysis was written before the data, the target was there first.
Where the link breaks
| The claim | The unasked question |
|---|---|
| This subgroup responded unusually well | How many subgroups were compared before this one was reported? |
| The outbreak clusters here | Where would a random scatter of the same size have clustered? |
| Every one of the winners had the same habit | How many non-winners had the habit too? |
EXAMPLES
Of the twelve countries with the highest longevity, nine eat a lot of a particular vegetable — so that vegetable is the key to a long life.
Why it fails
The twelve countries were selected because their longevity is high, then searched for shared features. Any list of twelve countries will share dozens of features by coincidence. Without the rate of that vegetable's consumption in countries with ordinary longevity, the nine-out-of-twelve count carries no information.
Kindred entries
- [F] survivorship You only studied the ones that made it, then asked what they have in common.
- [F] base-rate A vivid new detail outweighs the much larger number it should be measured against.
- index — /t/fallacy/
[+] Notes and objections 3
- Does this mean all pattern-finding is invalid?
- No. Exploratory pattern-finding is how hypotheses are generated. It becomes the fallacy when the generated hypothesis is presented as though it had been tested rather than produced.
- How many comparisons make a result meaningless?
- There is no fixed number, but the arithmetic is simple: at a 5 per cent threshold, about one in twenty independent tests will look positive by chance. Twenty subgroups is enough to manufacture a headline.
- What is the practical difference from p-hacking?
- They overlap heavily. P-hacking is the deliberate or careless version — trying analyses until one is significant. The Texas sharpshooter framing also covers innocent cases where the analyst genuinely did not notice they had chosen the boundary after seeing the data.
Scope — Reference material for reading arguments, not professional advice.