Representativeness and the neglected base rate

In 1974 Tversky and Kahneman summarised in Science several heuristics people use when judging under uncertainty (Tversky and Kahneman 1974). One is representativeness: we judge which category a person belongs to by how much he “resembles” the category, not by how common the category is. In an earlier experiment, participants read a short personality sketch and were told that the person came from a group of 70 engineers and 30 lawyers (or the reverse, 30 engineers and 70 lawyers), then estimated the probability that the person was an engineer. The estimates were almost the same in the two conditions: participants looked almost only at how much the sketch resembled an engineer, and largely ignored the 70-to-30 base rate (Kahneman and Tversky 1973).

Base-rate neglect does not mean that people never use base rates, but that once they have case information they give the base rate far less weight than it deserves.

Doctors and a positive test

The answer 95 % confuses two conditional probabilities: $P(\text{positive} \mid \text{disease})$ and $P(\text{disease} \mid \text{positive})$. The sensitivity of a test describes the first; doctors and patients care about the second, the positive predictive value.

# Positive predictive value: prevalence 1 %, sensitivity and specificity 95 %; formula and
# natural-frequency trees. A repeat test is assumed independent given the condition.


def ppv(prev, sens, spec):
    tp = sens * prev
    fp = (1 - spec) * (1 - prev)
    return tp / (tp + fp)

def tree(n, prev, sens, spec):
    sick = n * prev
    healthy = n - sick
    tp, fp = sick * sens, healthy * (1 - spec)
    print(f"{n:,} people")
    print(f"  {sick:>5g} with the condition -> {tp:>4g} positive, {sick - tp:g} neg")
    print(f"  {healthy:>5g} without it         -> {fp:>4g} positive, {healthy - fp:g} neg")
    print(f"  positives: {tp:g} + {fp:g} = {tp + fp:g}; true among them {tp:g}/{tp + fp:g}"
          f" = {tp / (tp + fp):.1%}")

if __name__ == "__main__":
    print(f"formula: P(condition | +) = 0.95 x 0.01 / (0.95 x 0.01 + 0.05 x 0.99)"
          f" = {ppv(0.01, 0.95, 0.95):.4f}")
    tree(1000, 0.01, 0.95, 0.95)
    tree(10000, 0.01, 0.95, 0.95)
    print("PPV against the base rate (sensitivity = specificity = 95 %):")
    print("  " + "   ".join(f"{p:.1%}: {ppv(p, 0.95, 0.95):.1%}"
                            for p in (0.001, 0.01, 0.05, 0.10, 0.30, 0.50)))
    first = ppv(0.01, 0.95, 0.95)
    second = ppv(first, 0.95, 0.95)
    print(f"second independent positive test, prior {first:.1%} -> {second:.1%}")
    print("the common wrong answer 95 % is P(+ | condition), not P(condition | +)")
# formula: P(condition | +) = 0.95 x 0.01 / (0.95 x 0.01 + 0.05 x 0.99) = 0.1610
# 1,000 people
#      10 with the condition ->  9.5 positive, 0.5 neg
#     990 without it         -> 49.5 positive, 940.5 neg
#   positives: 9.5 + 49.5 = 59; true among them 9.5/59 = 16.1%
# 10,000 people
#     100 with the condition ->   95 positive, 5 neg
#    9900 without it         ->  495 positive, 9405 neg
#   positives: 95 + 495 = 590; true among them 95/590 = 16.1%
# PPV against the base rate (sensitivity = specificity = 95 %):
#   0.1%: 1.9%   1.0%: 16.1%   5.0%: 50.0%   10.0%: 67.9%   30.0%: 89.1%   50.0%: 95.0%
# second independent positive test, prior 16.1% -> 78.5%
# the common wrong answer 95 % is P(+ | condition), not P(condition | +)

Of 1,000 people, about 10 have the condition, of whom 9.5 (in expectation) test positive; of the 990 healthy people, 49.5 test falsely positive. About 59 test positive in all, of whom only 9.5, about 16.1 %, are truly ill. With 10,000 people there are no half-people: 95 of the 590 positives are true. The same test has a positive predictive value of only 1.9 % in a population with a prevalence of 1 in 1,000, 67.9 % at a prevalence of 10 %, and reaches 95 % only at a prevalence of 50 %. What a “95 % accurate” result means depends on who is being tested.

If a second, independent test after the first positive is also positive, the posterior rises from 16.1 % to 78.5 %: the result of the first test becomes the prior for the second. But real repeat tests are often not independent (the same interfering factor can make both falsely positive), and then a second positive tells us much less.

Natural frequencies

These studies give two grounds for the trial in Chapter 8. First, converting to natural frequencies is a skill that can be taught in a short time. Second, even after learning it, most people still make errors on new problems, so the effect of training must be tested on new items different from the practice items.

Chapter summary

People judging individual cases often go by representativeness and give base rates far less weight than they deserve; they use them more when base rates seem relevant to the case. For a test with 95 % sensitivity and specificity and a prevalence of 1 %, the positive predictive value is about 16.1 %; more than four in ten doctors answered 95 % both in 1978 and in 2014. Natural-frequency formats clearly improve Bayesian reasoning (in the meta-analysis from about 4 % to 24 % correct), though most people still answer wrongly; the effect of frequency-tree training persists for a while. The next chapter discusses a closely related problem: where to set the threshold of judgement when real signals are rare.