What does Calibration and Collective Intelligence conclude, in numbers, about “Base-Rate Neglect and Natural Frequencies”?

Answer: People judging individual cases often go by representativeness and give base rates far less weight than they deserve; they use them more when base rates seem relevant to the case. For a test with 95 % sensitivity and specificity and a prevalence of 1 %, the positive predictive value is about 16.1 %; more than four in ten doctors answered 95 % both in 1978 and in 2014. Natural-frequency formats clearly improve Bayesian reasoning (in the meta-analysis from about 4 % to 24 % correct), though most people still answer wrongly; the effect of frequency-tree training persists for a while. The next chapter discusses a closely related problem: where to set the threshold of judgement when real signals are rare.

Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. printed values that match the answer: 16.1, 95

formula: P(condition | +)0.95 x 0.01 / (0.95 x 0.01 + 0.05 x 0.99) = 0.1610
1,000 people
10 with the condition9.5 positive, 0.5 neg
990 without it49.5 positive, 940.5 neg
positives9.5 + 49.5 = 59; true among them 9.5/59 = 16.1%
10,000 people
100 with the condition95 positive, 5 neg
9900 without it495 positive, 9405 neg
positives95 + 495 = 590; true among them 95/590 = 16.1%
PPV against the base rate (sensitivity = specificity95 %):
0.1%1.9% 1.0%: 16.1% 5.0%: 50.0% 10.0%: 67.9% 30.0%: 89.1% 50.0%: 95.0%
second independent positive test, prior 16.1%78.5%
the common wrong answer 95 % is P(+ | condition), not P(condition | +)

Key formula

\[P(\text{患病} \mid +) = \frac{0.95 \times 0.01}{0.95 \times 0.01 + 0.05 \times 0.99} \approx 0.161.\]
Show the full program
# Positive predictive value: prevalence 1 %, sensitivity and specificity 95 %; formula and
# natural-frequency trees. A repeat test is assumed independent given the condition.


def ppv(prev, sens, spec):
    tp = sens * prev
    fp = (1 - spec) * (1 - prev)
    return tp / (tp + fp)

def tree(n, prev, sens, spec):
    sick = n * prev
    healthy = n - sick
    tp, fp = sick * sens, healthy * (1 - spec)
    print(f"{n:,} people")
    print(f"  {sick:>5g} with the condition -> {tp:>4g} positive, {sick - tp:g} neg")
    print(f"  {healthy:>5g} without it         -> {fp:>4g} positive, {healthy - fp:g} neg")
    print(f"  positives: {tp:g} + {fp:g} = {tp + fp:g}; true among them {tp:g}/{tp + fp:g}"
          f" = {tp / (tp + fp):.1%}")

if __name__ == "__main__":
    print(f"formula: P(condition | +) = 0.95 x 0.01 / (0.95 x 0.01 + 0.05 x 0.99)"
          f" = {ppv(0.01, 0.95, 0.95):.4f}")
    tree(1000, 0.01, 0.95, 0.95)
    tree(10000, 0.01, 0.95, 0.95)
    print("PPV against the base rate (sensitivity = specificity = 95 %):")
    print("  " + "   ".join(f"{p:.1%}: {ppv(p, 0.95, 0.95):.1%}"
                            for p in (0.001, 0.01, 0.05, 0.10, 0.30, 0.50)))
    first = ppv(0.01, 0.95, 0.95)
    second = ppv(first, 0.95, 0.95)
    print(f"second independent positive test, prior {first:.1%} -> {second:.1%}")
    print("the common wrong answer 95 % is P(+ | condition), not P(condition | +)")

Single source and citation

Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 3: Base-Rate Neglect and Natural Frequencies. 5D Library. https://doi.org/10.5281/zenodo.23764998

DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026026,
  author    = {Fan, Yafu},
  title     = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23764998},
  url       = {https://doi.org/10.5281/zenodo.23764998}
}

Other chapter answers in this book