What does Calibration and Collective Intelligence conclude, in numbers, about “Base-Rate Neglect and Natural Frequencies”?
Answer: People judging individual cases often go by representativeness and give base rates far less weight than they deserve; they use them more when base rates seem relevant to the case. For a test with 95 % sensitivity and specificity and a prevalence of 1 %, the positive predictive value is about 16.1 %; more than four in ten doctors answered 95 % both in 1978 and in 2014. Natural-frequency formats clearly improve Bayesian reasoning (in the meta-analysis from about 4 % to 24 % correct), though most people still answer wrongly; the effect of frequency-tree training persists for a while. The next chapter discusses a closely related problem: where to set the threshold of judgement when real signals are rare.
Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. printed values that match the answer: 16.1, 95
| formula: P(condition | +) | 0.95 x 0.01 / (0.95 x 0.01 + 0.05 x 0.99) = 0.1610 |
| 1,000 people | |
| 10 with the condition | 9.5 positive, 0.5 neg |
| 990 without it | 49.5 positive, 940.5 neg |
| positives | 9.5 + 49.5 = 59; true among them 9.5/59 = 16.1% |
| 10,000 people | |
| 100 with the condition | 95 positive, 5 neg |
| 9900 without it | 495 positive, 9405 neg |
| positives | 95 + 495 = 590; true among them 95/590 = 16.1% |
| PPV against the base rate (sensitivity = specificity | 95 %): |
| 0.1% | 1.9% 1.0%: 16.1% 5.0%: 50.0% 10.0%: 67.9% 30.0%: 89.1% 50.0%: 95.0% |
| second independent positive test, prior 16.1% | 78.5% |
| the common wrong answer 95 % is P(+ | condition), not P(condition | +) |
Key formula
Show the full program
# Positive predictive value: prevalence 1 %, sensitivity and specificity 95 %; formula and
# natural-frequency trees. A repeat test is assumed independent given the condition.
def ppv(prev, sens, spec):
tp = sens * prev
fp = (1 - spec) * (1 - prev)
return tp / (tp + fp)
def tree(n, prev, sens, spec):
sick = n * prev
healthy = n - sick
tp, fp = sick * sens, healthy * (1 - spec)
print(f"{n:,} people")
print(f" {sick:>5g} with the condition -> {tp:>4g} positive, {sick - tp:g} neg")
print(f" {healthy:>5g} without it -> {fp:>4g} positive, {healthy - fp:g} neg")
print(f" positives: {tp:g} + {fp:g} = {tp + fp:g}; true among them {tp:g}/{tp + fp:g}"
f" = {tp / (tp + fp):.1%}")
if __name__ == "__main__":
print(f"formula: P(condition | +) = 0.95 x 0.01 / (0.95 x 0.01 + 0.05 x 0.99)"
f" = {ppv(0.01, 0.95, 0.95):.4f}")
tree(1000, 0.01, 0.95, 0.95)
tree(10000, 0.01, 0.95, 0.95)
print("PPV against the base rate (sensitivity = specificity = 95 %):")
print(" " + " ".join(f"{p:.1%}: {ppv(p, 0.95, 0.95):.1%}"
for p in (0.001, 0.01, 0.05, 0.10, 0.30, 0.50)))
first = ppv(0.01, 0.95, 0.95)
second = ppv(first, 0.95, 0.95)
print(f"second independent positive test, prior {first:.1%} -> {second:.1%}")
print("the common wrong answer 95 % is P(+ | condition), not P(condition | +)")Single source and citation
Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 3: Base-Rate Neglect and Natural Frequencies. 5D Library. https://doi.org/10.5281/zenodo.23764998
DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026026,
author = {Fan, Yafu},
title = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23764998},
url = {https://doi.org/10.5281/zenodo.23764998}
}Other chapter answers in this book
- Chapter 1 · Tetlock's Forecasting Research: Experts, Foxes and Superforecasters
- Chapter 2 · Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition
- Chapter 4 · Signal Detection Theory: Seeing Clearly and Where to Set the Threshold
- Chapter 5 · Reference-Class Forecasting and the Planning Fallacy
- Chapter 6 · The Statistics of Collective Intelligence
- Chapter 8 · Practice and a Pre-registered Trial