Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence: chapter answers
How likely, exactly, is "it'll probably work out"? How often has the speaker been right? This book uses the psychology of judgement and decision making, forecast verification and statistics to ask how the accuracy of probabilistic judgements can be measured, in what systematic ways they go wrong, and when combining many people's judgements makes them more accurate. It covers Tetlock's study of 284 experts' forecasts over about two decades (some 28,000 forecasts, with a count of 82,361 also quoted by reviewers; experts did slightly better than random guessing but worse than simple extrapolatio…
DOI 10.5281/zenodo.23764998 · Read the monograph · 中文版
- Chapter 1 · Tetlock's Forecasting Research: Experts, Foxes and Superforecasters
Tetlock had 284 experts make tens of thousands of numerical probability forecasts over about two decades. On average they did only slightly better than "one third each", worse than simple extrapolation and statistical m… - Chapter 2 · Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition
The Brier score is the mean squared difference between forecasts and outcomes; Murphy split it into reliability (calibration), resolution and uncertainty. The decomposition is exact when the forecasts within a bin are e… - Chapter 3 · Base-Rate Neglect and Natural Frequencies
People judging individual cases often go by representativeness and give base rates far less weight than they deserve; they use them more when base rates seem relevant to the case. For a test with 95 % sensitivity and sp… - Chapter 4 · Signal Detection Theory: Seeing Clearly and Where to Set the Threshold
The first three judges have almost the same sensitivity ($d'$ about 2); the difference lies entirely in the threshold. The neutral judge has $c = 0$ and $\beta = 1$; the liberal one has $c = -0.70$ and a hit rate of 95… - Chapter 5 · Reference-Class Forecasting and the Planning Fallacy
The planning fallacy is the tendency to take the inside view and underestimate time and cost systematically; students took longer to write their theses than even their "worst case" estimates. Almost nine in ten (86 %) o… - Chapter 6 · The Statistics of Collective Intelligence
The crowd median Galton reported in 1907 was 1,207 lb (it should have been 1,208); the mean of 1,197 lb came later and exactly equals the corrected actual weight. The diversity prediction theorem is an identity: crowd e… - Chapter 8 · Practice and a Pre-registered Trial
The four-step calibration checklist turns the conclusions of earlier chapters into four recordable steps: replace vague words with numbers, find the base rate of an outside reference class first, write down update trigg…
Illustrative model calculation, not a live quote