What the book covers

Chapter 1 starts from Tetlock’s long-term forecasting research: tens of thousands of forecasts by 284 experts over two decades, and the “superforecasters” found later in forecasting tournaments funded by the US intelligence community; a program illustrates one way of having plenty of information and still forecasting badly: overconfidence. Chapter 2 introduces the Brier score and Murphy’s decomposition, separates calibration (do the things you call 70 % happen about seven times in ten?) from resolution (can you tell what will happen from what will not?), and derives why the Brier and logarithmic scores are “strictly proper” scoring rules. Chapter 3 discusses base-rate neglect and natural frequencies. Chapter 4 uses signal detection theory to separate “how well you see” from “where you set the threshold”, and derives the optimal threshold when errors carry unequal costs. Chapter 5 discusses the planning fallacy and reference-class forecasting, and uses empirical-Bayes shrinkage to combine “our own experience” with “the history of the class”. Chapter 6 turns to the wisdom of crowds: Galton’s ox-weighing competition, the diversity prediction theorem, correlated errors and echo chambers, and why averaged probabilities often need to be “extremised”.

Chapter 7 reads this research beside the “temple calculation” of the Sunzi’s chapter “Estimates”, the “weighing of power” in the Guiguzi’s chapter “Assessing”, and the “cross-checking” of the Han Feizi’s chapter “Eminent Teachings”, and makes clear that these texts are about pre-war planning and testing words against facts, not probability theory. Chapter 8 offers a four-step probability-calibration checklist and a pre-registered randomised controlled trial.

Claims corrected on checking

Several points in the author’s original framework needed correction or qualification once they were checked.

First, the size of Tetlock’s study. The 284 experts are correct; the figures of “28,000 forecasts” and “82,000 forecasts” both have sources and probably count different things. Tetlock and Gardner’s popular essay of 2011 speaks of some 28,000 predictions (a University of California, Berkeley press release says about 27,000), while reviewers quote the book as having accumulated 82,361 “forecasts” by 2003. The two figures differ by a factor of about three (82,361 ÷ 3 ≈ 27,454), and most questions asked for probabilities on three outcomes (“status quo”, “more”, “less”); the author conjectures that 82,361 counts the probability estimates for individual outcomes and 28,000 the forecasts at the level of questions. This reconciliation is the author’s conjecture; neither the reviews nor the book’s summary say so, and the safe citation is “some 28,000 forecasts”, noting that a count of 82,361 also circulates. The study ran for about two decades, from the mid-1980s to 2003.

Second, “experts did no better than dart-throwing chimpanzees” is a popular paraphrase. The book’s conclusion is that experts on average did slightly better than random guessing (the “chimp” giving every outcome the same probability), but worse than simple statistical extrapolation and worse still than more sophisticated statistical models; eclectic, self-critical “foxes” clearly beat “hedgehogs” committed to one big theory, and hedgehogs did especially badly on long-range forecasts within their own specialty.

Third, “someone who always says 50 % is perfectly calibrated” holds only if the base rate of the events is exactly 50 %. The correct general statement is that someone who always states the base rate is perfectly calibrated but has zero resolution; if the base rate is 30 %, always saying 50 % gives a calibration error of 0.04 (program in Chapter 2).

Fourth, “almost every project overruns its time and budget” should be stated with the actual data: Flyvbjerg and colleagues’ study of 258 transport infrastructure projects in 20 nations found that almost nine in ten (86 %) had cost overruns, with an average size that depends on the type of project: 44.7 % for rail, 33.8 % for bridges and tunnels, 20.4 % for roads.

Fifth, in 1907 Galton reported the median of the crowd’s estimates, 1,207 lb (against an actual weight of 1,198 lb); the mean of 1,197 lb appeared only in a later letter. A modern re-examination found that the median should have been 1,208 lb and the actual weight 1,197 lb. Nor is “the crowd beats the expert” a general law: it holds only under conditions such as weakly correlated errors.

Sixth, on the classical texts. The Guiguzi has no chapter called “Weighing Power” (liang quan); the term appears at the opening of the chapter “Assessing” (Chuai). The sentence quoted in the framework, “weighing power is the means to measure great and small and to plan for strong and weak”, is not in the received text, which reads “What is weighing power? It is to measure great and small and to reckon many and few…”. Whether the Sunzi’s suan means counting rods, and whether the “five factors and seven assessments” were “weighted item by item”, are later interpretations; the text says only that one “measures by the five factors and compares by the assessments”.

Each chapter says so where the point arises, and the Editor’s Notes published with the book files collect them.

How to read the boxes

As in other 5D Library books, different kinds of statement are marked separately.

When the book applies concepts such as signal detection or statistical aggregation to everyday judgements and personal situations, they are analogies and simplified models: they help us see structure, but a person’s situation is far richer than any model.

About this edition

This is version 1.0.0, the first complete edition: a preface and eight chapters, in Chinese and English. The text, boxes and code were drafted with an AI writing tool from the author’s title, framework and notes; the author is responsible for reviewing and revising them and for checking every source. Corrections and missing references are welcome through the 5D Library website.