What does Calibration and Collective Intelligence conclude, in numbers, about “Tetlock's Forecasting Research: Experts, Foxes and Superforecasters”?
Answer: Tetlock had 284 experts make tens of thousands of numerical probability forecasts over about two decades. On average they did only slightly better than "one third each", worse than simple extrapolation and statistical models; foxes beat hedgehogs, and hedgehogs did especially badly on long-range forecasts in their own field. "Worse than a chimp" is a popular paraphrase. A toy model shows that a forecaster with the same information can fall to the level of random guessing through overconfidence alone. Forecasting tournaments show that brief probability training, teaming and tracking improve accuracy, and that superforecasters are partly discovered and partly created; weather forecasting and intelligence assessment, which have feedback, can also be well calibrated. The next chapter turns to how to keep score: the Brier score, Murphy's decomposition and proper scoring rules.
Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
| outcome frequencies (status quo, more, less): 0.522, 0.238, 0.239 | |
| forecaster | | mean Brier score (3 outcomes, 0 best, 2 worst) |
| chimp | | 0.667 |
| base rate | | 0.613 |
| fox | | 0.551 |
| overconfident | 0.665 | |
| hedgehog | | 0.710 |
| chimp's expected score is exactly 1/3 - 2/3 + 1 = 0.667 whatever happens |
Show the full program
# Toy three-outcome questions (status quo / more / less), scored with Brier's original
# multi-category score (0 best, 2 worst). Truth: logits = base + s, s ~ N(0, 1).
# Forecasters see s + N(0, 1) noise: fox shrinks it by 0.5 (Bayes), overconfident x1.5,
# hedgehog x1.5 and leans towards change. Illustrative parameters.
import math, random
rng = random.Random(2005)
BASE = (1.0, 0.0, 0.0) # status quo is the most common outcome
def softmax(z):
m = max(z)
e = [math.exp(v - m) for v in z]
t = sum(e)
return [v / t for v in e]
def brier(f, k):
return sum((f[j] - (1.0 if j == k else 0.0)) ** 2 for j in range(3))
def run(nq=20000):
names = ("chimp", "base rate", "fox", "overconfident", "hedgehog")
tot = dict.fromkeys(names, 0.0)
freq = [0, 0, 0]
rows = []
for _ in range(nq):
s = [rng.gauss(0, 1) for _ in range(3)]
p = softmax([b + v for b, v in zip(BASE, s)])
k = rng.choices(range(3), weights=p)[0]
freq[k] += 1
obs = [v + rng.gauss(0, 1) for v in s]
rows.append((obs, k))
base = [c / nq for c in freq]
for obs, k in rows:
fc = {"chimp": [1 / 3] * 3,
"base rate": base,
"fox": softmax([b + 0.5 * v for b, v in zip(BASE, obs)]),
"overconfident": softmax([b + 1.5 * v for b, v in zip(BASE, obs)]),
"hedgehog": softmax([b - (0.8 if j == 0 else 0) + 1.5 * v
for j, (b, v) in enumerate(zip(BASE, obs))])}
for n in names:
tot[n] += brier(fc[n], k)
return base, {n: tot[n] / nq for n in names}
if __name__ == "__main__":
base, score = run()
print("outcome frequencies (status quo, more, less): "
+ ", ".join(f"{b:.3f}" for b in base))
print("forecaster | mean Brier score (3 outcomes, 0 best, 2 worst)")
for n, v in score.items():
print(f"{n:13s} | {v:.3f}")
print(f"chimp's expected score is exactly 1/3 - 2/3 + 1 = {1 / 3 - 2 / 3 + 1:.3f}"
" whatever happens")Single source and citation
Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 1: Tetlock's Forecasting Research: Experts, Foxes and Superforecasters. 5D Library. https://doi.org/10.5281/zenodo.23764998
DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026026,
author = {Fan, Yafu},
title = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23764998},
url = {https://doi.org/10.5281/zenodo.23764998}
}Other chapter answers in this book
- Chapter 2 · Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition
- Chapter 3 · Base-Rate Neglect and Natural Frequencies
- Chapter 4 · Signal Detection Theory: Seeing Clearly and Where to Set the Threshold
- Chapter 5 · Reference-Class Forecasting and the Planning Fallacy
- Chapter 6 · The Statistics of Collective Intelligence
- Chapter 8 · Practice and a Pre-registered Trial