What does Calibration and Collective Intelligence conclude, in numbers, about “Tetlock's Forecasting Research: Experts, Foxes and Superforecasters”?

Answer: Tetlock had 284 experts make tens of thousands of numerical probability forecasts over about two decades. On average they did only slightly better than "one third each", worse than simple extrapolation and statistical models; foxes beat hedgehogs, and hedgehogs did especially badly on long-range forecasts in their own field. "Worse than a chimp" is a popular paraphrase. A toy model shows that a forecaster with the same information can fall to the level of random guessing through overconfidence alone. Forecasting tournaments show that brief probability training, teaming and tracking improve accuracy, and that superforecasters are partly discovered and partly created; weather forecasting and intelligence assessment, which have feedback, can also be well calibrated. The next chapter turns to how to keep score: the Brier score, Murphy's decomposition and proper scoring rules.

Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

outcome frequencies (status quo, more, less): 0.522, 0.238, 0.239
forecaster| mean Brier score (3 outcomes, 0 best, 2 worst)
chimp| 0.667
base rate| 0.613
fox| 0.551
overconfident | 0.665
hedgehog| 0.710
chimp's expected score is exactly 1/3 - 2/3 + 1 = 0.667 whatever happens
Show the full program
# Toy three-outcome questions (status quo / more / less), scored with Brier's original
# multi-category score (0 best, 2 worst). Truth: logits = base + s, s ~ N(0, 1).
# Forecasters see s + N(0, 1) noise: fox shrinks it by 0.5 (Bayes), overconfident x1.5,
# hedgehog x1.5 and leans towards change. Illustrative parameters.
import math, random

rng = random.Random(2005)
BASE = (1.0, 0.0, 0.0)                     # status quo is the most common outcome

def softmax(z):
    m = max(z)
    e = [math.exp(v - m) for v in z]
    t = sum(e)
    return [v / t for v in e]

def brier(f, k):
    return sum((f[j] - (1.0 if j == k else 0.0)) ** 2 for j in range(3))

def run(nq=20000):
    names = ("chimp", "base rate", "fox", "overconfident", "hedgehog")
    tot = dict.fromkeys(names, 0.0)
    freq = [0, 0, 0]
    rows = []
    for _ in range(nq):
        s = [rng.gauss(0, 1) for _ in range(3)]
        p = softmax([b + v for b, v in zip(BASE, s)])
        k = rng.choices(range(3), weights=p)[0]
        freq[k] += 1
        obs = [v + rng.gauss(0, 1) for v in s]
        rows.append((obs, k))
    base = [c / nq for c in freq]
    for obs, k in rows:
        fc = {"chimp": [1 / 3] * 3,
              "base rate": base,
              "fox": softmax([b + 0.5 * v for b, v in zip(BASE, obs)]),
              "overconfident": softmax([b + 1.5 * v for b, v in zip(BASE, obs)]),
              "hedgehog": softmax([b - (0.8 if j == 0 else 0) + 1.5 * v
                                   for j, (b, v) in enumerate(zip(BASE, obs))])}
        for n in names:
            tot[n] += brier(fc[n], k)
    return base, {n: tot[n] / nq for n in names}

if __name__ == "__main__":
    base, score = run()
    print("outcome frequencies (status quo, more, less): "
          + ", ".join(f"{b:.3f}" for b in base))
    print("forecaster    | mean Brier score (3 outcomes, 0 best, 2 worst)")
    for n, v in score.items():
        print(f"{n:13s} | {v:.3f}")
    print(f"chimp's expected score is exactly 1/3 - 2/3 + 1 = {1 / 3 - 2 / 3 + 1:.3f}"
          " whatever happens")

Single source and citation

Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 1: Tetlock's Forecasting Research: Experts, Foxes and Superforecasters. 5D Library. https://doi.org/10.5281/zenodo.23764998

DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026026,
  author    = {Fan, Yafu},
  title     = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23764998},
  url       = {https://doi.org/10.5281/zenodo.23764998}
}

Other chapter answers in this book