Keeping score on judgement

Television pundits, newspaper columnists and consultants in meeting rooms forecast all the time: will a regime fall, will a reform pass, will the economy go into recession next year? These forecasts are rarely recorded systematically and even more rarely checked. Even when someone tries to check them, there is often nothing to grip: “a crisis is possible” counts as right if a crisis comes, and is not wrong if it does not.

From the mid-1980s the psychologist Philip Tetlock did something few people were then doing: he asked experts to give numerical probabilities for questions that could be unambiguously resolved, sealed the answers, and, years later when the outcomes were known, scored them all on the same scale (Tetlock 2005). Most questions had three options: within a stated period, would some indicator (a country’s economic growth, a party’s support, the intensity of a conflict) show the “status quo”, “more” or “less”? The experts gave a probability to each of the three outcomes.

The chimp and the extrapolation algorithm

So “experts are worse than chimps” is not the book’s conclusion. More accurately: on average, experts were only a little better than the chimp and a long way behind the statistical models, and some experts (especially confident hedgehogs) were indeed beaten by the chimp on some questions. In their later book Tetlock and Gardner also note that the joke is often misread as “experts are useless”, whereas the difference between foxes and hedgehogs is what deserves attention (Tetlock and Gardner 2015).

How can people with knowledge lose to “one third each”? One simple mechanism is overconfidence: information gets inflated into numbers that are too certain.

# Toy three-outcome questions (status quo / more / less), scored with Brier's original
# multi-category score (0 best, 2 worst). Truth: logits = base + s, s ~ N(0, 1).
# Forecasters see s + N(0, 1) noise: fox shrinks it by 0.5 (Bayes), overconfident x1.5,
# hedgehog x1.5 and leans towards change. Illustrative parameters.
import math, random

rng = random.Random(2005)
BASE = (1.0, 0.0, 0.0)                     # status quo is the most common outcome

def softmax(z):
    m = max(z)
    e = [math.exp(v - m) for v in z]
    t = sum(e)
    return [v / t for v in e]

def brier(f, k):
    return sum((f[j] - (1.0 if j == k else 0.0)) ** 2 for j in range(3))

def run(nq=20000):
    names = ("chimp", "base rate", "fox", "overconfident", "hedgehog")
    tot = dict.fromkeys(names, 0.0)
    freq = [0, 0, 0]
    rows = []
    for _ in range(nq):
        s = [rng.gauss(0, 1) for _ in range(3)]
        p = softmax([b + v for b, v in zip(BASE, s)])
        k = rng.choices(range(3), weights=p)[0]
        freq[k] += 1
        obs = [v + rng.gauss(0, 1) for v in s]
        rows.append((obs, k))
    base = [c / nq for c in freq]
    for obs, k in rows:
        fc = {"chimp": [1 / 3] * 3,
              "base rate": base,
              "fox": softmax([b + 0.5 * v for b, v in zip(BASE, obs)]),
              "overconfident": softmax([b + 1.5 * v for b, v in zip(BASE, obs)]),
              "hedgehog": softmax([b - (0.8 if j == 0 else 0) + 1.5 * v
                                   for j, (b, v) in enumerate(zip(BASE, obs))])}
        for n in names:
            tot[n] += brier(fc[n], k)
    return base, {n: tot[n] / nq for n in names}

if __name__ == "__main__":
    base, score = run()
    print("outcome frequencies (status quo, more, less): "
          + ", ".join(f"{b:.3f}" for b in base))
    print("forecaster    | mean Brier score (3 outcomes, 0 best, 2 worst)")
    for n, v in score.items():
        print(f"{n:13s} | {v:.3f}")
    print(f"chimp's expected score is exactly 1/3 - 2/3 + 1 = {1 / 3 - 2 / 3 + 1:.3f}"
          " whatever happens")
# outcome frequencies (status quo, more, less): 0.522, 0.238, 0.239
# forecaster    | mean Brier score (3 outcomes, 0 best, 2 worst)
# chimp         | 0.667
# base rate     | 0.613
# fox           | 0.551
# overconfident | 0.665
# hedgehog      | 0.710
# chimp's expected score is exactly 1/3 - 2/3 + 1 = 0.667 whatever happens

The results show several things. The chimp’s expected score is exactly 2/3 ≈ 0.667, whatever happens: the score anyone “knowing nothing” can get. Merely stating the long-run frequencies of this class of question (status quo about 52 %) brings the score down to 0.613. The fox, using the same information as everyone else, gets 0.551. The overconfident forecaster has exactly the same information as the fox, yet scores 0.665, almost the same as the chimp; the hedgehog, adding a lean towards change, gets 0.710, worse than the chimp. In this model the reason for losing to the chimp is not ignorance but stating useful information too strongly. The parameters are illustrative and the model does not reproduce Tetlock’s data.

Updating beliefs: foxes revise more, hedgehogs less

Tetlock also examined how experts revised their beliefs once outcomes were known. In one analysis in the book, when their forecasts were wrong, hedgehogs moved only about a fifth of the way Bayes’ theorem requires (about 19 %), foxes about six tenths (about 59 %) (Tetlock 2005). Revising too little wastes the evidence; but revising too much does harm as well.

# Updating in log-odds: prior 30 %, 8 news items, each with likelihood ratio 2 (or 1/2).
# A forecaster adds w x log LR per item (w = 1 Bayesian, < 1 under-, > 1 over-reaction).
# Binary Brier score after 0, 2, 4, 8 items. Toy model, not Tetlock's analysis.
import math, random

rng = random.Random(42)
PRIOR, LR, N_NEWS = 0.30, 2.0, 8
P_GOOD = LR / (1 + LR)          # a piece of news points the right way with probability 2/3

def logit(p):
    return math.log(p / (1 - p))

def inv(x):
    return 1 / (1 + math.exp(-x))

def simulate(weights, nq=40000, checkpoints=(0, 2, 4, 8)):
    score = {w: {c: 0.0 for c in checkpoints} for w in weights}
    for _ in range(nq):
        y = 1 if rng.random() < PRIOR else 0
        news = []
        for _ in range(N_NEWS):
            right = rng.random() < P_GOOD
            news.append(1 if (right == (y == 1)) else -1)       # +1 favours yes
        for w in weights:
            z = logit(PRIOR)
            if 0 in checkpoints:
                score[w][0] += (inv(z) - y) ** 2
            for t, e in enumerate(news, 1):
                z += w * e * math.log(LR)
                if t in checkpoints:
                    score[w][t] += (inv(z) - y) ** 2
    return {w: {c: v / nq for c, v in d.items()} for w, d in score.items()}

if __name__ == "__main__":
    ws = (0.0, 0.2, 0.6, 1.0, 2.0)
    res = simulate(ws)
    label = {0.0: "never updates", 0.2: "under-reacts", 0.6: "partly",
             1.0: "Bayesian", 2.0: "over-reacts"}
    print("weight w on each news item | Brier after 0, 2, 4, 8 items of news")
    for w in ws:
        r = res[w]
        tag = "(" + label[w] + ")"
        print(f"  w = {w:3.1f} {tag:17s}   | "
              + "  ".join(f"{r[c]:.3f}" for c in (0, 2, 4, 8)))
    z = logit(PRIOR) + 3 * math.log(LR) - 1 * math.log(LR)
    print(f"example: prior 30 %, three items for and one against -> {inv(z):.1%}")
# weight w on each news item | Brier after 0, 2, 4, 8 items of news
#   w = 0.0 (never updates)     | 0.210  0.210  0.210  0.210
#   w = 0.2 (under-reacts)      | 0.210  0.195  0.181  0.156
#   w = 0.6 (partly)            | 0.210  0.177  0.150  0.110
#   w = 1.0 (Bayesian)          | 0.210  0.172  0.143  0.102
#   w = 2.0 (over-reacts)       | 0.210  0.185  0.156  0.112
# example: prior 30 %, three items for and one against -> 63.2%

Before any news, everyone scores 0.210, that is $0.3 \times 0.7$: the score of simply stating the base rate. After eight pieces of news the Bayesian forecaster is down to 0.102; the under-reactor using a fifth of the evidence stays at 0.156, and the one using six tenths reaches about 0.110; the over-reactor who doubles each piece of news gets 0.112, also worse than Bayes. Errors in both directions cost something, which is exactly what the later research on “superforecasters” stresses again and again: update often, in small steps. The last line gives a concrete example: a prior of 30 %, three items for and one against, gives a posterior of about 63.2 %.

Forecasting tournaments and superforecasters

In 2011 the US Intelligence Advanced Research Projects Activity (IARPA) funded a geopolitical forecasting tournament lasting several years. Several university teams recruited forecasters, elicited and aggregated their forecasts with different methods, and were ranked by Brier score. The Good Judgment Project, led by Tetlock and Barbara Mellers, won year after year.

These findings do not contradict the first part of the chapter. Most experts in Tetlock’s early study received no feedback: their forecasts were sealed and resolved years later, and the results never came back to them. Tournament forecasters saw their Brier score as soon as each question closed.

Chapter summary

Tetlock had 284 experts make tens of thousands of numerical probability forecasts over about two decades. On average they did only slightly better than “one third each”, worse than simple extrapolation and statistical models; foxes beat hedgehogs, and hedgehogs did especially badly on long-range forecasts in their own field. “Worse than a chimp” is a popular paraphrase. A toy model shows that a forecaster with the same information can fall to the level of random guessing through overconfidence alone. Forecasting tournaments show that brief probability training, teaming and tracking improve accuracy, and that superforecasters are partly discovered and partly created; weather forecasting and intelligence assessment, which have feedback, can also be well calibrated. The next chapter turns to how to keep score: the Brier score, Murphy’s decomposition and proper scoring rules.