What does Calibration and Collective Intelligence conclude, in numbers, about “Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition”?
Answer: The Brier score is the mean squared difference between forecasts and outcomes; Murphy split it into reliability (calibration), resolution and uncertainty. The decomposition is exact when the forecasts within a bin are equal; after binning, within-bin variance and covariance terms appear. Always saying 50 % is perfectly calibrated only when the base rate is 50 %; in general, always stating the base rate is perfectly calibrated with zero resolution. An overconfident forecaster can have the same resolution as a calibrated one and still score worse by saying too much. The Brier and logarithmic scores are strictly proper; the absolute error is not. Computing an individual's calibration error from very few forecasts overstates it. The next chapter discusses the best-known error in probabilistic judgement: neglecting the base rate.
Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 50
forecaster | BS REL RES UNC always 50 % | 0.2500 0.0403 0.0000 0.2097 always base rate | 0.2097 0.0000 0.0000 0.2097 calibrated | 0.1677 0.0001 0.0413 0.2097 overconfident | 0.1797 0.0115 0.0407 0.2097 underconfident | 0.1757 0.0075 0.0394 0.2097 overconfident: mean forecast -> observed frequency (bins with n >= 200): 0.02->0.09 0.09->0.25 0.20->0.33 0.30->0.37 0.40->0.44 0.50->0.49 0.60->0.55 0.70->0.59 0.80->0.67 0.90->0.74 0.97->0.88 perfectly calibrated, only 20 forecasts: REL mean 0.0631, median 0.0583 (truth 0)
Key formula
Show the full program
# Five forecasters, 20,000 events with base rate 30 %: always 50 %, always 30 %,
# calibrated (states the true chance), overconfident (2 x logit), underconfident (0.5 x
# logit about 30 %). Last part: REL from only 20 forecasts of a perfectly calibrated
# person.
import math, random
rng = random.Random(1973)
def logit(p):
return math.log(p / (1 - p))
def inv(x):
return 1 / (1 + math.exp(-x))
def decompose(f, o, nb=11):
n, obar = len(f), sum(o) / len(o)
bins = {}
for fi, oi in zip(f, o):
bins.setdefault(round(fi * (nb - 1)), []).append((fi, oi))
rel = res = 0.0
table = []
for k in sorted(bins):
pts = bins[k]
fk = sum(p[0] for p in pts) / len(pts)
ok = sum(p[1] for p in pts) / len(pts)
rel += len(pts) * (fk - ok) ** 2
res += len(pts) * (ok - obar) ** 2
table.append((fk, ok, len(pts)))
bs = sum((a - b) ** 2 for a, b in zip(f, o)) / n
return bs, rel / n, res / n, obar * (1 - obar), table
def cases(n=20000):
ps = [min(max(rng.betavariate(1.2, 2.8), 0.005), 0.995) for _ in range(n)]
o = [1 if rng.random() < p else 0 for p in ps]
fc = {"always 50 %": [0.5] * n,
"always base rate": [0.3] * n,
"calibrated": ps,
"overconfident": [inv(2 * logit(p)) for p in ps],
"underconfident": [inv(0.5 * logit(p) + 0.5 * logit(0.3)) for p in ps]}
return {k: decompose(v, o) for k, v in fc.items()}
def small_sample_rel(m=20, reps=4000):
vals = []
for _ in range(reps):
ps = [rng.betavariate(1.2, 2.8) for _ in range(m)]
f = [round(p * 10) / 10 for p in ps] # states its chance, rounded
o = [1 if rng.random() < q else 0 for q in f] # events happen at that rate
vals.append(decompose(f, o)[1])
vals.sort()
return sum(vals) / reps, vals[reps // 2]
if __name__ == "__main__":
res = cases()
print("forecaster | BS REL RES UNC")
for k, (bs, rel, rs, unc, _) in res.items():
print(f"{k:17s} | {bs:.4f} {rel:.4f} {rs:.4f} {unc:.4f}")
print("overconfident: mean forecast -> observed frequency (bins with n >= 200):")
cells = [f"{f:.2f}->{ob:.2f}" for f, ob, n in res["overconfident"][4] if n >= 200]
print(" " + " ".join(cells[:6]))
print(" " + " ".join(cells[6:]))
mean, med = small_sample_rel()
print(f"perfectly calibrated, only 20 forecasts: REL mean {mean:.4f},"
f" median {med:.4f} (truth 0)")Single source and citation
Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 2: Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition. 5D Library. https://doi.org/10.5281/zenodo.23764998
DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026026,
author = {Fan, Yafu},
title = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23764998},
url = {https://doi.org/10.5281/zenodo.23764998}
}Other chapter answers in this book
- Chapter 1 · Tetlock's Forecasting Research: Experts, Foxes and Superforecasters
- Chapter 3 · Base-Rate Neglect and Natural Frequencies
- Chapter 4 · Signal Detection Theory: Seeing Clearly and Where to Set the Threshold
- Chapter 5 · Reference-Class Forecasting and the Planning Fallacy
- Chapter 6 · The Statistics of Collective Intelligence
- Chapter 8 · Practice and a Pre-registered Trial