What does Calibration and Collective Intelligence conclude, in numbers, about “Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition”?

Answer: The Brier score is the mean squared difference between forecasts and outcomes; Murphy split it into reliability (calibration), resolution and uncertainty. The decomposition is exact when the forecasts within a bin are equal; after binning, within-bin variance and covariance terms appear. Always saying 50 % is perfectly calibrated only when the base rate is 50 %; in general, always stating the base rate is perfectly calibrated with zero resolution. An overconfident forecaster can have the same resolution as a calibrated one and still score worse by saying too much. The Brier and logarithmic scores are strictly proper; the absolute error is not. Computing an individual's calibration error from very few forecasts overstates it. The next chapter discusses the best-known error in probabilistic judgement: neglecting the base rate.

Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 50

forecaster        |   BS    REL    RES    UNC
always 50 %       | 0.2500 0.0403 0.0000 0.2097
always base rate  | 0.2097 0.0000 0.0000 0.2097
calibrated        | 0.1677 0.0001 0.0413 0.2097
overconfident     | 0.1797 0.0115 0.0407 0.2097
underconfident    | 0.1757 0.0075 0.0394 0.2097
overconfident: mean forecast -> observed frequency (bins with n >= 200):
  0.02->0.09  0.09->0.25  0.20->0.33  0.30->0.37  0.40->0.44  0.50->0.49
  0.60->0.55  0.70->0.59  0.80->0.67  0.90->0.74  0.97->0.88
perfectly calibrated, only 20 forecasts: REL mean 0.0631, median 0.0583 (truth 0)

Key formula

\[P = \frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{r}\left(f_{ij} - o_{ij}\right)^2\]
Show the full program
# Five forecasters, 20,000 events with base rate 30 %: always 50 %, always 30 %,
# calibrated (states the true chance), overconfident (2 x logit), underconfident (0.5 x
# logit about 30 %). Last part: REL from only 20 forecasts of a perfectly calibrated
# person.
import math, random

rng = random.Random(1973)

def logit(p):
    return math.log(p / (1 - p))

def inv(x):
    return 1 / (1 + math.exp(-x))

def decompose(f, o, nb=11):
    n, obar = len(f), sum(o) / len(o)
    bins = {}
    for fi, oi in zip(f, o):
        bins.setdefault(round(fi * (nb - 1)), []).append((fi, oi))
    rel = res = 0.0
    table = []
    for k in sorted(bins):
        pts = bins[k]
        fk = sum(p[0] for p in pts) / len(pts)
        ok = sum(p[1] for p in pts) / len(pts)
        rel += len(pts) * (fk - ok) ** 2
        res += len(pts) * (ok - obar) ** 2
        table.append((fk, ok, len(pts)))
    bs = sum((a - b) ** 2 for a, b in zip(f, o)) / n
    return bs, rel / n, res / n, obar * (1 - obar), table

def cases(n=20000):
    ps = [min(max(rng.betavariate(1.2, 2.8), 0.005), 0.995) for _ in range(n)]
    o = [1 if rng.random() < p else 0 for p in ps]
    fc = {"always 50 %": [0.5] * n,
          "always base rate": [0.3] * n,
          "calibrated": ps,
          "overconfident": [inv(2 * logit(p)) for p in ps],
          "underconfident": [inv(0.5 * logit(p) + 0.5 * logit(0.3)) for p in ps]}
    return {k: decompose(v, o) for k, v in fc.items()}

def small_sample_rel(m=20, reps=4000):
    vals = []
    for _ in range(reps):
        ps = [rng.betavariate(1.2, 2.8) for _ in range(m)]
        f = [round(p * 10) / 10 for p in ps]                 # states its chance, rounded
        o = [1 if rng.random() < q else 0 for q in f]        # events happen at that rate
        vals.append(decompose(f, o)[1])
    vals.sort()
    return sum(vals) / reps, vals[reps // 2]

if __name__ == "__main__":
    res = cases()
    print("forecaster        |   BS    REL    RES    UNC")
    for k, (bs, rel, rs, unc, _) in res.items():
        print(f"{k:17s} | {bs:.4f} {rel:.4f} {rs:.4f} {unc:.4f}")
    print("overconfident: mean forecast -> observed frequency (bins with n >= 200):")
    cells = [f"{f:.2f}->{ob:.2f}" for f, ob, n in res["overconfident"][4] if n >= 200]
    print("  " + "  ".join(cells[:6]))
    print("  " + "  ".join(cells[6:]))
    mean, med = small_sample_rel()
    print(f"perfectly calibrated, only 20 forecasts: REL mean {mean:.4f},"
          f" median {med:.4f} (truth 0)")

Single source and citation

Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 2: Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition. 5D Library. https://doi.org/10.5281/zenodo.23764998

DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026026,
  author    = {Fan, Yafu},
  title     = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23764998},
  url       = {https://doi.org/10.5281/zenodo.23764998}
}

Other chapter answers in this book