Chapter 2 Strictly Proper Scoring Rules: The Brier Score and Murphy’s Decomposition

A meteorologist’s scoring rule

In 1950 Glenn Brier of the US Weather Bureau published a three-page note in Monthly Weather Review proposing a single number for verifying probability forecasts (Brier 1950). Suppose there are $N$ forecasts, each with $r$ possible outcomes; the forecaster gives probability $f_{ij}$ to outcome $j$ , the outcome that occurs is coded $o_{ij} = 1$ and the others 0:

$P = \frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{r}\left(f_{ij} – o_{ij}\right)^2$

This is Brier’s original form, which runs from 0 to 2. Today the binary form is more common: look only at the outcome “the event happens”,

$BS = \frac{1}{N}\sum_{i=1}^{N}\left(f_i – o_i\right)^2,$

which runs from 0 to 1 and is half of the original form. The three-outcome questions of Chapter 1 used the original form; the rest of this chapter uses the binary one. The score has a valuable property that was later studied systematically: a forecaster cannot improve the expected score by reporting a probability he does not believe (Gneiting and Raftery 2007). A later section proves this.

Murphy’s decomposition: reliability, resolution and uncertainty

In 1973 the meteorologist Allan Murphy split the Brier score into three parts (Murphy 1973). Group the forecasts by value into $K$ bins (for example, all forecasts of 30 % in one bin); bin $k$ holds $n_k$ forecasts with mean forecast $\bar f_k$ , of which a fraction $\bar o_k$ came true; across all forecasts the event happened in a fraction $\bar o$ . Then

$BS = \underbrace{\frac{1}{N}\sum_k n_k(\bar f_k – \bar o_k)^2}_{\text{REL}} \;-\; \underbrace{\frac{1}{N}\sum_k n_k(\bar o_k – \bar o)^2}_{\text{RES}} \;+\; \underbrace{\bar o(1 – \bar o)}_{\text{UNC}}.$

Reliability (REL, also called calibration error): do the things you call 30 % happen about three times in ten? Smaller is better; 0 for perfect calibration.
Resolution (RES): how far the observed frequencies in different bins lie from the overall base rate, that is, whether the forecaster can separate “more likely” from “less likely” events. Larger is better.
Uncertainty (UNC): depends only on the base rate of the events, not on the forecaster.

Calculation

Derivation. Within bin $k$ , if every forecast equals $\bar f_k$ , then $\sum_{i \in k}(f_i – o_i)^2 = n_k(\bar f_k – \bar o_k)^2 + \sum_{i\in k}(o_i – \bar o_k)^2$ , and the last term equals $n_k\bar o_k(1 – \bar o_k)$ ; using $\bar o(1-\bar o) – \frac{1}{N}\sum_k n_k(\bar o_k – \bar o)^2 = \frac{1}{N}\sum_k n_k \bar o_k(1-\bar o_k)$ gives the formula. If the forecasts in a bin are not all equal (for instance when continuous forecasts are binned), two more terms appear: the within-bin variance of the forecasts, WBV, and the within-bin covariance of forecasts and outcomes, WBC, so that $BS = \text{REL} – \text{RES} + \text{UNC} + \text{WBV} – 2\,\text{WBC}$ (Stephenson, Coelho, and Jolliffe 2008).

The program generates 5,000 synthetic forecasts, first letting the forecaster say only 0 %, 10 %, …, 100 %, then letting him say any number and binning afterwards into 11 bins, and checks both identities.

Calculation

# Brier score and Murphy's decomposition BS = REL - RES + UNC, computed by binning;
# exact when all forecasts in a bin are equal. With binned continuous forecasts two
# within-bin terms appear (Stephenson et al. 2008): BS = REL - RES + UNC + WBV - 2 WBC.
import random

rng = random.Random(1950)

def murphy(f, o, nbins=11):
    n = len(f)
    obar = sum(o) / n
    bins = {}
    for fi, oi in zip(f, o):
        k = min(int(fi * (nbins - 1) + 0.5), nbins - 1)      # nearest of 0, 0.1, ..., 1
        bins.setdefault(k, []).append((fi, oi))
    rel = res = wbv = wbc = 0.0
    for pts in bins.values():
        nk = len(pts)
        fk = sum(p[0] for p in pts) / nk
        ok = sum(p[1] for p in pts) / nk
        rel += nk * (fk - ok) ** 2
        res += nk * (ok - obar) ** 2
        wbv += sum((p[0] - fk) ** 2 for p in pts)
        wbc += sum((p[0] - fk) * (p[1] - ok) for p in pts)
    bs = sum((fi - oi) ** 2 for fi, oi in zip(f, o)) / n
    return bs, rel / n, res / n, obar * (1 - obar), wbv / n, wbc / n

def forecaster(n, rounded):
    f, o = [], []
    for _ in range(n):
        p = rng.betavariate(2, 3)                    # the true chance of this event
        q = min(max(p + rng.gauss(0, 0.08), 0.0), 1.0)  # the forecaster's noisy judgement
        if rounded:
            q = round(q * 10) / 10                   # says only 0 %, 10 %, ..., 100 %
        f.append(q)
        o.append(1 if rng.random() < p else 0)
    return f, o

if __name__ == "__main__":
    for rounded in (True, False):
        f, o = forecaster(5000, rounded)
        bs, rel, res, unc, wbv, wbc = murphy(f, o)
        print("forecasts in steps of 10 %" if rounded else "continuous forecasts, 11 bins")
        print(f"  BS = {bs:.5f}   REL = {rel:.5f}   RES = {res:.5f}   UNC = {unc:.5f}")
        tot = rel - res + unc
        print(f"  REL - RES + UNC = {tot:.5f}   gap = {bs - tot:+.2e}")
        if not rounded:
            print(f"  WBV = {wbv:.5f}  WBC = {wbc:.5f}  REL - RES + UNC + WBV - 2 WBC = "
                  f"{rel - res + unc + wbv - 2 * wbc:.5f}")
# forecasts in steps of 10 %
#   BS = 0.20955   REL = 0.00146   RES = 0.03396   UNC = 0.24204
#   REL - RES + UNC = 0.20955   gap = +2.78e-17
# continuous forecasts, 11 bins
#   BS = 0.20418   REL = 0.00089   RES = 0.03614   UNC = 0.23988
#   REL - RES + UNC = 0.20464   gap = -4.51e-04
#   WBV = 0.00079  WBC = 0.00062  REL - RES + UNC + WBV - 2 WBC = 0.20418

When forecasts take only 11 discrete values, $\text{REL} – \text{RES} + \text{UNC}$ differs from the Brier score by about $3 \times 10^{-17}$ , mere rounding error: the decomposition is an exact identity. When forecasts are continuous and binned afterwards, the three terms add up to about 0.00045 more than the Brier score; adding back the within-bin variance and covariance makes them equal again. In practice the three-term decomposition is usually computed after binning, and it should be understood as an approximation.

Who is “perfectly calibrated”?

A common claim is that “someone who always says 50 % is perfectly calibrated, because he is never overconfident”. This holds only when the base rate of the events is exactly 50 %.

Calculation

20,000 events with a base rate of about 30 %. Five forecasters: always 50 %; always the base rate, 30 %; a “calibrated” forecaster who states each event’s true chance; an “overconfident” one who pushes the same judgement outwards (log-odds times 2); and an “underconfident” one who pulls it inwards (log-odds times 0.5, centred on the base rate). The last part computes the average REL, with the same binning, for a perfectly calibrated person who has made only 20 forecasts.

Calculation

# Five forecasters, 20,000 events with base rate 30 %: always 50 %, always 30 %,
# calibrated (states the true chance), overconfident (2 x logit), underconfident (0.5 x
# logit about 30 %). Last part: REL from only 20 forecasts of a perfectly calibrated
# person.
import math, random

rng = random.Random(1973)

def logit(p):
    return math.log(p / (1 - p))

def inv(x):
    return 1 / (1 + math.exp(-x))

def decompose(f, o, nb=11):
    n, obar = len(f), sum(o) / len(o)
    bins = {}
    for fi, oi in zip(f, o):
        bins.setdefault(round(fi * (nb - 1)), []).append((fi, oi))
    rel = res = 0.0
    table = []
    for k in sorted(bins):
        pts = bins[k]
        fk = sum(p[0] for p in pts) / len(pts)
        ok = sum(p[1] for p in pts) / len(pts)
        rel += len(pts) * (fk - ok) ** 2
        res += len(pts) * (ok - obar) ** 2
        table.append((fk, ok, len(pts)))
    bs = sum((a - b) ** 2 for a, b in zip(f, o)) / n
    return bs, rel / n, res / n, obar * (1 - obar), table

def cases(n=20000):
    ps = [min(max(rng.betavariate(1.2, 2.8), 0.005), 0.995) for _ in range(n)]
    o = [1 if rng.random() < p else 0 for p in ps]
    fc = {"always 50 %": [0.5] * n,
          "always base rate": [0.3] * n,
          "calibrated": ps,
          "overconfident": [inv(2 * logit(p)) for p in ps],
          "underconfident": [inv(0.5 * logit(p) + 0.5 * logit(0.3)) for p in ps]}
    return {k: decompose(v, o) for k, v in fc.items()}

def small_sample_rel(m=20, reps=4000):
    vals = []
    for _ in range(reps):
        ps = [rng.betavariate(1.2, 2.8) for _ in range(m)]
        f = [round(p * 10) / 10 for p in ps]                 # states its chance, rounded
        o = [1 if rng.random() < q else 0 for q in f]        # events happen at that rate
        vals.append(decompose(f, o)[1])
    vals.sort()
    return sum(vals) / reps, vals[reps // 2]

if __name__ == "__main__":
    res = cases()
    print("forecaster        |   BS    REL    RES    UNC")
    for k, (bs, rel, rs, unc, _) in res.items():
        print(f"{k:17s} | {bs:.4f} {rel:.4f} {rs:.4f} {unc:.4f}")
    print("overconfident: mean forecast -> observed frequency (bins with n >= 200):")
    cells = [f"{f:.2f}->{ob:.2f}" for f, ob, n in res["overconfident"][4] if n >= 200]
    print("  " + "  ".join(cells[:6]))
    print("  " + "  ".join(cells[6:]))
    mean, med = small_sample_rel()
    print(f"perfectly calibrated, only 20 forecasts: REL mean {mean:.4f},"
          f" median {med:.4f} (truth 0)")
# forecaster        |   BS    REL    RES    UNC
# always 50 %       | 0.2500 0.0403 0.0000 0.2097
# always base rate  | 0.2097 0.0000 0.0000 0.2097
# calibrated        | 0.1677 0.0001 0.0413 0.2097
# overconfident     | 0.1797 0.0115 0.0407 0.2097
# underconfident    | 0.1757 0.0075 0.0394 0.2097
# overconfident: mean forecast -> observed frequency (bins with n >= 200):
#   0.02->0.09  0.09->0.25  0.20->0.33  0.30->0.37  0.40->0.44  0.50->0.49
#   0.60->0.55  0.70->0.59  0.80->0.67  0.90->0.74  0.97->0.88
# perfectly calibrated, only 20 forecasts: REL mean 0.0631, median 0.0583 (truth 0)

Reliability diagram for the five forecasters. The horizontal axis is the mean probability of a bin of forecasts, the vertical axis the fraction of those events that happened; the diagonal is perfect calibration. The overconfident forecaster’s curve is flatter than the diagonal: of the events he calls 90 %, only about 74 % happen, while of those he calls 9 %, 25 % happen. Always-50 % and always-base-rate each give a single point.

The always-50 % forecaster has REL 0.0403, about $(0.5 – 0.3)^2$ : he systematically overstates this kind of event. The always-base-rate forecaster has REL 0, perfectly calibrated, but RES is also 0, and his Brier score equals the uncertainty, 0.2097: a score available to someone who understands nothing. The calibrated forecaster who states the true chances has REL close to 0, RES 0.0413, and a Brier score of 0.1677.

The overconfident forecaster has the same information as the calibrated one and almost the same RES (0.0407), but REL rises to 0.0115 and the Brier score worsens to 0.1797. Figure 1 shows the typical shape of overconfidence: of the events he calls 90 %, about 74 % happen, and of those he calls 97 %, about 88 %; at the other end, events he calls 9 % in fact happen about 25 % of the time. The underconfident forecaster errs in the opposite direction, with REL 0.0075. Calibration and resolution are thus two different abilities: being well calibrated is not the same as being useful, and high resolution is not the same as stating the right numbers.

The last line flags a measurement problem: for a perfectly calibrated person who has made only 20 forecasts, REL computed with 11 bins averages 0.063 (median 0.058), far from the true value of 0. The reason is that each bin holds only one or two forecasts, so the observed frequencies jump between 0 and 1. Computing a Murphy decomposition for an individual from very few forecasts therefore systematically overstates calibration error, and the trial in Chapter 8 does not use individual REL as its primary outcome.

Proper scoring rules

A scoring rule is strictly proper if the forecaster’s expected score is best only when he reports the probability he actually believes. This matters: a rule that rewards “saying something more extreme” will train honest forecasters to be overconfident.

Calculation

Derivation. Suppose you believe the event has probability $p$ and you report $q$ . The expected Brier score (lower is better) is $E(q) = p(1-q)^2 + (1-p)q^2$ ; differentiating, $E'(q) = -2p(1-q) + 2(1-p)q = 2(q – p)$ , which vanishes only at $q = p$ , and the second derivative is positive: honesty is the unique best report. The logarithmic score, $-\ln q$ if the event happens and $-\ln(1-q)$ if not, has expectation $-p\ln q – (1-p)\ln(1-q)$ , whose derivative $-p/q + (1-p)/(1-q)$ likewise vanishes only at $q = p$ (Good 1952; Gneiting and Raftery 2007). The equally natural-looking absolute error $|q – o|$ has expectation $p(1-q) + (1-p)q$ , linear in $q$ : whenever $p > 0.5$ , the closer the report to 1 the better.

Calculation

# Expected penalty for reporting q when you believe p = 0.70 (lower is better):
# quadratic (Brier), logarithmic, absolute. Strictly proper = honest q = p is the unique
# best.
import math

P = 0.70

def e_quad(q, p=P):
    return p * (1 - q) ** 2 + (1 - p) * q ** 2

def e_log(q, p=P):
    return -p * math.log(q) - (1 - p) * math.log(1 - q)

def e_abs(q, p=P):
    return p * (1 - q) + (1 - p) * q

if __name__ == "__main__":
    grid = [i / 100 for i in range(1, 100)]
    print(" report q | quadratic | logarithmic | absolute")
    for q in (0.50, 0.60, 0.70, 0.80, 0.90, 0.99):
        print(f"   {q:.2f}   |  {e_quad(q):.4f}   |   {e_log(q):.4f}    |  {e_abs(q):.4f}")
    for name, fn in (("quadratic", e_quad), ("logarithmic", e_log), ("absolute", e_abs)):
        best = min(grid, key=fn)
        print(f"{name:11s}: best report on the grid 0.01..0.99 is q = {best:.2f}")
    print(f"log score if you said 0.99 and it did not happen: {-math.log(0.01):.2f}"
          f" (vs {-math.log(0.30):.2f} after saying 0.70)")
#  report q | quadratic | logarithmic | absolute
#    0.50   |  0.2500   |   0.6931    |  0.5000
#    0.60   |  0.2200   |   0.6325    |  0.4600
#    0.70   |  0.2100   |   0.6109    |  0.4200
#    0.80   |  0.2200   |   0.6390    |  0.3800
#    0.90   |  0.2500   |   0.7645    |  0.3400
#    0.99   |  0.2941   |   1.3886    |  0.3040
# quadratic  : best report on the grid 0.01..0.99 is q = 0.70
# logarithmic: best report on the grid 0.01..0.99 is q = 0.70
# absolute   : best report on the grid 0.01..0.99 is q = 0.99
# log score if you said 0.99 and it did not happen: 4.61 (vs 1.20 after saying 0.70)

When you believe 70 %, the Brier and logarithmic scores are both best for a report of 0.70; the absolute error is best for a report of 0.99 and is not a proper scoring rule. The two proper rules punish “saying too much” to different degrees: if you say 99 % and the event does not happen, the logarithmic penalty is 4.61, against 1.20 had you said 70 %. The logarithmic score is extremely harsh on confident errors (saying 100 % and being wrong costs infinity), while the Brier score is bounded. Forecasting tournaments mostly use the Brier score, partly because one slip cannot leave a forecaster unable ever to recover.

Kinds of overconfidence

Evidence

Basic findings of calibration research. Early calibration studies found that on general-knowledge questions, when people say “I am 90 % sure”, they are often right clearly less than 90 % of the time; they are overconfident on hard questions and may even be underconfident on very easy ones (the “hard–easy effect”) (Lichtenstein, Fischhoff, and Phillips 1982). Alpert and Raiffa asked students for 98 % confidence intervals for unknown quantities; the true values fell outside the intervals far more often than the nominal 2 % (Alpert and Raiffa 1982). Moore and Healy distinguished three things often lumped together as “overconfidence”: overestimating one’s own performance, believing one is better than others, and being too precise in one’s estimates (intervals too narrow), and reported that the third is more persistent than the other two (Moore and Healy 2008). In three experiments by Haran, Moore and Morewedge, people’s 90 % confidence intervals contained the true value only 29 % to 74 % of the time (Haran, Moore, and Morewedge 2010).

Chapter summary

The Brier score is the mean squared difference between forecasts and outcomes; Murphy split it into reliability (calibration), resolution and uncertainty. The decomposition is exact when the forecasts within a bin are equal; after binning, within-bin variance and covariance terms appear. Always saying 50 % is perfectly calibrated only when the base rate is 50 %; in general, always stating the base rate is perfectly calibrated with zero resolution. An overconfident forecaster can have the same resolution as a calibrated one and still score worse by saying too much. The Brier and logarithmic scores are strictly proper; the absolute error is not. Computing an individual’s calibration error from very few forecasts overstates it. The next chapter discusses the best-known error in probabilistic judgement: neglecting the base rate.

References

Alpert, Marc, and Howard Raiffa. 1982. “A Progress Report on the Training of Probability Assessors.” In Judgment Under Uncertainty: Heuristics and Biases, edited by Daniel Kahneman, Paul Slovic, and Amos Tversky, 294–305. Cambridge: Cambridge University Press.

Brier, Glenn W. 1950. “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review 78 (1): 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.

Gneiting, Tilmann, and Adrian E. Raftery. 2007. “Strictly Proper Scoring Rules, Prediction, and Estimation.” Journal of the American Statistical Association 102 (477): 359–78. https://doi.org/10.1198/016214506000001437.

Good, I. J. 1952. “Rational Decisions.” Journal of the Royal Statistical Society, Series B (Methodological) 14 (1): 107–14.

Haran, Uriel, Don A. Moore, and Carey K. Morewedge. 2010. “A Simple Remedy for Overprecision in Judgment.” Judgment and Decision Making 5 (7): 467–76.

Lichtenstein, Sarah, Baruch Fischhoff, and Lawrence D. Phillips. 1982. “Calibration of Probabilities: The State of the Art to 1980.” In Judgment Under Uncertainty: Heuristics and Biases, edited by Daniel Kahneman, Paul Slovic, and Amos Tversky, 306–34. Cambridge: Cambridge University Press.

Moore, Don A., and Paul J. Healy. 2008. “The Trouble with Overconfidence.” Psychological Review 115 (2): 502–17. https://doi.org/10.1037/0033-295X.115.2.502.

Murphy, Allan H. 1973. “A New Vector Partition of the Probability Score.” Journal of Applied Meteorology 12 (4): 595–600. https://doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.

Stephenson, David B., Caio A. S. Coelho, and Ian T. Jolliffe. 2008. “Two Extra Components in the Brier Score Decomposition.” Weather and Forecasting 23 (4): 752–57. https://doi.org/10.1175/2007WAF2006116.1.

Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence: From Base-Rate Neglect and Signal Detection Theory to Superforecasting, Read Beside Classical Chinese Strategic Calculation [概率判断、布莱尔分数校准与群体智慧的统计理论] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23764998 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).