Chapter 6 The Statistics of Collective Intelligence

Weighing the ox

In the autumn of 1906 a weight-judging competition was held at a livestock show in Plymouth, England: how much would an ox weigh once slaughtered and dressed? Entrants bought a ticket and wrote down their estimate. Francis Galton obtained the tickets afterwards and in March 1907 published a short note in Nature entitled “Vox populi”.

Evidence

Galton’s figures and their correction. After discarding 13 illegible tickets, Galton analysed 787 estimates. He reported the median: 1,207 lb, against an actual weight of 1,198 lb, an error of under 1 % (Galton 1907b). Replying to a reader’s letter, he then reported the mean of all the estimates as 1,197 lb (Galton 1907a). More than a century later Wallis re-examined Galton’s surviving papers and found that the median should have been 1,208 lb (the draft had 1,208; the published note mistakenly gave 1,207), and that the actual weight given in the organiser’s letter converts to 1,197 lb, not 1,198 lb. So the correct statement is that the median was out by 11 lb (about 0.9 %) and the mean was exactly the actual weight (Wallis 2014).

Surowiecki’s The Wisdom of Crowds opens with this story and summarises the conditions for accurate group judgement in four points: diversity of opinion, independence, decentralisation (each person drawing on local knowledge) and a suitable way of aggregating (Surowiecki 2004). Of these, statistics can say most about the first two.

The diversity prediction theorem

The political scientist Scott Page calls a simple algebraic identity the diversity prediction theorem (Page 2007). Let $n$ people’s estimates of a true value $\theta$ be $s_1, \dots, s_n$ and the crowd mean $c = \frac{1}{n}\sum s_i$ . Then

$\underbrace{(c – \theta)^2}_{\text{crowd error}} = \underbrace{\frac{1}{n}\sum_i (s_i – \theta)^2}_{\text{average individual error}} – \underbrace{\frac{1}{n}\sum_i (s_i – c)^2}_{\text{diversity}}.$

Calculation

Derivation. Write $s_i – \theta$ as $(s_i – c) + (c – \theta)$ , square and average: $\frac{1}{n}\sum(s_i – \theta)^2 = \frac{1}{n}\sum(s_i – c)^2 + (c – \theta)^2 + \frac{2}{n}(c – \theta)\sum(s_i – c)$ . In the last term $\sum(s_i – c) = 0$ ; rearranging gives the result. It is an identity that holds for any numbers.

The program generates 787 simulated estimates (not Galton’s data), each equal to the true weight of 1,197 lb plus an error shared by everyone plus an individual error (standard deviation 75 lb), and then tests the identity on 10,000 random sets of numbers.

Calculation

# Diversity prediction theorem: crowd error = average individual error - diversity. 787
# simulated guesses (not Galton's data): truth 1,197 lb + shared error + private N(0,
# 75).
import random

rng = random.Random(1907)

def parts(s, theta):
    n = len(s)
    c = sum(s) / n
    crowd = (c - theta) ** 2
    avg = sum((x - theta) ** 2 for x in s) / n
    div = sum((x - c) ** 2 for x in s) / n
    return c, crowd, avg, div

if __name__ == "__main__":
    theta = 1197
    for shared in (0, 30):
        s = [theta + shared + rng.gauss(0, 75) for _ in range(787)]
        c, crowd, avg, div = parts(s, theta)
        beat = sum((x - theta) ** 2 < crowd for x in s)
        print(f"shared error {shared:+d} lb: crowd mean {c:.1f} lb;"
              f" {beat} of 787 individuals closer than it")
        print(f"   crowd error {crowd:.2f} = average error {avg:.2f} - diversity {div:.2f}"
              f"  (difference {avg - div:.2f})")
    worst = 0.0
    for _ in range(10000):
        n = rng.randint(2, 30)
        s = [rng.uniform(-100, 100) for _ in range(n)]
        _, crowd, avg, div = parts(s, rng.uniform(-50, 50))
        worst = max(worst, abs(crowd - (avg - div)))
    print(f"identity on 10,000 random sets of guesses: largest discrepancy {worst:.1e}")
# shared error +0 lb: crowd mean 1197.2 lb; 2 of 787 individuals closer than it
#    crowd error 0.04 = average error 5706.59 - diversity 5706.54  (difference 0.04)
# shared error +30 lb: crowd mean 1230.4 lb; 244 of 787 individuals closer than it
#    crowd error 1113.20 = average error 7159.50 - diversity 6046.30  (difference 1113.20)
# identity on 10,000 random sets of guesses: largest discrepancy 3.6e-12

With no shared error, the crowd mean is 1,197.2 lb, the crowd error only 0.04 (square pounds), and only 2 of the 787 individuals are closer to the truth than the crowd mean. When everyone is misled by the same +30 lb, the crowd mean becomes 1,230.4 lb and the crowd error rises to 1,113.20; now 244 individuals beat the crowd. In both cases the identity holds exactly; across 10,000 random sets of numbers the largest discrepancy is $3.6 \times 10^{-12}$ , rounding error.

The theorem is often cited as evidence that “diversity trumps ability”, which calls for caution: it is an identity, not an empirical finding; it guarantees that the crowd’s error is never larger than the average individual error, but says nothing about when the crowd beats the best individual, and “diversity” is merely the spread of estimates, which may reflect different information or mere noise. Hong and Page reported simulations in which, in a certain problem-solving model, randomly chosen diverse teams beat teams of the “most able” (Hong and Page 2004); Thompson criticised the conclusion as depending on particular features of the model and not supporting general claims about real groups (Thompson 2014). This book uses only the identity itself and the simulations below under explicit assumptions.

Correlated errors: when does the crowd beat the expert?

The crowd mean is accurate because individual errors cancel in averaging. But only the uncorrelated part of the errors cancels; the part everyone shares does not shrink however many people are averaged.

Calculation

Let each non-expert’s error have variance 1 and any two people’s errors correlation $\rho$ ; let the expert’s error variance be 0.25 (error standard deviation half that of a non-expert). The mean squared error of the average of $n$ people is

$\text{MSE}_n = \rho + \frac{1 – \rho}{n},$

which tends to $\rho$ , not to 0, as $n$ grows. The effective number of people is $n_{\text{eff}} = n/[1 + (n-1)\rho]$ . The program compares theory with simulation and computes how many people the crowd needs to beat the expert. The numbers are illustrative.

Calculation

# Crowd of n non-experts (error variance 1, pairwise correlation rho) against one expert
# (error variance 0.25): MSE(crowd) = rho + (1 - rho)/n. Illustrative numbers.
import math, random

rng = random.Random(2004)
V_EXPERT = 0.25

def mse_theory(n, rho):
    return rho + (1 - rho) / n

def mse_sim(n, rho, reps=20000):
    tot = 0.0
    for _ in range(reps):
        shared = rng.gauss(0, math.sqrt(rho))
        m = sum(shared + rng.gauss(0, math.sqrt(1 - rho)) for _ in range(n)) / n
        tot += m * m
    return tot / reps

def n_needed(rho, v=V_EXPERT):
    if rho >= v:
        return None
    return math.floor((1 - rho) / (v - rho) + 1e-9) + 1      # smallest n with MSE < v

if __name__ == "__main__":
    print("rho | crowd MSE, theory: n = 5, 20, 100 | simulated, n = 20 | n to beat expert")
    for rho in (0.0, 0.1, 0.2, 0.3, 0.5):
        th = [mse_theory(n, rho) for n in (5, 20, 100)]
        nn = n_needed(rho)
        need = f"{nn}" if nn else "never"
        print(f"{rho:.1f} |       {th[0]:.3f}  {th[1]:.3f}  {th[2]:.3f}        |"
              f"       {mse_sim(20, rho):.3f}       | {need}")
    for rho in (0.1, 0.3):
        print(f"n_eff of 100 forecasters at rho = {rho}: {100 / (1 + 99 * rho):.1f}")
# rho | crowd MSE, theory: n = 5, 20, 100 | simulated, n = 20 | n to beat expert
# 0.0 |       0.200  0.050  0.010        |       0.050       | 5
# 0.1 |       0.280  0.145  0.109        |       0.145       | 7
# 0.2 |       0.360  0.240  0.208        |       0.244       | 17
# 0.3 |       0.440  0.335  0.307        |       0.330       | never
# 0.5 |       0.600  0.525  0.505        |       0.522       | never
# n_eff of 100 forecasters at rho = 0.1: 9.2
# n_eff of 100 forecasters at rho = 0.3: 3.3

Mean squared error of the crowd mean against the number of people. With uncorrelated errors, 5 people beat the expert (black horizontal line); with an error correlation of 0.2, 17 are needed; at a correlation of 0.3 or more no number of people beats the expert, and the curve levels off near \rho. — Mean squared error of the crowd mean against the number of people. With uncorrelated errors, 5 people beat the expert (black horizontal line); with an error correlation of 0.2, 17 are needed; at a correlation of 0.3 or more no number of people beats the expert, and the curve levels off near $\rho$ .

With uncorrelated errors, the average of 5 non-experts beats the expert; at a correlation of 0.1, 7 are needed, and at 0.2, 17; at a correlation of 0.3, the average of 100 people still has a mean squared error of 0.307 and never beats the expert’s 0.25. At a correlation of 0.1, 100 people are worth only about 9.2 independent people; at 0.3, only about 3.3. So “the crowd beats the expert” is conditional: it requires the errors to be weakly enough correlated, and the expert not to be better than that limit.

Evidence

Social influence undermines the wisdom of crowds. Lorenz and colleagues had participants answer quantity-estimation questions over several rounds, either seeing or not seeing other people’s answers. After seeing others’ answers, the spread of estimates narrowed markedly, but the crowd mean did not become more accurate; at the same time participants’ confidence rose (Lorenz et al. 2011). In this chapter’s terms: social influence raised the correlation between errors and reduced diversity. By the diversity prediction theorem, if diversity falls while the crowd error stays the same, any fall in the average individual error is exactly offset by the loss of diversity: people moved towards each other but not, together, towards the truth, so the crowd error did not improve while people became more confident. “Echo chambers” are the extreme case: people who read the same sources and constantly exchange views have highly correlated errors, and however many they are, they are worth only a few independent judgements.

Why extremise after averaging?

Chapter 1 mentioned that the best aggregation algorithms in the forecasting tournaments “extremise” the averaged probabilities. This seems to contradict Chapter 2: should we not avoid overconfidence?

Evidence

Two reasons. Baron, Mellers, Tetlock, Stone and Ungar point out that the simple average of many people’s probabilities tends to sit too close to 50 %, and that pushing it towards 0 or 1 matches outcomes better. There are two reasons. First, probabilities are “compressed” near 0 and 1, so random error pulls the mean towards the middle; this does not affect the median, and arguably not the mean of the log-odds. Second, each forecaster rightly allows for the incompleteness of his own information, but together the crowd holds more information than any one member; simple averaging (including the median and the mean of log-odds) does not add up this dispersed information (Baron et al. 2014).

Calculation

A model of the second reason. The event has a prior of 50 %; each clue follows $N(+0.5, 1)$ if the event will happen and $N(-0.5, 1)$ otherwise, so each clue adds its own value to the log-odds. All 10 forecasters see the same public clue, and each has one private clue; each reports the correct Bayesian probability given what he saw, so each is individually well calibrated. Compared are: a single forecaster, the mean of probabilities, the mean of log-odds, the mean of log-odds multiplied by a factor $a$ (fitted on 20,000 training questions, scored on 20,000 new ones), and an ideal Bayesian who sees all 11 clues.

Calculation

# Shared information and extremising (Baron et al. 2014). Clues ~ N(+-0.5, 1); 10
# forecasters share one public clue and each has one private clue; each reports a
# correct Bayesian probability. Factor a fitted on 20,000 training questions, scored on
# 20,000 new ones.
import math, random

rng = random.Random(2014)
K, MU = 10, 0.5

def inv(z):
    return 1 / (1 + math.exp(-z))

def questions(n):
    out = []
    for _ in range(n):
        y = 1 if rng.random() < 0.5 else 0
        m = MU if y else -MU
        pub = rng.gauss(m, 1)
        priv = [rng.gauss(m, 1) for _ in range(K)]
        logits = [2 * MU * (pub + x) for x in priv]          # each forecaster's posterior
        ideal = 2 * MU * (pub + sum(priv))
        out.append((y, logits, ideal))
    return out

def brier(pairs):
    return sum((p - y) ** 2 for p, y in pairs) / len(pairs)

def score(qs, a):
    return brier([(inv(a * sum(l) / K), y) for y, l, _ in qs])

if __name__ == "__main__":
    train, test = questions(20000), questions(20000)
    a_best = min((i / 20 for i in range(20, 161)), key=lambda a: score(train, a))
    print(f"extremising factor fitted on training questions: a = {a_best:.2f}")
    rows = [("a single forecaster", brier([(inv(l[0]), y) for y, l, _ in test])),
            ("mean of probabilities",
             brier([(sum(inv(z) for z in l) / K, y) for y, l, _ in test])),
            ("mean of log-odds", score(test, 1.0)),
            (f"extremised, a = {a_best:.2f}", score(test, a_best)),
            ("ideal: all 11 clues pooled", brier([(inv(z), y) for y, _, z in test]))]
    for name, b in rows:
        print(f"  {name:28s} Brier {b:.4f}")
    print("mean log-odds = public + (1/10) x sum of private clues; ideal counts each once")
# extremising factor fitted on training questions: a = 1.75
#   a single forecaster          Brier 0.1617
#   mean of probabilities        Brier 0.1354
#   mean of log-odds             Brier 0.1290
#   extremised, a = 1.75         Brier 0.1187
#   ideal: all 11 clues pooled   Brier 0.0366
# mean log-odds = public + (1/10) x sum of private clues; ideal counts each once

A single forecaster’s Brier score is 0.1617; the mean of probabilities gives 0.1354, the mean of log-odds 0.1290; the fitted factor is $a = 1.75$ , and extremising brings the score down to 0.1187. But the ideal Bayesian scores 0.0366, far better than any aggregate. The reason is in the last line: the mean of log-odds counts the public clue once and each of the 10 private clues only one tenth of a time, whereas the ideal counts every clue once. A single extremising factor cannot separate public from private information, so it can only partly make up the difference. This also shows that how far to extremise depends on how much information forecasters share: the more they share, the less one should extremise.

Chapter summary

The crowd median Galton reported in 1907 was 1,207 lb (it should have been 1,208); the mean of 1,197 lb came later and exactly equals the corrected actual weight. The diversity prediction theorem is an identity: crowd error equals average individual error minus diversity; it guarantees that the crowd is no worse than the average individual, not that it beats the best one. With correlated errors, averaging removes only the uncorrelated part; once the correlation exceeds the expert’s relative error, no number of people beats the expert. Social influence reduces diversity and raises confidence without improving accuracy. Moderately extremising the average of many probabilities can improve scores, because each person has only part of the information; how far to go depends on how much is shared. The next chapter reads this modern research beside the “temple calculation” of the Chinese military classics.

References

Baron, Jonathan, Barbara A. Mellers, Philip E. Tetlock, Eric Stone, and Lyle H. Ungar. 2014. “Two Reasons to Make Aggregated Probability Forecasts More Extreme.” Decision Analysis 11 (2): 133–45. https://doi.org/10.1287/deca.2014.0293.

Galton, Francis. 1907a. “The Ballot-Box.” Nature 75: 509–10.

———. 1907b. “Vox Populi.” Nature 75: 450–51. https://doi.org/10.1038/075450a0.

Hong, Lu, and Scott E. Page. 2004. “Groups of Diverse Problem Solvers Can Outperform Groups of High-Ability Problem Solvers.” Proceedings of the National Academy of Sciences 101 (46): 16385–89. https://doi.org/10.1073/pnas.0403723101.

Lorenz, Jan, Heiko Rauhut, Frank Schweitzer, and Dirk Helbing. 2011. “How Social Influence Can Undermine the Wisdom of Crowd Effect.” Proceedings of the National Academy of Sciences 108 (22): 9020–25. https://doi.org/10.1073/pnas.1008636108.

Page, Scott E. 2007. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton, NJ: Princeton University Press.

Surowiecki, James. 2004. The Wisdom of Crowds. New York: Doubleday.

Thompson, Abigail. 2014. “Does Diversity Trump Ability? An Example of the Misuse of Mathematics in the Social Sciences.” Notices of the American Mathematical Society 61 (9): 1024–30. https://doi.org/10.1090/noti1163.

Wallis, Kenneth F. 2014. “Revisiting Francis Galton’s Forecasting Competition.” Statistical Science 29 (3): 420–24. https://doi.org/10.1214/14-STS468.

Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence: From Base-Rate Neglect and Signal Detection Theory to Superforecasting, Read Beside Classical Chinese Strategic Calculation [概率判断、布莱尔分数校准与群体智慧的统计理论] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23764998 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).