What does Calibration and Collective Intelligence conclude, in numbers, about “Practice and a Pre-registered Trial”?

Answer: The four-step calibration checklist turns the conclusions of earlier chapters into four recordable steps: replace vague words with numbers, find the base rate of an outside reference class first, write down update triggers in advance, and at the end of the quarter review with the Brier score against the base rates. The fictional example shows that intuitive numbers can be worse than stating the base rates alone, and that updating as agreed in advance helps, though 12 questions in one quarter are not enough to conclude anything. The trial protocol compares a 15-minute calibration micro-training with time-matched reading on logical thinking; its single primary outcome is the calibration error \(|H - 0.80|\) of 80 % intervals on 20 items, with 133 analysable and 148 recruited participants per arm; the decision rule, including thresholds for a manipulation check and for exclusions, is written as code and published in advance; the results will be published whether positive or negative.

Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.80, 133, 148, 20

simulated control: mean CE 0.304, SD 0.173; hit chance 0.57 gives mean CE 0.243
analysable participants per arm | rho0.0 | 0.3 | 0.5
CE difference 0.05 (d0.29) | 253 | 231 | 190
CE difference 0.06 (d0.35) | 176 | 161 | 133
CE difference 0.08 (d0.46) | 100 | 91 | 75
chosen0.06, rho = 0.5 -> 133 per arm
allowing 10 % loss (incomplete sessions, failed attention checks):
recruit 148 per arm, 296 in all
check without covariate176 per arm, simulated power 0.93
floor: a perfectly calibrated person (h0.80) has expected CE 0.070 on 20 items
Show the full program
# Sample size. Primary outcome CE = |H - 0.80| over 20 items (H = share of 80 %
# intervals that hit). ANCOVA on pre-test CE (correlation rho); alpha 0.05 two-sided,
# power 0.90. SD of CE from a simulated control arm: hit chance ~ Beta(mean 0.50, SD
# 0.15), an assumption.
import math, random
from statistics import NormalDist, mean, stdev

rng = random.Random(15)
N = NormalDist()

def beta_ab(m, sd):
    k = m * (1 - m) / sd ** 2 - 1
    return m * k, (1 - m) * k

def ce_sample(n, m, sd=0.15, items=20):
    a, b = beta_ab(m, sd)
    out = []
    for _ in range(n):
        h = rng.betavariate(a, b)
        hits = sum(rng.random() < h for _ in range(items))
        out.append(abs(hits / items - 0.80))
    return out

def per_arm(delta, sigma, rho, alpha=0.05, power=0.90):
    z = N.inv_cdf(1 - alpha / 2) + N.inv_cdf(power)
    return math.ceil(2 * z * z * sigma ** 2 * (1 - rho ** 2) / delta ** 2) + 1

if __name__ == "__main__":
    ctrl, trt = ce_sample(20000, 0.50), ce_sample(20000, 0.57)
    sigma = stdev(ctrl)
    print(f"simulated control: mean CE {mean(ctrl):.3f}, SD {sigma:.3f};"
          f" hit chance 0.57 gives mean CE {mean(trt):.3f}")
    print("analysable participants per arm | rho = 0.0 | 0.3 | 0.5")
    for d in (0.05, 0.06, 0.08):
        ns = [per_arm(d, sigma, r) for r in (0.0, 0.3, 0.5)]
        print(f"   CE difference {d:.2f} (d = {d / sigma:.2f})  |"
              f" {ns[0]:9d} | {ns[1]:3d} | {ns[2]:3d}")
    n = per_arm(0.06, sigma, 0.5)
    rec = math.ceil(n / 0.90)
    print(f"chosen: 0.06, rho = 0.5 -> {n} per arm")
    print(f"   allowing 10 % loss (incomplete sessions, failed attention checks):")
    print(f"   recruit {rec} per arm, {2 * rec} in all")
    n0, reps, hits = per_arm(0.06, sigma, 0.0), 1000, 0
    for _ in range(reps):
        a, b = ce_sample(n0, 0.50), ce_sample(n0, 0.57)
        se = math.sqrt(stdev(a) ** 2 / n0 + stdev(b) ** 2 / n0)
        hits += abs(mean(a) - mean(b)) / se > N.inv_cdf(0.975)
    print(f"check without covariate: {n0} per arm, simulated power {hits / reps:.2f}")
    pk = [math.comb(20, k) * 0.8 ** k * 0.2 ** (20 - k) for k in range(21)]
    floor = sum(p * abs(k / 20 - 0.8) for k, p in enumerate(pk))
    print(f"floor: a perfectly calibrated person (h = 0.80) has expected CE {floor:.3f}"
          " on 20 items")

Single source and citation

Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 8: Practice and a Pre-registered Trial. 5D Library. https://doi.org/10.5281/zenodo.23764998

DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026026,
  author    = {Fan, Yafu},
  title     = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23764998},
  url       = {https://doi.org/10.5281/zenodo.23764998}
}

Other chapter answers in this book