Chapter 8 Practice and a Pre-registered Trial

Part 1 A four-step probability-calibration checklist

This checklist is for anyone who regularly has to forecast (project leads, researchers, editors, students), to be used over a quarter. It needs no special tools, only a dated log.

Replace vague words with numbers. Write every “possibly”, “probably” and “shouldn’t be a problem” as a number between 0 and 100 %. The same word stands for very different probabilities for different people, and even for the same person on different occasions (Wallsten et al. 1986); without a number there is nothing to score. (Chapter 2)
Find the base rate of an outside reference class first. Before looking at the details of the case, ask: in similar situations, how often does this kind of thing happen? Write the base rate beside the forecast, then write down why, and how far, you depart from it. (Chapters 3 and 5)
Write down your update triggers in advance. For each forecast, write: “If such-and-such news arrives, I will move the probability to so much.” When the news arrives, update in small agreed steps; do not swing wildly on one headline, and do not stay put because you dislike admitting error. (Chapter 1)
Review your Brier score each quarter. When the quarter ends and the questions resolve, compute your Brier score and compare it with the score of “stating the base rate only”; take your forecasts of 70 % or more and see how many of them happened. (Chapter 2)

Calculation

The checklist applied to one fictional forecaster’s 12 questions in a quarter. Questions, numbers and outcomes are invented. Question 7 concerns an interest-rate decision; it serves only to demonstrate scoring and is not a market view or investment advice of any kind.

Calculation

# Four-step checklist on one FICTIONAL forecaster's quarter (12 invented questions).
# Question 7 (interest rates) is a scoring exercise, not a market view or investment
# advice. Columns: question, vague word, base rate, first forecast, trigger fired,
# final, outcome.
Q = [("project A delivered by 30 June",     "likely",      0.35, 0.75, True, 0.55, 0),
     ("new hire accepts the offer",         "probably",    0.70, 0.80, False, 0.80, 1),
     ("supplier price rise above 5 %",      "might",       0.30, 0.40, True, 0.65, 1),
     ("app passes review first time",       "likely",      0.60, 0.85, False, 0.85, 1),
     ("conference paper accepted",          "good chance", 0.25, 0.60, False, 0.60, 0),
     ("quarterly sales above last quarter", "probably",    0.55, 0.70, True, 0.50, 0),
     ("central bank raises its rate",       "unlikely",    0.20, 0.15, True, 0.40, 1),
     ("office move completed in May",       "almost sure", 0.40, 0.95, True, 0.70, 0),
     ("survey response rate above 2/3",     "doubtful",    0.15, 0.10, False, 0.10, 0),
     ("grant renewed",                      "likely",      0.50, 0.70, False, 0.70, 1),
     ("pilot customer signs contract",      "might",       0.30, 0.45, True, 0.30, 0),
     ("website redesign live before July",  "probably",    0.45, 0.80, True, 0.60, 1)]

def brier(fs, os_):
    return sum((f - o) ** 2 for f, o in zip(fs, os_)) / len(fs)

if __name__ == "__main__":
    o = [q[6] for q in Q]
    words = {}
    for q in Q:
        words.setdefault(q[1], []).append(q[3])
    print("step 1 - the same word, different numbers:")
    for w, v in words.items():
        if len(v) > 1:
            print(f"   '{w}': " + ", ".join(f"{x:.0%}" for x in v))
    gap = [q[3] - q[2] for q in Q]
    print(f"step 2 - first forecast minus base rate: mean {sum(gap) / len(gap):+.2f};"
          f" above the base rate in {sum(g > 0 for g in gap)} of 12")
    half = [q[2] + 0.5 * (q[3] - q[2]) for q in Q]
    print(f"   had each first forecast been moved halfway back to its base rate: Brier"
          f" {brier(half, o):.3f}")
    fired = [q for q in Q if q[4]]
    right = sum((q[5] - q[3]) * (q[6] - 0.5) > 0 for q in fired)
    print(f"step 3 - triggers fired in {len(fired)} questions;"
          f" the update moved towards the outcome in {right}")
    bs_base, bs_first, bs_final = (brier([q[i] for q in Q], o) for i in (2, 3, 5))
    print(f"step 4 - Brier: base rates only {bs_base:.3f}; first forecasts {bs_first:.3f};"
          f" final forecasts {bs_final:.3f}")
    sk1, sk2 = 1 - bs_first / bs_base, 1 - bs_final / bs_base
    print(f"   skill vs base rates (1 - BS / BS_base): first {sk1:+.0%}, final {sk2:+.0%}")
    hi = [q for q in Q if q[3] >= 0.7]
    mh = sum(q[3] for q in hi) / len(hi)
    print(f"   first forecasts of 70 % or more: {len(hi)}, mean {mh:.0%},"
          f" happened {sum(q[6] for q in hi)} of {len(hi)}")
# step 1 - the same word, different numbers:
#    'likely': 75%, 85%, 70%
#    'probably': 80%, 70%, 80%
#    'might': 40%, 45%
# step 2 - first forecast minus base rate: mean +0.21; above the base rate in 10 of 12
#    had each first forecast been moved halfway back to its base rate: Brier 0.252
# step 3 - triggers fired in 7 questions; the update moved towards the outcome in 6
# step 4 - Brier: base rates only 0.224; first forecasts 0.317; final forecasts 0.191
#    skill vs base rates (1 - BS / BS_base): first -41%, final +15%
#    first forecasts of 70 % or more: 7, mean 79%, happened 4 of 7

This fictional record shows several things. Step 1: the same word “likely” sometimes meant 70 % and sometimes 85 % to this forecaster; without numbers such inconsistency would never come to light. Step 2: his first, intuitive numbers were on average 0.21 above the base rate, and above it in 10 of 12 questions, the classic inside view. Step 3: the pre-written triggers fired in 7 questions, and 6 of those updates moved in the right direction. Step 4: stating only the base rates would have scored 0.224; the first intuitive numbers scored 0.317, 41 % worse than the base rates; the final forecasts after trigger-based updates scored 0.191, 15 % better than the base rates. Of the 7 first forecasts of 70 % or more, averaging 79 %, only 4 happened.

One more line deserves attention: mechanically pulling each first forecast halfway back towards its base rate would have scored 0.252, better than intuition but still worse than the base rates alone. Twelve questions are too few, and any one quarter’s score contains a great deal of luck; the checklist earns its keep by building up a record over many quarters, not by drawing conclusions from one.

Part 2 A pre-registered randomised controlled trial protocol

Question

Compared with spending the same time reading standard material on logical thinking, does a 15-minute “calibration micro-training” (natural-frequency conversion + anchoring on an outside reference class + Brier-score and interval-hit feedback) make adults better calibrated on subsequent probabilistic-judgement and interval-estimation tasks?

Choice of primary outcome

There is a single primary outcome: interval calibration error, $CE = |H – 0.80|$ , where $H$ is the proportion of the 20 post-test items for which the participant’s own 80 % interval contains the correct answer. Excess precision is the most stable form of overconfidence (Chapter 2); an interval task with a nominal level of 80 % measures it with one number, and both base-rate and quantity items feed into it. The alternative, Murphy’s reliability REL computed from each person’s probability judgements, is badly biased upwards with only 20 judgements per person (Chapter 2: about 0.063 on average for a perfectly calibrated person) and depends on the binning. $CE$ has a sampling floor too (about 0.070 expected on 20 items for a perfectly calibrated person; see the sample-size program), but the floor is the same in both arms and involves no bins.

Design

Element	Specification
Participants	Adults aged 18–70 who read the test language fluently; recruited through an online research-participant platform, not from the author’s community, students, customers or card-deck users
Exclusions	A degree in statistics or mathematics; previous participation in a forecasting tournament or any probability-calibration training; participation in this study’s pilot
Procedure	Informed consent → pre-test (10 items, parallel form) → randomisation → 15-minute module → post-test (20 items) → online follow-up two weeks later (10 items, parallel form)
Test items	Each item asks for a point estimate and an 80 % interval. Of the 20 post-test items, 10 are Bayesian reasoning problems with base-rate traps (presented in probability format; the correct answer is the posterior probability computed by Bayes’ theorem) and 10 are quantity-estimation items with verifiable answers (suited to reasoning from a reference class). Test and practice items do not overlap; items are piloted in advance, and parallel forms are counterbalanced across participants
Randomisation	After the pre-test, by computer, 1:1, in variable-size blocks, stratified by whether the participant has a university degree; the allocation sequence is concealed from participants and researchers until the pre-test is complete
Arm A (calibration micro-training)	15 minutes in three parts: ① natural-frequency conversion (about 5 minutes): rewriting probability problems as “1,000 people” frequency trees, with two practice problems; ② anchoring on an outside reference class (about 4 minutes): writing down the base rate of similar events first, then adjusting for the case; ③ feedback (about 6 minutes): 8 interval practice items, each followed at once by whether it hit and the running hit rate (against the 80 % target), and the Brier score on 4 probability practice items
Arm B (active control)	The same 15 minutes reading standard material on logical thinking (deductive arguments, common informal fallacies), with the same number of practice questions, each likewise followed by right/wrong feedback; no probability, base rates or interval estimation
Blinding	Participants are told only that the study compares two short “thinking-skills” modules and are not told the hypothesis; scoring is automatic; analysts see only coded arm labels until unblinding
Primary outcome	Interval calibration error on the 20 post-test items, $CE = \|H – 0.80\|$
Secondary outcomes	Signed calibration gap $0.80 – H$ (positive = too precise); mean absolute error (percentage points) of point estimates against the correct answers on the 10 base-rate items; proportion of base-rate-neglect answers (point estimate within ±5 points of the sensitivity); median interval width (standardised by item); $CE$ at the two-week follow-up
Analysis	Analysis of covariance: post-test $CE$ as the dependent variable, arm as the independent variable, pre-test $CE$ and the stratification variable as covariates; two-sided $\alpha = 0.05$ . Analysable participants are those who complete the post-test and pass the attention checks; a sensitivity analysis includes those excluded

Sample size

Calculation

# Sample size. Primary outcome CE = |H - 0.80| over 20 items (H = share of 80 %
# intervals that hit). ANCOVA on pre-test CE (correlation rho); alpha 0.05 two-sided,
# power 0.90. SD of CE from a simulated control arm: hit chance ~ Beta(mean 0.50, SD
# 0.15), an assumption.
import math, random
from statistics import NormalDist, mean, stdev

rng = random.Random(15)
N = NormalDist()

def beta_ab(m, sd):
    k = m * (1 - m) / sd ** 2 - 1
    return m * k, (1 - m) * k

def ce_sample(n, m, sd=0.15, items=20):
    a, b = beta_ab(m, sd)
    out = []
    for _ in range(n):
        h = rng.betavariate(a, b)
        hits = sum(rng.random() < h for _ in range(items))
        out.append(abs(hits / items - 0.80))
    return out

def per_arm(delta, sigma, rho, alpha=0.05, power=0.90):
    z = N.inv_cdf(1 - alpha / 2) + N.inv_cdf(power)
    return math.ceil(2 * z * z * sigma ** 2 * (1 - rho ** 2) / delta ** 2) + 1

if __name__ == "__main__":
    ctrl, trt = ce_sample(20000, 0.50), ce_sample(20000, 0.57)
    sigma = stdev(ctrl)
    print(f"simulated control: mean CE {mean(ctrl):.3f}, SD {sigma:.3f};"
          f" hit chance 0.57 gives mean CE {mean(trt):.3f}")
    print("analysable participants per arm | rho = 0.0 | 0.3 | 0.5")
    for d in (0.05, 0.06, 0.08):
        ns = [per_arm(d, sigma, r) for r in (0.0, 0.3, 0.5)]
        print(f"   CE difference {d:.2f} (d = {d / sigma:.2f})  |"
              f" {ns[0]:9d} | {ns[1]:3d} | {ns[2]:3d}")
    n = per_arm(0.06, sigma, 0.5)
    rec = math.ceil(n / 0.90)
    print(f"chosen: 0.06, rho = 0.5 -> {n} per arm")
    print(f"   allowing 10 % loss (incomplete sessions, failed attention checks):")
    print(f"   recruit {rec} per arm, {2 * rec} in all")
    n0, reps, hits = per_arm(0.06, sigma, 0.0), 1000, 0
    for _ in range(reps):
        a, b = ce_sample(n0, 0.50), ce_sample(n0, 0.57)
        se = math.sqrt(stdev(a) ** 2 / n0 + stdev(b) ** 2 / n0)
        hits += abs(mean(a) - mean(b)) / se > N.inv_cdf(0.975)
    print(f"check without covariate: {n0} per arm, simulated power {hits / reps:.2f}")
    pk = [math.comb(20, k) * 0.8 ** k * 0.2 ** (20 - k) for k in range(21)]
    floor = sum(p * abs(k / 20 - 0.8) for k, p in enumerate(pk))
    print(f"floor: a perfectly calibrated person (h = 0.80) has expected CE {floor:.3f}"
          " on 20 items")
# simulated control: mean CE 0.304, SD 0.173; hit chance 0.57 gives mean CE 0.243
# analysable participants per arm | rho = 0.0 | 0.3 | 0.5
#    CE difference 0.05 (d = 0.29)  |       253 | 231 | 190
#    CE difference 0.06 (d = 0.35)  |       176 | 161 | 133
#    CE difference 0.08 (d = 0.46)  |       100 |  91 |  75
# chosen: 0.06, rho = 0.5 -> 133 per arm
#    allowing 10 % loss (incomplete sessions, failed attention checks):
#    recruit 148 per arm, 296 in all
# check without covariate: 176 per arm, simulated power 0.93
# floor: a perfectly calibrated person (h = 0.80) has expected CE 0.070 on 20 items

In the simulated control arm the mean $CE$ is about 0.304 with a standard deviation of about 0.173 (assuming that control participants’ 80 % intervals have an average hit probability of 50 %, with a between-person standard deviation of 0.15); raising the average hit probability to about 0.57 lowers mean $CE$ to 0.243, a difference of about 0.06. With 0.06 (a standardised effect of about 0.35) as the smallest effect of interest, a correlation of 0.5 between pre- and post-test $CE$ and 90 % power, each arm needs 133 analysable participants; allowing about 10 % for incomplete sessions and failed attention checks, each arm recruits 148, 296 in all. Without the covariate each arm would need 176, and the simulated power is then about 0.93 (the simulated difference is slightly larger than 0.06). The pre–post correlation of 0.5 is the author’s assumption and should be estimated in the pilot; if the real correlation is lower, the sample must grow accordingly (the table gives the numbers for correlations of 0 and 0.3).

Decision rule

Calculation

# Decision rule fixed in advance. D = arm A minus arm B in post-test CE (ANCOVA-adjusted);
# negative = better calibrated in A; SESOI 0.06. Manipulation check: A has lower base-rate
# error than B (one-sided p < 0.05). Integrity: at most 15 % excluded.
SESOI = 0.06

def verdict(d, lo, hi, manip_ok, excluded):
    if excluded > 0.15:
        return f"D = {d:+.3f}: NOT INTERPRETABLE - {excluded:.0%} excluded (limit 15 %)"
    if not manip_ok:
        return f"D = {d:+.3f}: NOT INTERPRETABLE - manipulation check failed"
    if hi <= -SESOI:
        v = "improves calibration by at least 0.06"
    elif hi < 0:
        v = "improves calibration; not shown to reach 0.06"
    elif lo > 0:
        v = "training WORSENS calibration"
    elif lo >= -SESOI and hi <= SESOI:
        v = "no difference worth acting on"
    else:
        v = "unclear: the interval is too wide to decide"
    return f"D = {d:+.3f} ({lo:+.3f} to {hi:+.3f}): {v}"

for case in ((-0.095, -0.130, -0.060, True, 0.08), (-0.040, -0.075, -0.005, True, 0.07),
             (-0.010, -0.045, 0.025, True, 0.09), (0.045, 0.010, 0.080, True, 0.06),
             (-0.020, -0.080, 0.040, True, 0.10), (-0.110, -0.145, -0.075, False, 0.05),
             (-0.090, -0.125, -0.055, True, 0.21)):
    print(verdict(*case))
print("secondary (signed gap 0.80 - H, base-rate error, interval width, 2-week CE):")
print("   reported with intervals; none decides the verdict (the base-rate error is used")
print("   only as the manipulation check, which decides interpretability)")
# D = -0.095 (-0.130 to -0.060): improves calibration by at least 0.06
# D = -0.040 (-0.075 to -0.005): improves calibration; not shown to reach 0.06
# D = -0.010 (-0.045 to +0.025): no difference worth acting on
# D = +0.045 (+0.010 to +0.080): training WORSENS calibration
# D = -0.020 (-0.080 to +0.040): unclear: the interval is too wide to decide
# D = -0.110: NOT INTERPRETABLE - manipulation check failed
# D = -0.090: NOT INTERPRETABLE - 21% excluded (limit 15 %)
# secondary (signed gap 0.80 - H, base-rate error, interval width, 2-week CE):
#    reported with intervals; none decides the verdict (the base-rate error is used
#    only as the manipulation check, which decides interpretability)

The example results are invented. If arm A’s mean absolute error on the 10 base-rate items is not lower than arm B’s (one-sided $p < 0.05$ ), the content of the training was not acquired, and the report says “not interpretable”: the trial cannot distinguish “the training failed” from “the skill was learned but does not improve calibration”. If more than 15 % of those randomised are excluded for incomplete sessions or failed attention checks, the report likewise says “not interpretable”. Only if the upper limit of the 95 % confidence interval for the difference is no higher than −0.06 does the report say “the training reduced calibration error by at least the smallest effect of interest”.

What this trial cannot show

First, this is a short laboratory task; it cannot show that the training improves people’s forecasts of real events at work or in life, still less the outcomes of any decision. Second, arm A contains three components at once; even a positive result cannot say which one worked; separating them would need a factorial design. Third, $CE$ can fall simply because every interval is widened; the secondary outcomes of interval width and base-rate error are therefore needed to interpret the primary outcome, and if a fall in $CE$ comes only from intervals widening across the board with no improvement in point estimates, that must be reported as such. Fourth, arm A’s feedback may let participants guess that “intervals should be wider”, producing expectancy effects; arm B’s identical feedback structure only partly mitigates this. Fifth, volunteers recruited online may differ from the general population. Sixth, the two-week follow-up can show only short-term retention, not long-term effects.

Ethics and conflicts of interest

This is a minimal-risk behavioural study. It must be approved by an independent ethics committee; participants give informed consent, are paid at the platform’s fair rate and may withdraw at any time; data are stored de-identified. The protocol, test items, analysis code and decision rule are publicly registered before the first participant is enrolled (Nosek et al. 2018); whatever the result, positive or negative, the results, data and code will be published. Any financial or interest-rate items in the materials serve only as scoring exercises, and the materials state that they are not investment advice.

Disclosure

The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check. The author holds no financial-services licence and provides no investment advice, account management or trading-signal service. To limit the influence of these interests, the training materials and analysis code are published before enrolment, and the results, data and code will be published whatever they show.

Chapter summary

The four-step calibration checklist turns the conclusions of earlier chapters into four recordable steps: replace vague words with numbers, find the base rate of an outside reference class first, write down update triggers in advance, and at the end of the quarter review with the Brier score against the base rates. The fictional example shows that intuitive numbers can be worse than stating the base rates alone, and that updating as agreed in advance helps, though 12 questions in one quarter are not enough to conclude anything. The trial protocol compares a 15-minute calibration micro-training with time-matched reading on logical thinking; its single primary outcome is the calibration error $|H – 0.80|$ of 80 % intervals on 20 items, with 133 analysable and 148 recruited participants per arm; the decision rule, including thresholds for a manipulation check and for exclusions, is written as code and published in advance; the results will be published whether positive or negative.

References

Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115 (11): 2600–2606. https://doi.org/10.1073/pnas.1708274114.

Wallsten, Thomas S., David V. Budescu, Amnon Rapoport, Rami Zwick, and Barbara Forsyth. 1986. “Measuring the Vague Meanings of Probability Terms.” Journal of Experimental Psychology: General 115 (4): 348–65. https://doi.org/10.1037/0096-3445.115.4.348.

Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence: From Base-Rate Neglect and Signal Detection Theory to Superforecasting, Read Beside Classical Chinese Strategic Calculation [概率判断、布莱尔分数校准与群体智慧的统计理论] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23764998 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).