What does Falsifiability and the Replication Crisis conclude, in numbers, about “Practice and a Pre-Registered Trial”?

Answer: The four-step checklist asks in turn whether a claim was falsifiable in advance, whether it went through multiple comparisons and optional stopping, whether the sample was so small that the effect must be exaggerated, and whether it has been independently replicated with interests disclosed. Applied to three fictional claims, it clearly separates the case report, the supplement advertisement and the pre-registered trial. Applied to the series itself, it shows that the series' protocols meet the requirements of pre-registration only on paper (they have not been registered), and that there are no data, the effect-size assumptions lack a prior basis, the text is AI-drafted, the author has a commercial interest, and there has been no independent replication. The pre-registered trial has \(d'\) as its single primary outcome, 176 analysable and 208 recruited per arm, a decision rule written as code, and will be published whatever the result.

Scope of the numbers: This book is not medical, psychological or investment advice. Marked boxes separate published evidence, runnable Python calculations (14 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 176, 208

control arm: observed d' mean 0.96 (true 1.00), SD between people 0.72
true d' raised by 0.25 -> observed difference 0.23 (hits 0.670.71, false alarms 0.33 -> 0.29)
only more sceptical (c lowered by 0.4) -> d' difference -0.02 (hits 0.78, false alarms 0.46)
people per arm, 90 % power, alpha 0.05 | observed-d' SESOI 0.20 | 0.25 | 0.30
SD of observed d'0.60 | 191 | 123 | 86
SD of observed d'0.72 | 274 | 176 | 123
SD of observed d'0.90 | 427 | 274 | 191
chosen: SESOI 0.25 (observed d'), SD 0.72176 analysable per arm
allowing 15 % exclusions (attention checks, incomplete sessions) -> recruit 208 per arm, 416 in all
Show the full program
# Sample size for the Chapter 8 trial. Primary outcome: discrimination d' when judging 24
# news reports (12 contain an exaggerated claim from a methodological flaw, 12 are
# accurate), with the log-linear correction (add 0.5 to each count, 1 to each total;
# Hautus 1995). ASSUMED, not taken from any study: true d' in the control arm ~ N(1.0,
# 0.5) between people, criterion c ~ N(0, 0.5).
import math, random
from statistics import NormalDist, mean, stdev
N = NormalDist()

def observed_dprime(rng, d, c, n=12):
    h = sum(rng.random() < N.cdf(d / 2 - c) for _ in range(n))
    f = sum(rng.random() < N.cdf(-d / 2 - c) for _ in range(n))
    return N.inv_cdf((h + 0.5) / (n + 1)) - N.inv_cdf((f + 0.5) / (n + 1)), h / n, f / n

def arm(rng, shift_d=0.0, shift_c=0.0, people=20000):
    out = [observed_dprime(rng, rng.gauss(1.0 + shift_d, 0.5), rng.gauss(shift_c, 0.5))
           for _ in range(people)]
    return [o[0] for o in out], mean(o[1] for o in out), mean(o[2] for o in out)

def per_arm(sd, delta, alpha=0.05, power=0.90):
    z = N.inv_cdf(1 - alpha / 2) + N.inv_cdf(power)
    return math.ceil(2 * z ** 2 * sd ** 2 / delta ** 2) + 1

if __name__ == "__main__":
    rng = random.Random(1995)
    b, hb, fb = arm(rng)
    a, ha, fa = arm(rng, shift_d=0.25)
    s, hs, fs = arm(rng, shift_c=-0.4)
    sd = stdev(b)
    print(f"control arm: observed d' mean {mean(b):.2f} (true 1.00),"
          f" SD between people {sd:.2f}")
    print(f"true d' raised by 0.25 -> observed difference {mean(a) - mean(b):.2f}"
          f" (hits {hb:.2f} -> {ha:.2f}, false alarms {fb:.2f} -> {fa:.2f})")
    print(f"only more sceptical (c lowered by 0.4) ->"
          f" d' difference {mean(s) - mean(b):+.2f}"
          f" (hits {hs:.2f}, false alarms {fs:.2f})")
    print("people per arm, 90 % power, alpha 0.05 | observed-d' SESOI 0.20 | 0.25 | 0.30")
    for v in (0.60, round(sd, 2), 0.90):
        ns = " |".join(f"{per_arm(v, dl):5d}" for dl in (0.20, 0.25, 0.30))
        print(f"   SD of observed d' = {v:.2f} {'':16s}|" + ns)
    n = per_arm(round(sd, 2), 0.25)
    rec = math.ceil(n / 0.85)
    print(f"chosen: SESOI 0.25 (observed d'), SD {sd:.2f} -> {n} analysable per arm")
    print(f"allowing 15 % exclusions (attention checks, incomplete sessions)"
          f" -> recruit {rec} per arm,"
          f" {2 * rec} in all")

Single source and citation

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis (Version 1.0.0), Chapter 8: Practice and a Pre-Registered Trial. 5D Library. https://doi.org/10.5281/zenodo.23782104

DOI: 10.5281/zenodo.23782104 · Read the monograph · 5DL-B-033 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026033,
  author    = {Fan, Yafu},
  title     = {Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23782104},
  url       = {https://doi.org/10.5281/zenodo.23782104}
}

Other chapter answers in this book