5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文
Chapter 6 The Remedies of Meta-Science: Pre-Registration, Registered Reports and Equivalence Tests
Separating prediction from explanation after the fact
Nosek and colleagues summed up the case for pre-registration in 2018: scientific progress requires testing hypotheses generated from existing observations against new observations, and researchers often confuse the two (Nosek et al. 2018). Using data to generate hypotheses (exploration, postdiction) is legitimate, and so is using data to test hypotheses (confirmation, prediction); the problem is using the same data for both and reporting the result as a test. Hindsight bias makes it hard, after the fact, to tell what one had expected. Pre-registration means writing down and publicly archiving the research question, primary outcome, sample size and analysis plan before the results are known, so that readers can tell which results were predicted and which were discovered afterwards.
This is the same thing as Chapter 1’s falsifiability. A pre-registration records exactly what the theory forbids in this study: if the difference in the primary outcome falls in one range, the hypothesis is taken as supported; in another, as not supported. The forks of Chapter 3 are closed before the data are seen.
Why locking the plan restores the nominal false-positive rate
True effect zero, two groups of 50, three outcomes per person correlated at 0.5; 20,000 simulations per plan. Compared: a single pre-registered outcome; the best of the three outcomes; the best of three with a Bonferroni correction (); a look at 25 and at 50 per group, each at 0.05; and a pre-registered pair of interim analyses with Pocock boundaries (nominal at each look). Finally, two studies of the share of positive results, with Wilson 95 % confidence intervals.
Calculation
# Why fixing the analysis in advance restores the nominal error rate. True effect ZERO;
# two groups; 20,000 simulated studies per plan; two-sided t tests (critical values by
# Cornish-Fisher). The Pocock boundary for two equally spaced looks uses nominal alpha
# 0.0294 at each look.
import math, random
from statistics import NormalDist
def t_crit(alpha, df):
z = NormalDist().inv_cdf(1 - alpha / 2)
return (z + (z ** 3 + z) / (4 * df)
+ (5 * z ** 5 + 16 * z ** 3 + 3 * z) / (96 * df ** 2)
+ (3 * z ** 7 + 19 * z ** 5 + 17 * z ** 3 - 15 * z) / (384 * df ** 3))
def t_abs(xs, ys):
n = len(xs); mx, my = sum(xs) / n, sum(ys) / n
sp2 = (sum((x - mx) ** 2 for x in xs) + sum((y - my) ** 2 for y in ys)) / (2 * n - 2)
return abs(mx - my) / math.sqrt(2 * sp2 / n)
def study(rng, n=50):
# three outcomes per person, pairwise correlation 0.5 (shared component)
def person():
c = rng.gauss(0, 1)
return [math.sqrt(0.5) * c + math.sqrt(0.5) * rng.gauss(0, 1) for _ in range(3)]
return [person() for _ in range(n)], [person() for _ in range(n)]
def wilson(k, n):
z = 1.96; p = k / n
c = (p + z * z / (2 * n)) / (1 + z * z / n)
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return f"{p:.1%} ({c - h:.1%} to {c + h:.1%})"
if __name__ == "__main__":
rng = random.Random(2600)
c05_98, c05_48 = t_crit(0.05, 98), t_crit(0.05, 48)
cbon, cp98, cp48 = t_crit(0.05 / 3, 98), t_crit(0.0294, 98), t_crit(0.0294, 48)
hits = [0] * 5
sims = 20000
for _ in range(sims):
a, b = study(rng)
ts = [t_abs([p[j] for p in a], [p[j] for p in b]) for j in range(3)]
half = t_abs([p[0] for p in a[:25]], [p[0] for p in b[:25]])
hits[0] += ts[0] > c05_98 # the one registered outcome
hits[1] += max(ts) > c05_98 # best of three outcomes
hits[2] += max(ts) > cbon # best of three, Bonferroni
hits[3] += half > c05_48 or ts[0] > c05_98 # peek at 25, then 50
hits[4] += half > cp48 or ts[0] > cp98 # peek, Pocock boundary
labels = ["registered: one outcome, one test at 50 per group",
"report the best of 3 outcomes", "best of 3 with Bonferroni (alpha / 3)",
"look at 25 and at 50, p < .05 each time",
"look at 25 and at 50, registered Pocock boundary"]
for lab, h in zip(labels, hits):
print(f"{lab:52s} false-positive rate {h / sims:6.1%}")
print("positive first hypothesis, Scheel et al. 2021: standard reports 146 / 152 =",
wilson(146, 152))
print(" Registered Reports 31 / 71 =",
wilson(31, 71))
print("benefit on primary outcome, Kaplan & Irvin 2015: before 2000 17 / 30 =",
wilson(17, 30))
print(" after 2000 2 / 25 =",
wilson(2, 25))
# registered: one outcome, one test at 50 per group false-positive rate 5.0%
# report the best of 3 outcomes false-positive rate 12.7%
# best of 3 with Bonferroni (alpha / 3) false-positive rate 4.5%
# look at 25 and at 50, p < .05 each time false-positive rate 8.4%
# look at 25 and at 50, registered Pocock boundary false-positive rate 5.0%
# positive first hypothesis, Scheel et al. 2021: standard reports 146 / 152 = 96.1% (91.7% to 98.2%)
# Registered Reports 31 / 71 = 43.7% (32.7% to 55.2%)
# benefit on primary outcome, Kaplan & Irvin 2015: before 2000 17 / 30 = 56.7% (39.2% to 72.6%)
# after 2000 2 / 25 = 8.0% (2.2% to 25.0%)A single pre-registered outcome gives a false-positive rate of exactly 5.0 %. Picking the best of three outcomes raises it to 12.7 %; the same picking with a Bonferroni correction brings it down to 4.5 % (a little under 5 % because the three outcomes are correlated, which makes the correction conservative). Looking at 25 and at 50 per group, at 0.05 each time, raises it to 8.4 %; registering the interim analysis in advance with stricter pre-specified boundaries for the two tests brings it back to 5.0 %. So pre-registration does not forbid multiple outcomes or interim analyses; it requires that they be stated in advance, with the decision criteria adjusted accordingly.
Registered Reports
Pre-registration deals with “choosing after the fact” but not with publication bias: a study completed as planned with a negative result can still be rejected by journals or left in a drawer by its authors. In 2013 the neuroscientist Chambers announced that the journal Cortex would accept Registered Reports (Chambers 2013): authors submit their introduction, methods and analysis plan before collecting data; after peer review the journal issues an “in-principle acceptance”, and provided the study is carried out as planned the paper is published whatever the results. Publication then depends on whether the question matters and the method is sound, not on whether the result is significant.
Evidence
The share of positive results in Registered Reports. Scheel, Schijen and Lakens in 2021 compared 71 published Registered Reports in psychology with 152 randomly sampled hypothesis-testing studies from the standard literature, looking only at each paper’s first hypothesis. In the standard literature 96 % reported a positive result, in Registered Reports only 44 %; excluding direct replications, 96 % and 50 % (Scheel et al. 2021). The authors argue that the gap shows negative results to be heavily under-reported in psychology; but the study was not randomised and cannot by itself show that Registered Reports reduce bias, since studies that choose the Registered Report format may differ to begin with (for example, more of them are replications).
The program gives 146/152 = 96.1 % (91.7 % to 98.2 %) for the standard literature and 31/71 = 43.7 % (32.7 % to 55.2 %) for Registered Reports: two intervals far apart, consistent with the paper’s 96 % and 44 %. The framework’s two figures check out.
Evidence
After clinical-trial registration. Kaplan and Irvin analysed large clinical trials funded by the US National Heart, Lung, and Blood Institute: of 30 published before 2000, 17 (57 %) reported a significant benefit on the primary outcome; of 25 published after 2000, only 2 (8 %). The post-2000 trials had all registered their primary outcomes in advance on ClinicalTrials.gov. The authors stress that this is an association and cannot show that the registration requirement caused the change (Kaplan and Irvin 2015).
Proving “no difference”: equivalence tests
A conventional significance test can reach only two conclusions, “significant” or “not significant”. “Not significant” does not mean “no difference”; the sample may simply be too small. To show that “the difference between two practices is too small to matter”, one must specify in advance a smallest effect size of interest (SESOI) and test whether the true difference lies within SESOI. In 2017 Lakens set out for psychologists the two one-sided tests procedure (TOST) long used in pharmacology (Lakens 2017; Schuirmann 1987): test separately that “the difference is not below SESOI” and that “it is not above SESOI”, and declare equivalence only if both one-sided hypotheses are rejected; this is equivalent to checking whether the 90 % confidence interval of the difference lies wholly within SESOI. The SESOI can be set from clinical or practical importance, from the distribution of effects in earlier studies or from resource limits, but it must be set in advance (Lakens, Scheel, et al. 2018).
The statistical properties of this series’ decision rule
The 5D Library books generally write in their Chapter 8 a decision rule coded in advance. The framework calls it a “four-tier” rule; in fact it has five verdicts, checked in order:
- the lower confidence limit is at least the SESOI: a benefit of at least the SESOI;
- the lower limit is above zero: a benefit, but its size not shown to reach the SESOI;
- the upper limit is below zero: harm;
- the interval lies wholly within SESOI: no difference worth acting on;
- otherwise: the interval is too wide, unclear.
There are also prior conditions specified in advance (for example insufficient adherence, a failed manipulation check, a safety event) which, if triggered, lead to “not interpretable” or a pause for review whatever the result.
The trial is designed for 90 % power at the SESOI with two-sided , so the standard error is SESOI / 3.24. The program computes the probability of each verdict at different true effects (in units of the SESOI) and compares it with the probability that TOST (90 % confidence interval) declares equivalence. Finally it classifies 100,000 simulated estimates with the coded rule itself, as a check on the formulas.
Calculation
# The five-verdict rule of the 5D Library's Chapter 8 trials, and TOST (Lakens 2017), for
# a trial sized for 90 % power at the SESOI (= delta), two-sided alpha 0.05, so SE = delta
# / 3.24. Verdicts from the 95 % CI (lo, hi) of the effect, positive = benefit, checked in
# this order:
# V1 lo >= delta: benefit of at least the SESOI
# V2 lo > 0: benefit, size not shown to reach it
# V3 hi < 0: HARM
# V4 -delta <= lo and hi <= delta: no difference worth acting on
# V5 otherwise: unclear
import math, random
from statistics import NormalDist
N = NormalDist()
Z95, Z90 = N.inv_cdf(0.975), N.inv_cdf(0.90)
def verdict(est, se, delta, z=Z95):
lo, hi = est - z * se, est + z * se
if lo >= delta: return 0
if lo > 0: return 1
if hi < 0: return 2
if lo >= -delta and hi <= delta: return 3
return 4
def probs(theta, delta=1.0):
se = delta / (Z95 + Z90)
cuts = [(-math.inf, -Z95 * se, 2), (-Z95 * se, -delta + Z95 * se, 4),
(-delta + Z95 * se, delta - Z95 * se, 3), (delta - Z95 * se, Z95 * se, 4),
(Z95 * se, delta + Z95 * se, 1), (delta + Z95 * se, math.inf, 0)]
out = [0.0] * 5
for a, b, v in cuts:
out[v] += N.cdf((b - theta) / se) - N.cdf((a - theta) / se)
tost = (N.cdf((delta - 1.645 * se - theta) / se)
- N.cdf((-delta + 1.645 * se - theta) / se))
return out, tost
if __name__ == "__main__":
print("true effect / SESOI | V1 >=SESOI V2 benefit V3 harm V4 no diff V5 unclear"
" | TOST 90% CI")
for r in (-0.5, 0.0, 0.25, 0.5, 0.75, 1.0, 1.5, 2.0):
p, tost = probs(r)
cols = " ".join(f"{x:9.3f}" for x in p)
print(f" {r:5.2f} | " + cols + f" | {tost:9.3f}")
# check probs() against verdict() directly
rng, se = random.Random(355), 1 / (Z95 + Z90)
for r in (0.0, 1.0):
c = [0] * 5
for _ in range(100000):
c[verdict(rng.gauss(r, se), se, 1.0)] += 1
vs = " ".join(f"V{k + 1} {x / 1e5:.3f}" for k, x in enumerate(c))
print(f"check by simulation, true effect {r:.0f} SESOI: " + vs)
# true effect / SESOI | V1 >=SESOI V2 benefit V3 harm V4 no diff V5 unclear | TOST 90% CI
# -0.50 | 0.000 0.000 0.367 0.365 0.267 | 0.490
# 0.00 | 0.000 0.025 0.025 0.800 0.150 | 0.890
# 0.25 | 0.000 0.125 0.003 0.663 0.209 | 0.776
# 0.50 | 0.000 0.367 0.000 0.365 0.267 | 0.490
# 0.75 | 0.003 0.678 0.000 0.125 0.194 | 0.202
# 1.00 | 0.025 0.875 0.000 0.025 0.075 | 0.050
# 1.50 | 0.367 0.631 0.000 0.000 0.002 | 0.001
# 2.00 | 0.900 0.100 0.000 0.000 0.000 | 0.000
# check by simulation, true effect 0 SESOI: V1 0.000 V2 0.025 V3 0.026 V4 0.799 V5 0.150
# check by simulation, true effect 1 SESOI: V1 0.026 V2 0.875 V3 0.000 V4 0.025 V5 0.074The results show several properties of the rule, some of which are important self-corrections for this series.
- The Type I error is controlled. With a true effect of zero, the chance of any verdict of benefit (1 or 2) is 2.5 %, and of “harm” also 2.5 %; the commonest verdict is “no difference worth acting on” (80.0 %), followed by “unclear” (15.0 %).
- “90 % power” means reaching some verdict of benefit. When the true effect exactly equals the SESOI, verdicts 1 and 2 together have 90.0 %, but the strongest, verdict 1, has only 2.5 %; the other 87.5 % is “a benefit, size not shown”. To have an even chance of verdict 1 the true effect must be about 1.6 times the SESOI; for a 90 % chance, about 2 times. Most books in this series state “90 % power” in their sample-size calculations, and readers should know that this is not “a 90 % chance of showing an effect of at least the SESOI” (Chapter 8 of volume 30 already explains this for its own trial).
- Intermediate values are hardest. With a true effect of half the SESOI, “benefit” (36.7 %) and “no difference worth acting on” (36.5 %) are almost equally likely, with 26.7 % unclear. This is not a flaw in the rule but the honest consequence of the sample size.
- The “no difference” verdict is stricter than TOST. With a true effect of zero, TOST declares equivalence 89.0 % of the time, this series’ rule 80.0 %, because the latter uses the 95 % confidence interval (equivalent to TOST with on each side). With a true effect exactly at the SESOI, TOST wrongly declares equivalence 5.0 % of the time, this series’ rule 2.5 %.
- Formulas and code agree. Classifying simulated estimates with the coded rule itself, a zero true effect gives verdicts 2, 3 and 4 in 2.5 %, 2.6 % and 79.9 %, and a true effect at the SESOI gives verdicts 1 and 2 in 2.6 % and 87.5 %, matching the formulas.
Note
Why not lower the bar for the strongest verdict? If it required only a point estimate of at least the SESOI, it would often appear when the true effect is only half that, and “at least the SESOI” would no longer be reliable. The strict wording costs larger samples; each book should state plainly what “90 % power” means.
Other proposals and debates
This book concentrates on pre-registration and equivalence tests, but meta-science has other proposals. Benjamin and more than seventy co-authors, for example, proposed moving the significance threshold for new discoveries from 0.05 to 0.005 (Benjamin et al. 2018); another group led by Lakens replied that rather than a new fixed threshold, researchers should be required to justify their chosen in advance (Lakens et al. 2018). Meta-science, too, improves through debate; it offers no once-and-for-all answer.
Chapter summary
Pre-registration writes down what a theory forbids before the data are seen, bringing the false-positive rate back to its nominal level; it does not forbid multiple outcomes or interim analyses, only requires that they be stated in advance and the criteria adjusted. Registered Reports make publication independent of the result; the positive rate for first hypotheses is about 44 % in Registered Reports and about 96 % in the standard literature. Equivalence tests let “no difference” be shown. This series’ decision rule has five verdicts; it controls the Type I error, but “90 % power” refers to reaching some verdict of benefit, and the strongest verdict appears only 2.5 % of the time when the true effect exactly equals the SESOI. The next chapter returns to classical Chinese scholarship to see what “seek truth from facts” and “without evidence there is no trust” meant in their original texts.
References
Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).