What does Falsifiability and the Replication Crisis conclude, in numbers, about “Large-Scale Replication: Results and Controversy”?
Answer: Three large replication projects found same-direction significance rates of 36.1 % (psychology, 35/97), 61 % (economics, 11/18) and 62 % (social-science experiments in *Nature* and *Science*, 13/21), with replication effect sizes about a half to two-thirds of the originals. A replication rate cannot be converted directly into a false discovery rate: a field with no false findings but small effects can also replicate poorly. Gilbert et al.'s criticisms about heterogeneity, single replications and protocol departures, and the Open Science Collaboration's reply, each have merit. When replication is judged by confidence intervals, the reasonable benchmark is far below 95 %. The next chapter discusses how to tackle these problems at their source.
Scope of the numbers: This book is not medical, psychological or investment advice. Marked boxes separate published evidence, runnable Python calculations (14 programs, all checked), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
| share real true d | published: share real | replications significant | effect ratio | |
| 1.00 | 0.50 | 1.00 | 70% | 0.72 |
| 1.00 | 0.20 | 1.00 | 22% | 0.32 |
| 0.50 | 0.50 | 0.95 | 67% | 0.69 |
| 0.50 | 0.30 | 0.90 | 37% | 0.43 |
| 0.30 | 0.50 | 0.90 | 64% | 0.66 |
| 0.00 | 0.00 | 0.00 | 3% | 0.00 |
| replication size tau | replication in original CI | original in replication CI | |
| 1.0 x | 0.0 | 83.4% | 83.4% |
| 1.0 x | 0.5 | 78.5% | 78.5% |
| 2.5 x | 0.0 | 90.2% | 70.5% |
| 2.5 x | 0.5 | 84.5% | 63.2% |
| 5.0 x | 0.0 | 92.6% | 57.6% |
| 5.0 x | 0.5 | 86.7% | 49.9% |
Show the full program
# A replication project modelled: a share pi of studied effects is real (true d), the rest
# zero; originals (15-50 per group) are published only if p < .05 in the claimed
# direction; replications are sized for 92 % power at the ORIGINAL estimate (as planned in
# OSC 2015). Illustrative values. Part 2, no bias at all: chance that one study's 95 % CI
# captures the other's estimate (original SE = 1, replication k times larger; tau = SD of
# true effects between settings).
import math, random
from statistics import NormalDist
Z, Phi = NormalDist().inv_cdf, NormalDist().cdf
def project(rng, pi, d, n_orig=(15, 50), attempts=100000):
succ, orig_sum, rep_sum, k, real = 0, 0.0, 0.0, 0, 0
for _ in range(attempts):
true_d = d if rng.random() < pi else 0.0
n = rng.randint(*n_orig)
se = math.sqrt(2 / n)
est = rng.gauss(true_d, se)
if est / se <= 1.96:
continue # not significant: stays in the drawer
n_rep = math.ceil(2 * (Z(0.975) + Z(0.92)) ** 2 / est ** 2)
se_r = math.sqrt(2 / n_rep)
rep = rng.gauss(true_d, se_r)
k += 1; real += true_d > 0; succ += rep / se_r > 1.96
orig_sum += est; rep_sum += rep
return k, real / k, succ / k, rep_sum / orig_sum
if __name__ == "__main__":
rng = random.Random(2015)
print("share real true d | published: share real | replications significant"
" | effect ratio")
for pi, d in ((1.0, 0.5), (1.0, 0.2), (0.5, 0.5), (0.5, 0.3), (0.3, 0.5), (0.0, 0.0)):
k, real, succ, ratio = project(rng, pi, d)
print(f" {pi:4.2f} {d:4.2f} | {real:22.2f} | {succ:24.0%}"
f" | {ratio:12.2f}")
print("replication size tau | replication in original CI"
" | original in replication CI")
for k in (1, 2.5, 5):
for tau in (0.0, 0.5):
sd = math.sqrt(1 + 1 / k + 2 * tau ** 2)
a, b = 2 * Phi(1.96 / sd) - 1, 2 * Phi(1.96 / math.sqrt(k) / sd) - 1
print(f" {k:4.1f} x {tau:3.1f} | {a:26.1%} | {b:26.1%}")Single source and citation
Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis (Version 1.0.0), Chapter 5: Large-Scale Replication: Results and Controversy. 5D Library. https://doi.org/10.5281/zenodo.23782104
DOI: 10.5281/zenodo.23782104 · Read the monograph · 5DL-B-033 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026033,
author = {Fan, Yafu},
title = {Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23782104},
url = {https://doi.org/10.5281/zenodo.23782104}
}Other chapter answers in this book
- Chapter 1 · Falsifiability and Research Programmes: What Counts as a Scientific Statement?
- Chapter 2 · Why Most Published Findings May Be False: Ioannidis's Derivation
- Chapter 3 · p-Hacking, Researcher Degrees of Freedom and the Garden of Forking Paths
- Chapter 4 · Publication Bias, the File Drawer and the Winner's Curse
- Chapter 6 · The Remedies of Meta-Science: Pre-Registration, Registered Reports and Equivalence Tests
- Chapter 8 · Practice and a Pre-Registered Trial