What does Falsifiability and the Replication Crisis conclude, in numbers, about “Large-Scale Replication: Results and Controversy”?

Answer: Three large replication projects found same-direction significance rates of 36.1 % (psychology, 35/97), 61 % (economics, 11/18) and 62 % (social-science experiments in *Nature* and *Science*, 13/21), with replication effect sizes about a half to two-thirds of the originals. A replication rate cannot be converted directly into a false discovery rate: a field with no false findings but small effects can also replicate poorly. Gilbert et al.'s criticisms about heterogeneity, single replications and protocol departures, and the Open Science Collaboration's reply, each have merit. When replication is judged by confidence intervals, the reasonable benchmark is far below 95 %. The next chapter discusses how to tackle these problems at their source.

Scope of the numbers: This book is not medical, psychological or investment advice. Marked boxes separate published evidence, runnable Python calculations (14 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

share real true d | published: share real | replications significant | effect ratio
1.000.50 | 1.00 | 70% | 0.72
1.000.20 | 1.00 | 22% | 0.32
0.500.50 | 0.95 | 67% | 0.69
0.500.30 | 0.90 | 37% | 0.43
0.300.50 | 0.90 | 64% | 0.66
0.000.00 | 0.00 | 3% | 0.00
replication size tau | replication in original CI | original in replication CI
1.0 x0.0 | 83.4% | 83.4%
1.0 x0.5 | 78.5% | 78.5%
2.5 x0.0 | 90.2% | 70.5%
2.5 x0.5 | 84.5% | 63.2%
5.0 x0.0 | 92.6% | 57.6%
5.0 x0.5 | 86.7% | 49.9%
Show the full program
# A replication project modelled: a share pi of studied effects is real (true d), the rest
# zero; originals (15-50 per group) are published only if p < .05 in the claimed
# direction; replications are sized for 92 % power at the ORIGINAL estimate (as planned in
# OSC 2015). Illustrative values. Part 2, no bias at all: chance that one study's 95 % CI
# captures the other's estimate (original SE = 1, replication k times larger; tau = SD of
# true effects between settings).
import math, random
from statistics import NormalDist
Z, Phi = NormalDist().inv_cdf, NormalDist().cdf

def project(rng, pi, d, n_orig=(15, 50), attempts=100000):
    succ, orig_sum, rep_sum, k, real = 0, 0.0, 0.0, 0, 0
    for _ in range(attempts):
        true_d = d if rng.random() < pi else 0.0
        n = rng.randint(*n_orig)
        se = math.sqrt(2 / n)
        est = rng.gauss(true_d, se)
        if est / se <= 1.96:
            continue                        # not significant: stays in the drawer
        n_rep = math.ceil(2 * (Z(0.975) + Z(0.92)) ** 2 / est ** 2)
        se_r = math.sqrt(2 / n_rep)
        rep = rng.gauss(true_d, se_r)
        k += 1; real += true_d > 0; succ += rep / se_r > 1.96
        orig_sum += est; rep_sum += rep
    return k, real / k, succ / k, rep_sum / orig_sum

if __name__ == "__main__":
    rng = random.Random(2015)
    print("share real  true d | published: share real | replications significant"
          " | effect ratio")
    for pi, d in ((1.0, 0.5), (1.0, 0.2), (0.5, 0.5), (0.5, 0.3), (0.3, 0.5), (0.0, 0.0)):
        k, real, succ, ratio = project(rng, pi, d)
        print(f"   {pi:4.2f}      {d:4.2f}  | {real:22.2f} | {succ:24.0%}"
              f" | {ratio:12.2f}")
    print("replication size  tau | replication in original CI"
          " | original in replication CI")
    for k in (1, 2.5, 5):
        for tau in (0.0, 0.5):
            sd = math.sqrt(1 + 1 / k + 2 * tau ** 2)
            a, b = 2 * Phi(1.96 / sd) - 1, 2 * Phi(1.96 / math.sqrt(k) / sd) - 1
            print(f"   {k:4.1f} x        {tau:3.1f} | {a:26.1%} | {b:26.1%}")

Single source and citation

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis (Version 1.0.0), Chapter 5: Large-Scale Replication: Results and Controversy. 5D Library. https://doi.org/10.5281/zenodo.23782104

DOI: 10.5281/zenodo.23782104 · Read the monograph · 5DL-B-033 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026033,
  author    = {Fan, Yafu},
  title     = {Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23782104},
  url       = {https://doi.org/10.5281/zenodo.23782104}
}

Other chapter answers in this book