What does Falsifiability and the Replication Crisis conclude, in numbers, about “Publication Bias, the File Drawer and the Winner's Curse”?

Answer: Journals publish mainly significant results, and this filtering has barely changed since 1959. The fail-safe N answers only "does an effect exist?" and assumes the drawer averages zero effect; in this chapter's simulation it looked "extremely robust" while the published literature exaggerated the effect by 42 %. Egger's funnel-plot test has limited power, and its silence does not mean the absence of bias. In underpowered studies the effect sizes of significant results are systematically exaggerated: with a true \(d = 0.20\) and 20 % power, significant results average about 0.45 (about 2.2 times too large), with a 0.5 % chance of the wrong sign. The next chapter looks at what large-scale replication actually found.

Scope of the numbers: This book is not medical, psychological or investment advice. Marked boxes separate published evidence, runnable Python calculations (14 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.20, 0.45

true d = 0.2: 63 per group gives power 0.202 (SE of d = 0.178)
formula: Type S = 0.0051, exaggeration ratio = 2.24, mean significant |d| = 0.45
400,000 simulated: mean |d| if significant 0.448; wrong sign 0.0054; smallest significant 0.349
powerexaggerationType S(holds for any true effect D with standard error SE)
0.068.010.2013D / SE = 0.29
0.084.720.0850D / SE = 0.51
0.103.710.0450D / SE = 0.65
0.202.260.0053D / SE = 1.11
0.301.810.0011D / SE = 1.43
0.501.410.0001D / SE = 1.96
0.801.120.0000D / SE = 2.80
0.951.030.0000D / SE = 3.60

Key formula

\[N_{fs} = \frac{(\sum z)^2}{1.645^2} - k.\]
Show the full program
# Type M and Type S errors (Gelman & Carlin 2014). SE of d is about sqrt(2 / n); only p <
# .05 is "published". Exaggeration = mean |estimate| if significant / true d; Type S =
# wrong-sign share.
import math, random
from statistics import NormalDist
N = NormalDist()

def retrodesign(D, s, alpha=0.05):
    z = N.inv_cdf(1 - alpha / 2)
    hi, lo = 1 - N.cdf(z - D / s), N.cdf(-z - D / s)
    power = hi + lo
    # E|est| over the significant region, from the truncated-normal means of the two tails
    m_hi = D + s * N.pdf(z - D / s) / hi
    m_lo = D - s * N.pdf(-z - D / s) / lo if lo > 0 else 0.0
    exag = (hi * m_hi - lo * m_lo) / power / D
    return power, lo / power, exag

if __name__ == "__main__":
    d = 0.20
    n = next(n for n in range(2, 1000) if retrodesign(d, math.sqrt(2 / n))[0] >= 0.20)
    s = math.sqrt(2 / n)
    pw, ts, ex = retrodesign(d, s)
    print(f"true d = {d}: {n} per group gives power {pw:.3f} (SE of d = {s:.3f})")
    print(f"  formula: Type S = {ts:.4f}, exaggeration ratio = {ex:.2f},"
          f" mean significant |d| = {ex * d:.2f}")
    rng = random.Random(641); est = [rng.gauss(d, s) for _ in range(400000)]
    sig = [e for e in est if abs(e) / s > 1.96]; pos = [e for e in sig if e > 0]
    mean_sig = sum(map(abs, sig)) / len(sig)
    print(f"  400,000 simulated: mean |d| if significant {mean_sig:.3f};"
          f" wrong sign {1 - len(pos) / len(sig):.4f};"
          f" smallest significant {min(pos):.3f}")
    print("power   exaggeration   Type S"
          "      (holds for any true effect D with standard error SE)")
    for target in (0.06, 0.08, 0.10, 0.20, 0.30, 0.50, 0.80, 0.95):
        lo, hi = 1e-6, 10.0                  # bisection on D / SE
        for _ in range(60):
            mid = (lo + hi) / 2
            lo, hi = (mid, hi) if retrodesign(mid, 1.0)[0] < target else (lo, mid)
        pw, ts, ex = retrodesign(hi, 1.0)
        print(f" {pw:.2f}   {ex:12.2f}   {ts:8.4f}      D / SE = {hi:.2f}")

Single source and citation

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis (Version 1.0.0), Chapter 4: Publication Bias, the File Drawer and the Winner's Curse. 5D Library. https://doi.org/10.5281/zenodo.23782104

DOI: 10.5281/zenodo.23782104 · Read the monograph · 5DL-B-033 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026033,
  author    = {Fan, Yafu},
  title     = {Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23782104},
  url       = {https://doi.org/10.5281/zenodo.23782104}
}

Other chapter answers in this book