5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文
Chapter 4 Publication Bias, the File Drawer and the Winner’s Curse
97 % “success”
In 1959 the statistician Sterling went through every research report published in one year in four psychology journals (Sterling 1959). Of the 294 articles that used significance tests, 286 (97.3 %) rejected the null hypothesis, and not one was a replication. If a fair share of tested hypotheses are false and power is not near 100 %, such a “success rate” can only mean that what appears in journals has been filtered. Sixty years later the figure has barely moved (Chapter 6: 96 % in a 2021 sample).
The file-drawer problem
In 1979 Rosenthal called this filtering the file-drawer problem: the journals hold the 5 % of false positives, the drawers the 95 % of unpublished non-significant results (Rosenthal 1979). He proposed an index that came into wide use, the fail-safe N: how many unpublished studies averaging zero effect would be needed to pull the combined result of a set of published studies back to non-significance? If studies have -values summing to and one-sided ,
Rosenthal suggested that when exceeds a result can be considered “robust” to the file drawer.
The trouble with the fail-safe N is that it answers a narrow question: would the combined result still be significant if the drawer’s studies averaged zero effect? It says nothing about the size of the effect, and assumes the drawer is unbiased. A more common tool is the funnel plot: each study’s effect size plotted against its standard error. Without publication bias, large studies (small standard errors) cluster near the true effect and small studies scatter on both sides, forming a symmetrical inverted funnel; if small non-significant studies stay in the drawer, one side of the funnel goes missing. Egger and colleagues proposed in 1997 to test this asymmetry by regression: regress each study’s -value on its precision (the inverse of its standard error); an intercept away from zero signals asymmetry (Egger et al. 1997). The slope of the same regression can be read as the effect extrapolated to an infinitely precise study, later called the PET estimate.
A simulated literature with a file drawer: 300 studies, all with true effect , 10 to 200 per group; every significant positive result is published, any other only with probability 15 %. For the published studies: a fixed-effect meta-analysis, Egger’s test, the PET estimate and the fail-safe N. Then 200 such literatures each with true effect 0.20 and 0.
Calculation
# A simulated literature with a file drawer. 300 studies of the same true effect d are run
# with 10 to 200 per group; every significant positive result is published, any other
# result only with probability 0.15. Then: inverse-variance (fixed-effect) meta-analysis
# of what was published, Egger's regression test for funnel asymmetry (Egger et al. 1997),
# the PET estimate, and Rosenthal's (1979) fail-safe N with his tolerance level 5k + 10.
import math, random
def literature(rng, d, k=300, p_other=0.15):
pub, drawer = [], 0
for _ in range(k):
n = rng.randint(10, 200)
se = math.sqrt(2 / n + d * d / (4 * n))
est = rng.gauss(d, se)
if est / se > 1.96 or rng.random() < p_other:
pub.append((est, se))
else:
drawer += 1
return pub, drawer
def ols(xs, ys):
n = len(xs); mx, my = sum(xs) / n, sum(ys) / n
sxx = sum((x - mx) ** 2 for x in xs)
b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sxx
a = my - b * mx
s2 = sum((y - a - b * x) ** 2 for x, y in zip(xs, ys)) / (n - 2)
# intercept, slope, SE of the intercept
return a, b, math.sqrt(s2 * (1 / n + mx * mx / sxx))
def analyse(pub):
w = [1 / se ** 2 for _, se in pub]
fe = sum(wi * e for wi, (e, _) in zip(w, pub)) / sum(w)
# Egger: regress z = d / SE on precision 1 / SE. The intercept measures funnel
# asymmetry; the slope is the PET estimate (the effect extrapolated to an infinitely
# precise study, SE -> 0)
a, pet, se_a = ols([1 / se for _, se in pub], [e / se for e, se in pub])
fsn = sum(e / se for e, se in pub) ** 2 / 1.645 ** 2 - len(pub)
return fe, a / se_a, pet, fsn
if __name__ == "__main__":
pub, drawer = literature(random.Random(1979), 0.20)
fe, t, pet, fsn = analyse(pub)
sig = sum(e / se > 1.96 for e, se in pub)
print(f"300 studies, true d = 0.20: {len(pub)} published ({sig} significant),"
f" {drawer} in the drawer")
print(f" fixed-effect d from the published studies = {fe:.3f}; PET d = {pet:.3f}")
print(f" Egger intercept t = {t:.1f}; fail-safe N = {fsn:.0f}"
f" (tolerance 5k + 10 = {5 * len(pub) + 10})")
print(f" but the real drawer holds only {drawer} studies,"
" and their true d is 0.20, not 0")
print("200 literatures per row: mean fixed-effect d mean PET d Egger t > 2"
" fail-safe N > 5k + 10")
for d in (0.20, 0.0):
lits = [literature(random.Random(s), d)[0] for s in range(200)]
res = [analyse(p) + (len(p),) for p in lits]
fe = sum(r[0] for r in res) / 200
pet = sum(r[2] for r in res) / 200
egger = sum(r[1] > 2 for r in res) / 200
fsn = sum(r[3] > 5 * r[4] + 10 for r in res) / 200
print(f" true d = {d:.2f} {'':10s} {fe:17.3f} {pet:12.3f}"
f" {egger:12.0%} {fsn:16.0%}")
# 300 studies, true d = 0.20: 116 published (90 significant), 184 in the drawer
# fixed-effect d from the published studies = 0.284; PET d = 0.270
# Egger intercept t = 0.3; fail-safe N = 23998 (tolerance 5k + 10 = 590)
# but the real drawer holds only 184 studies, and their true d is 0.20, not 0
# 200 literatures per row: mean fixed-effect d mean PET d Egger t > 2 fail-safe N > 5k + 10
# true d = 0.20 0.285 0.190 57% 100%
# true d = 0.00 0.038 0.002 6% 5%In this literature, 116 of the 300 studies were published (90 of them significant) and 184 stayed in the drawer. The fixed-effect meta-analysis gives , 42 % above the true 0.20. This time Egger’s test finds no asymmetry (), and the PET estimate of 0.270 corrects little. The fail-safe N is 23,998, far above Rosenthal’s tolerance of 590, which looks “extremely robust”; yet the drawer actually holds only 184 studies, and their true effect is 0.20, not 0. The fail-safe N correctly tells us that the effect exists, and says nothing about its being exaggerated by 42 %.
Over 200 repetitions the picture sharpens. With a true effect of 0.20, the fixed-effect estimate averages 0.285; the PET estimate averages 0.190, correcting the bias on average but varying widely from one literature to the next; Egger’s test detects asymmetry in only 57 % of literatures. With a true effect of 0, the published literature yields a spurious effect averaging 0.038; Egger’s test raises the alarm in only 6 % of literatures, barely more than chance. With a true effect of zero only about 2.5 % of the 300 studies (seven or eight) are significant by chance and selected; most published studies are non-significant ones published at random with probability 15 %, so the funnel is only slightly asymmetric, and the intercept, an extrapolation to zero precision, is estimated very imprecisely: the test has almost no power. Funnel-plot tests often have little power; their silence is no evidence that publication bias is absent.
The winner’s curse: Type M and Type S errors
Even if no researcher ever manipulates data, as long as journals publish only significant results the published effect sizes will be systematically overestimated. The reason is simple: in an underpowered study, only those samples whose estimates happen, by sampling error, to be large can cross the significance threshold. Ioannidis described this as “why most discovered true associations are inflated” (Ioannidis 2008).
Gelman and Carlin proposed in 2014 to compute two quantities besides power (Gelman and Carlin 2014): the Type S (sign) error, the probability that a significant result points in the opposite direction to the true effect; and the Type M (magnitude) error, expressed as the exaggeration ratio, the mean absolute size of significant results divided by the true effect. They illustrated it with a published example: if the true effect studied were only 2 percentage points with a standard error of about 8 points, the power would be only 0.06; even a significant result would have a 24 % chance of the wrong sign and would exaggerate the effect about 9.7 times on average. (The 0.06 is rounded from about 0.057; at such low power the Type S error and the exaggeration ratio are very sensitive to power, which is why the program below, at a power of exactly 0.06, gives slightly smaller figures, 20 % and 8.0 times.) Their rule of thumb: exaggeration becomes serious below power 0.5, sign errors below power 0.1.
True effect , two-group comparison; first find the group size giving power of about 20 %, then compute the Type S error and the exaggeration ratio by Gelman and Carlin’s formulas and by 400,000 simulations. The final table holds for any true effect: it depends only on the ratio of the true effect to the standard error.
Calculation
# Type M and Type S errors (Gelman & Carlin 2014). SE of d is about sqrt(2 / n); only p <
# .05 is "published". Exaggeration = mean |estimate| if significant / true d; Type S =
# wrong-sign share.
import math, random
from statistics import NormalDist
N = NormalDist()
def retrodesign(D, s, alpha=0.05):
z = N.inv_cdf(1 - alpha / 2)
hi, lo = 1 - N.cdf(z - D / s), N.cdf(-z - D / s)
power = hi + lo
# E|est| over the significant region, from the truncated-normal means of the two tails
m_hi = D + s * N.pdf(z - D / s) / hi
m_lo = D - s * N.pdf(-z - D / s) / lo if lo > 0 else 0.0
exag = (hi * m_hi - lo * m_lo) / power / D
return power, lo / power, exag
if __name__ == "__main__":
d = 0.20
n = next(n for n in range(2, 1000) if retrodesign(d, math.sqrt(2 / n))[0] >= 0.20)
s = math.sqrt(2 / n)
pw, ts, ex = retrodesign(d, s)
print(f"true d = {d}: {n} per group gives power {pw:.3f} (SE of d = {s:.3f})")
print(f" formula: Type S = {ts:.4f}, exaggeration ratio = {ex:.2f},"
f" mean significant |d| = {ex * d:.2f}")
rng = random.Random(641); est = [rng.gauss(d, s) for _ in range(400000)]
sig = [e for e in est if abs(e) / s > 1.96]; pos = [e for e in sig if e > 0]
mean_sig = sum(map(abs, sig)) / len(sig)
print(f" 400,000 simulated: mean |d| if significant {mean_sig:.3f};"
f" wrong sign {1 - len(pos) / len(sig):.4f};"
f" smallest significant {min(pos):.3f}")
print("power exaggeration Type S"
" (holds for any true effect D with standard error SE)")
for target in (0.06, 0.08, 0.10, 0.20, 0.30, 0.50, 0.80, 0.95):
lo, hi = 1e-6, 10.0 # bisection on D / SE
for _ in range(60):
mid = (lo + hi) / 2
lo, hi = (mid, hi) if retrodesign(mid, 1.0)[0] < target else (lo, mid)
pw, ts, ex = retrodesign(hi, 1.0)
print(f" {pw:.2f} {ex:12.2f} {ts:8.4f} D / SE = {hi:.2f}")
# true d = 0.2: 63 per group gives power 0.202 (SE of d = 0.178)
# formula: Type S = 0.0051, exaggeration ratio = 2.24, mean significant |d| = 0.45
# 400,000 simulated: mean |d| if significant 0.448; wrong sign 0.0054; smallest significant 0.349
# power exaggeration Type S (holds for any true effect D with standard error SE)
# 0.06 8.01 0.2013 D / SE = 0.29
# 0.08 4.72 0.0850 D / SE = 0.51
# 0.10 3.71 0.0450 D / SE = 0.65
# 0.20 2.26 0.0053 D / SE = 1.11
# 0.30 1.81 0.0011 D / SE = 1.43
# 0.50 1.41 0.0001 D / SE = 1.96
# 0.80 1.12 0.0000 D / SE = 2.80
# 0.95 1.03 0.0000 D / SE = 3.60With a true effect , 63 per group gives power of about 0.20. In such studies significant results report an effect of about 0.45 on average, an exaggeration ratio of about 2.2; the chance of the wrong sign is about 0.5 % (0.0051 by formula, 0.0054 by simulation). The author’s framework estimated a mean significant result of about , an exaggeration of more than 3 times; by the program, the exaggeration is about 2.2 times and the mean about 0.45. A more intuitive figure: in this design any significant positive result is at least 0.349, that is, 1.96 standard errors; it is impossible to obtain an estimate that is both significant and close to the true 0.20.
The table shows the exaggeration ratio growing quickly as power falls: 1.41 at power 0.50, 1.81 at 0.30, 2.26 at 0.20, 3.71 at 0.10 and 8.01 at 0.06; the Type S error is 4.5 % at power 0.10 and 20 % at 0.06. At power 0.80 the exaggeration ratio is only 1.12. So the strategy of “run a small study first and see if it is significant” almost always yields an exaggerated estimate, and if the next study’s sample size is computed from that estimate, its real power will fall far short of plan.
Evidence
Power in neuroscience. Button, Ioannidis and colleagues analysed 49 neuroscience meta-analyses published in 2011 (covering 730 studies) and estimated the median power of individual studies at about 21 % (Button et al. 2013). They pointed out that low power not only means missing real effects but also makes significant findings more likely to be false (Chapter 2’s positive predictive value) and inflates the effects that are found (this chapter’s winner’s curse).
Note
How to read a small study. Ask first: “if the true effect were only of the size common in this field (say ), what would this study’s power be?”, then look up the exaggeration ratio in the table above. This is no exact correction, only a reminder: a significant effect in a small sample is very probably larger than the true one (step 3 of the Chapter 8 checklist).
Hypothesis · not established
Not yet established: the winner’s curse explains most of the “halving of effect sizes” in replication projects. If original studies had a distribution of power like that in this chapter’s model, then even if all their effects were real, replication effect sizes should average only a half to two-thirds of the originals. If, in the subset of original studies with high power (say above 0.8), replication effects are still only about half the originals, then the winner’s curse is not the main cause and the hypothesis is false.
Chapter summary
Journals publish mainly significant results, and this filtering has barely changed since 1959. The fail-safe N answers only “does an effect exist?” and assumes the drawer averages zero effect; in this chapter’s simulation it looked “extremely robust” while the published literature exaggerated the effect by 42 %. Egger’s funnel-plot test has limited power, and its silence does not mean the absence of bias. In underpowered studies the effect sizes of significant results are systematically exaggerated: with a true and 20 % power, significant results average about 0.45 (about 2.2 times too large), with a 0.5 % chance of the wrong sign. The next chapter looks at what large-scale replication actually found.
References
Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).