5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文
Chapter 3 p-Hacking, Researcher Degrees of Freedom and the Garden of Forking Paths
Hear a song, become a year and a half younger
In 2011 the psychologists Simmons, Nelson and Simonsohn published a deliberately absurd experiment (Simmons et al. 2011): students who heard the Beatles’ “When I’m Sixty-Four” were, after controlling for their fathers’ age, nearly a year and a half “younger” than those who heard a control piece, . A song cannot change a date of birth. The result came from methods entirely within the conventions of the time: several variables measured and one reported, results inspected while collecting data before deciding whether to continue, one covariate picked from several analyses. They called these choices researcher degrees of freedom.
Four degrees of freedom
Simmons et al. quantified the consequences of four common practices by simulation. All data are normal noise with a true effect of zero; the baseline design is two groups of 20 and one test.
- A. Two dependent variables: measure two dependent variables correlated at 0.5, test each and their average, and report whichever is significant.
- B. Adding observations: test at 20 per group; if not significant, add 10 per group and test again.
- C. Covariates: run a plain test, an analysis with gender as a covariate, or a test of the gender × treatment interaction, and count any significant result.
- D. Dropping conditions: the experiment has three conditions (low, medium, high); compare any two, or test the linear trend.
Used alone, each raises the nominal 5 % false-positive rate to between 8 % and 13 %; all four together reach 60.7 %. These figures are in Table 1 of the paper and check out: A 9.5 %, B 7.7 %, C 11.7 %, D 12.6 %, A + B 14.4 %, A + B + C 30.9 %, A + B + C + D 60.7 % (the column).
A re-run of the simulation following the note to Table 1: 15,000 simulated samples, all observations drawn independently from a standard normal distribution; each person is female with probability 0.5; the two dependent variables correlate at 0.5. A sample counts as a “false positive” if any analysis allowed in that situation is significant. The paper does not say how gender was coded, and in the model with the interaction this matters: with effect coding () the “condition” coefficient tests the average effect over both genders; with 0/1 dummy coding it tests the effect within the reference gender, which amounts to one more, different test. The program runs both on the same samples.
Calculation
# Researcher degrees of freedom: a Monte Carlo re-run of Table 1 of Simmons, Nelson &
# Simonsohn (2011). Every effect is truly ZERO: all observations are independent N(0, 1)
# noise. Baseline: two conditions, 20 per cell, one t test. A: two dependent variables
# correlated r = .5; test DV1, DV2 or their mean. B: test at 20 per cell; if not
# significant add 10 per cell and test again. C: t test, ANCOVA with gender, or gender x
# condition (condition OR interaction counts); each person is female with probability .5.
# The paper does not state how gender is coded; in the interaction model this matters:
# effect coding (+-0.5) tests the AVERAGE condition effect, 0/1 coding tests it within
# one gender, a different test. Both are run on the same samples. D: three conditions;
# any of the three pairwise tests or the linear trend (-1, 0, 1). A result "counts" if
# ANY analysis allowed in that situation reaches the threshold. Critical t values by the
# Cornish-Fisher expansion (accurate to about 1e-4 for df > 30).
import math, random
from statistics import NormalDist
def t_crit(alpha, df):
z = NormalDist().inv_cdf(1 - alpha / 2)
return (z + (z ** 3 + z) / (4 * df)
+ (5 * z ** 5 + 16 * z ** 3 + 3 * z) / (96 * df ** 2)
+ (3 * z ** 7 + 19 * z ** 5 + 17 * z ** 3 - 15 * z) / (384 * df ** 3))
CRIT = {df: [t_crit(a, df) for a in (0.10, 0.05, 0.01)] for df in range(30, 90)}
def ols_level(cells, tests):
# cells: (n, sum y, sum y^2, x); level = thresholds (.10/.05/.01) passed by any
# tested beta
p = len(cells[0][3])
A = [[sum(n * x[i] * x[j] for n, _, _, x in cells) for j in range(p)]
for i in range(p)]
v = [sum(s * x[i] for _, s, _, x in cells) for i in range(p)]
M = [row + [float(i == j) for j in range(p)] for i, row in enumerate(A)]
for i in range(p): # Gauss-Jordan inverse of X'X
if abs(M[i][i]) < 1e-12:
return 0
M[i] = [e / M[i][i] for e in M[i]]
for k in range(p):
if k != i:
M[k] = [a - M[k][i] * b for a, b in zip(M[k], M[i])]
inv = [row[p:] for row in M]
b = [sum(inv[i][j] * v[j] for j in range(p)) for i in range(p)]
df = sum(c[0] for c in cells) - p
s2 = (sum(c[2] for c in cells) - sum(bi * vi for bi, vi in zip(b, v))) / df
return max(sum(abs(b[j]) / math.sqrt(s2 * inv[j][j]) > c for c in CRIT[df])
for j in tests)
DESIGNS = { # condition codes used by each comparison
"01": {0: -0.5, 1: 0.5}, "02": {0: -0.5, 2: 0.5}, "12": {1: -0.5, 2: 0.5},
"trend": {0: -1.0, 1: 0.0, 2: 1.0}}
def one_sample(rng):
data = [] # per condition: list of (gender, y1, y2)
for _ in range(3):
rows = []
for _ in range(30):
z1, z2 = rng.gauss(0, 1), rng.gauss(0, 1)
g = 0.5 if rng.random() < 0.5 else -0.5
rows.append((g, z1, 0.5 * z1 + math.sqrt(0.75) * z2))
data.append(rows)
levels = {}
for stage in (20, 30):
for dv in (0, 1, 2):
cell = {}
for k in range(3):
for g, y1, y2 in data[k][:stage]:
y = (y1, y2, (y1 + y2) / 2)[dv]
c = cell.setdefault((k, g), [0, 0.0, 0.0])
c[0] += 1; c[1] += y; c[2] += y * y
for dname, code in DESIGNS.items():
cs = [(c[0], c[1], c[2], code[k], g) for (k, g), c in cell.items()
if k in code]
models = {
"plain": ([(n, s, q, (1.0, x)) for n, s, q, x, g in cs], (1,)),
"cov": ([(n, s, q, (1.0, x, g)) for n, s, q, x, g in cs], (1,)),
"int": ([(n, s, q, (1.0, x, g, x * g)) for n, s, q, x, g in cs],
(1, 3)),
"int01": ([(n, s, q, (1.0, x, g + 0.5, x * (g + 0.5)))
for n, s, q, x, g in cs], (1, 3))}
for mname, (cells, tests) in models.items():
levels[(stage, dv, dname, mname)] = ols_level(cells, tests)
return levels
D3 = tuple(DESIGNS)
S2, DV3, C3, C01 = (20, 30), (0, 1, 2), ("plain", "cov", "int"), ("plain", "cov", "int01")
ROWS = [ # (label, stages, DVs, comparisons, models, Table 1: p < .10 / .05 / .01)
("baseline: one DV, one test", (20,), (0,), ("01",), ("plain",),
" (10.0, 5.0, 1.0 expected)"),
("A: two DVs (r = .5) or their mean", (20,), DV3, ("01",), ("plain",),
"17.8 9.5 2.2"),
("B: add 10 per cell if not significant", S2, (0,), ("01",), ("plain",),
"14.5 7.7 1.6"),
("C: gender covariate or interaction", (20,), (0,), ("01",), C3,
"21.6 11.7 2.7"),
("D: drop (or keep) one of 3 conditions", (20,), (0,), D3, ("plain",),
"23.2 12.6 2.8"),
("A + B", S2, DV3, ("01",), ("plain",), "26.0 14.4 3.3"),
("A + B + C", S2, DV3, ("01",), C3, "50.9 30.9 8.4"),
("A + B + C + D", S2, DV3, D3, C3, "81.5 60.7 21.5"),
("C, gender coded 0/1", (20,), (0,), ("01",), C01, "21.6 11.7 2.7"),
("A + B + C, gender coded 0/1", S2, DV3, ("01",), C01, "50.9 30.9 8.4"),
("A + B + C + D, gender coded 0/1", S2, DV3, D3, C01, "81.5 60.7 21.5")]
def run(n_sims=15000, seed=2011):
rng = random.Random(seed)
hits = [[0, 0, 0] for _ in ROWS]
for _ in range(n_sims):
lv = one_sample(rng)
for r, (_, st, dvs, ds, ms, _) in enumerate(ROWS):
best = max(lv[(s, v, d, m)] for s in st for v in dvs
for d in ds for m in ms)
for k in range(best):
hits[r][k] += 1
return [[100 * h / n_sims for h in row] for row in hits]
if __name__ == "__main__":
res = run()
print("15,000 simulated studies, no true effect this run: p<.10 p<.05 p<.01"
" | paper: .10 .05 .01")
for k, (row, r) in enumerate(zip(ROWS, res)):
if k == 8:
print("same samples, gender coded 0/1 instead of +-0.5:")
print(f"{row[0]:40s} {r[0]:5.1f} {r[1]:5.1f} {r[2]:5.1f} | {row[5]}")
# 15,000 simulated studies, no true effect this run: p<.10 p<.05 p<.01 | paper: .10 .05 .01
# baseline: one DV, one test 10.1 5.1 0.9 | (10.0, 5.0, 1.0 expected)
# A: two DVs (r = .5) or their mean 18.7 9.8 2.1 | 17.8 9.5 2.2
# B: add 10 per cell if not significant 14.9 7.9 1.5 | 14.5 7.7 1.6
# C: gender covariate or interaction 21.0 11.1 2.4 | 21.6 11.7 2.7
# D: drop (or keep) one of 3 conditions 23.1 12.4 2.7 | 23.2 12.6 2.8
# A + B 26.9 14.6 3.3 | 26.0 14.4 3.3
# A + B + C 48.7 29.1 7.5 | 50.9 30.9 8.4
# A + B + C + D 79.7 57.9 19.6 | 81.5 60.7 21.5
# same samples, gender coded 0/1 instead of +-0.5:
# C, gender coded 0/1 22.6 12.4 2.7 | 21.6 11.7 2.7
# A + B + C, gender coded 0/1 51.4 31.6 8.6 | 50.9 30.9 8.4
# A + B + C + D, gender coded 0/1 81.9 61.2 21.8 | 81.5 60.7 21.5The re-run comes very close to the paper. The baseline is 5.1 %, matching the nominal 5 %; situations A, B, C and D alone give 9.8 %, 7.9 %, 11.1 % and 12.4 % (paper 9.5 %, 7.7 %, 11.7 %, 12.6 %); A + B gives 14.6 %. With effect coding, A + B + C gives 29.1 % and all four together 57.9 %, about 3 percentage points below the paper’s 60.7 %, more than Monte Carlo error (about 0.4 points) can explain. On the same samples with 0/1 coding, C gives 12.4 %, A + B + C 31.6 % and all four together 61.2 %, only 0.5 points from the paper and within Monte Carlo error. So the gap comes mainly from the coding of gender: the paper probably used 0/1 coding (a guess; the paper does not say). This is itself an example: an inconspicuous coding choice in a report can move the false-positive rate by 3 points. The conclusion stands: a nominal 5 % false-positive rate rises to about sixty per cent once four innocent-looking choices are stacked.
Optional stopping
Situation B is the mildest form of optional stopping. Figure 1 of Simmons et al. shows more extreme versions: start with 10 or 20 per group, test after every 1, 5, 10 or 20 additional per group, stop as soon as , up to a maximum of 50 per group. Starting at 10 and testing after every observation gives a false-positive rate of about 22 %.
A re-run of Figure 1 of the paper on 40,000 simulated data paths with no true effect, so that Monte Carlo error is about 0.2 points; the last test is always at 50 per group. The second part gives -curves: all significant results sorted into five bins by -value (0 to 0.01, 0.01 to 0.02, and so on), with the share in each bin.
Calculation
# Optional stopping (Figure 1 of Simmons et al. 2011) and the shape of the "p-curve"
# (Simonsohn, Nelson & Simmons 2014). Two groups, true effect ZERO unless stated;
# two-sided t tests. The researcher starts with 10 or 20 per group, tests after every 1,
# 5, 10 or 20 more per group, and stops at p < .05 or at 50 per group. 40,000 simulated
# paths per case (Monte Carlo SE about 0.2 points). Critical t by Cornish-Fisher.
import math, random
from statistics import NormalDist
N_PATHS = 40000
def t_crit(alpha, df):
z = NormalDist().inv_cdf(1 - alpha / 2)
return (z + (z ** 3 + z) / (4 * df)
+ (5 * z ** 5 + 16 * z ** 3 + 3 * z) / (96 * df ** 2)
+ (3 * z ** 7 + 19 * z ** 5 + 17 * z ** 3 - 15 * z) / (384 * df ** 3))
BINS = (0.01, 0.02, 0.03, 0.04, 0.05)
CRIT = {df: [t_crit(a, df) for a in BINS] for df in range(10, 100)}
def t_stat(s1, q1, s2, q2, n):
m1, m2 = s1 / n, s2 / n
sp2 = (q1 - n * m1 * m1 + q2 - n * m2 * m2) / (2 * n - 2)
return abs(m1 - m2) / math.sqrt(2 * sp2 / n)
def p_bin(t, df):
# index of the p-value bin (0: p < .01 ... 4: .04-.05); None if p >= .05
return next((k for k in range(5) if t > CRIT[df][k]), None)
def path(rng, d, nmax=50):
s1 = q1 = s2 = q2 = 0.0; out = []
for n in range(1, nmax + 1):
a, b = rng.gauss(d, 1), rng.gauss(0, 1)
s1 += a; q1 += a * a; s2 += b; q2 += b * b
out.append((s1, q1, s2, q2))
return out
def first_hit(p, start, step, final=True):
looks = list(range(start, 51, step))
if final and looks[-1] != 50:
looks.append(50) # the last look is always at the 50-per-group limit
return next((k for k in (p_bin(t_stat(*p[n - 1], n), 2 * n - 2) for n in looks)
if k is not None), None)
if __name__ == "__main__":
rng = random.Random(1359)
paths = [path(rng, 0.0) for _ in range(N_PATHS)]
print("false-positive rate (%), stop at p < .05 or n = 50;"
" test after every k more per group")
for start in (10, 20):
rates = [100 * sum(first_hit(p, start, k) is not None for p in paths) / N_PATHS
for k in (1, 5, 10, 20)]
print(f" start at n = {start}: k = 1: {rates[0]:.1f} k = 5: {rates[1]:.1f}"
f" k = 10: {rates[2]:.1f} k = 20: {rates[3]:.1f}")
print("p-curve: share of significant results with p in"
" .00-.01 .01-.02 .02-.03 .03-.04 .04-.05")
curves = [("no effect, one test at n = 20",
[first_hit(p, 20, 99, False) for p in paths]),
("no effect, optional stopping (10, +1)",
[first_hit(p, 10, 1) for p in paths])]
for d in (0.5, 0.8):
ps = [path(rng, d, 20) for _ in range(N_PATHS)]
curves.append((f"true d = {d}, one test at n = 20",
[first_hit(p, 20, 99, False) for p in ps]))
for label, ks in curves:
sig = [k for k in ks if k is not None]
shares = " ".join(f"{sig.count(k) / len(sig):.2f}" for k in range(5))
print(f" {label:38s} {shares} (n sig = {len(sig)})")
# false-positive rate (%), stop at p < .05 or n = 50; test after every k more per group
# start at n = 10: k = 1: 21.5 k = 5: 16.6 k = 10: 13.9 k = 20: 11.2
# start at n = 20: k = 1: 16.2 k = 5: 12.9 k = 10: 11.2 k = 20: 9.9
# p-curve: share of significant results with p in .00-.01 .01-.02 .02-.03 .03-.04 .04-.05
# no effect, one test at n = 20 0.21 0.19 0.20 0.21 0.20 (n sig = 1908)
# no effect, optional stopping (10, +1) 0.05 0.05 0.13 0.29 0.48 (n sig = 8615)
# true d = 0.5, one test at n = 20 0.43 0.20 0.14 0.12 0.10 (n sig = 13464)
# true d = 0.8, one test at n = 20 0.63 0.15 0.10 0.07 0.05 (n sig = 27840)Starting at 10 per group and testing after every observation gives 21.5 %; testing after every 5, 10 or 20 gives 16.6 %, 13.9 % and 11.2 %; starting at 20 gives 16.2 %, 12.9 %, 11.2 % and 9.9 %. The paper’s text states about 22 %; the values marked in its figure are about 22.1 %, 17.0 %, 14.3 %, 12.7 % and 16.3 %, 13.3 %, 11.5 %, 10.4 % (as read from the figure; this book could not obtain the full text to check each one). The re-run lies 0.1 to 0.6 points below them, except “start at 10, test after every 20”, which is about 1.5 points lower, perhaps because the paper’s test points for that case differ from this book’s (10, 30 and 50 per group). Direction and size agree: the more often you look, the more false positives you get.
The -curve: clues in the distribution of significant results
Simonsohn, Nelson and Simmons went on to propose -curve analysis (Simonsohn et al. 2014). If a set of studies rests on a real effect, the significant -values should lean towards very small values (right skew); with no effect and no manipulation, they are roughly uniform between 0 and 0.05; with no effect but manipulation just past significance, they pile up near 0.05 (left skew).
The second part of the program shows all three shapes. With no effect and a single test, each of the five bins holds about 0.19 to 0.21, roughly uniform. With no effect but optional stopping, the 0.04-to-0.05 bin holds 48 % and the 0-to-0.01 bin only 5 %. With a true effect (20 per group), the 0-to-0.01 bin holds 43 %; with , 63 %. So the -curve can to some extent separate “a literature with evidential value” from “a literature produced by manipulation”. It has limits: it assumes the results analysed are independent and selected by a clear rule, and when real effects and manipulation coexist the two shapes can cancel.
The garden of forking paths: no deliberate fishing needed
The simulations of Simmons et al. are easily read as “researchers deliberately trying many analyses”. The statisticians Gelman and Loken pointed out that the problem is more general (Gelman and Loken 2014). Even if researchers run only one analysis, if that analysis was settled after seeing the data (whom to exclude, how to code variables, which groups to compare, which subgroup to look at), and with other data they would have made other, equally reasonable choices, the premises of the -value calculation already fail. Borrowing the title of a story by Borges, they called this the garden of forking paths: at each fork only one path is taken, but which one depends on the data in hand.
So -hacking is often not fraud but a chain of “reasonable” responses to data by researchers without a prior plan; the remedy cannot rest on exhortation alone but needs procedural constraint: write down the choices at the forks before seeing the data (Chapter 6).
Evidence
The American Statistical Association’s statement on -values. In 2016 the ASA published six principles on -values, among them: -values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by chance alone; scientific conclusions and business or policy decisions should not be based only on whether a -value passes a threshold; proper inference requires full reporting and transparency; a -value does not measure the size of an effect or the importance of a result (Wasserstein and Lazar 2016).
What Simmons et al. proposed
Simmons et al. set six requirements for authors: decide the rule for terminating data collection before it begins and report it; collect at least 20 observations per cell (or justify otherwise); list all variables collected; report all experimental conditions, including failed manipulations; if observations are eliminated, also report the results with them included; if an analysis includes a covariate, also report the results without it (Simmons et al. 2011). What these share is that they let readers see every fork in the garden, not only the path the researcher finally took.
Note
These numbers are not estimates for all research. The 60.7 % (or the re-run’s 57.9 % and 61.2 %) is a worst-case demonstration, with all four degrees of freedom exploited and a true effect of zero; it shows the mechanism, not the actual false-positive rate of any field.
Hypothesis · not established
Not yet established: degrees of freedom and failed replication. If researcher degrees of freedom are among the main causes of failed replication, then within a set of original studies, the more dependent variables, conditions or covariates the original paper reports (without prior registration), the lower the success rate of independent replication should be. If, after controlling for the original sample size and -value, there is no such association, the hypothesis is false.
Chapter summary
In data with no true effect, choosing among dependent variables, adding observations, adding covariates and dropping conditions each raise the false-positive rate from 5 % to roughly 8 % to 13 %, and together to about sixty per cent (60.7 % in the paper; in this book’s re-run 57.9 % with gender effect-coded and 61.2 % with 0/1 coding). The more often one looks at the data, the more false positives. The shape of the -curve can hint whether a literature has been manipulated. Gelman and Loken stressed that even without deliberate fishing, data-dependent choices of analysis rob -values of their meaning. The next chapter looks at another layer of selection: which results get published.
References
Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).