5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文

Chapter 3 p-Hacking, Researcher Degrees of Freedom and the Garden of Forking Paths

Hear a song, become a year and a half younger

In 2011 the psychologists Simmons, Nelson and Simonsohn published a deliberately absurd experiment (Simmons et al. 2011): students who heard the Beatles’ “When I’m Sixty-Four” were, after controlling for their fathers’ age, nearly a year and a half “younger” than those who heard a control piece, p=0.040p = 0.040. A song cannot change a date of birth. The result came from methods entirely within the conventions of the time: several variables measured and one reported, results inspected while collecting data before deciding whether to continue, one covariate picked from several analyses. They called these choices researcher degrees of freedom.

Four degrees of freedom

Simmons et al. quantified the consequences of four common practices by simulation. All data are normal noise with a true effect of zero; the baseline design is two groups of 20 and one tt test.

  • A. Two dependent variables: measure two dependent variables correlated at 0.5, test each and their average, and report whichever is significant.
  • B. Adding observations: test at 20 per group; if not significant, add 10 per group and test again.
  • C. Covariates: run a plain tt test, an analysis with gender as a covariate, or a test of the gender × treatment interaction, and count any significant result.
  • D. Dropping conditions: the experiment has three conditions (low, medium, high); compare any two, or test the linear trend.

Used alone, each raises the nominal 5 % false-positive rate to between 8 % and 13 %; all four together reach 60.7 %. These figures are in Table 1 of the paper and check out: A 9.5 %, B 7.7 %, C 11.7 %, D 12.6 %, A + B 14.4 %, A + B + C 30.9 %, A + B + C + D 60.7 % (the p<0.05p < 0.05 column).

A re-run of the simulation following the note to Table 1: 15,000 simulated samples, all observations drawn independently from a standard normal distribution; each person is female with probability 0.5; the two dependent variables correlate at 0.5. A sample counts as a “false positive” if any analysis allowed in that situation is significant. The paper does not say how gender was coded, and in the model with the interaction this matters: with effect coding (±0.5\pm 0.5) the “condition” coefficient tests the average effect over both genders; with 0/1 dummy coding it tests the effect within the reference gender, which amounts to one more, different test. The program runs both on the same samples.

Calculation

# Researcher degrees of freedom: a Monte Carlo re-run of Table 1 of Simmons, Nelson &
# Simonsohn (2011). Every effect is truly ZERO: all observations are independent N(0, 1)
# noise. Baseline: two conditions, 20 per cell, one t test. A: two dependent variables
# correlated r = .5; test DV1, DV2 or their mean. B: test at 20 per cell; if not
# significant add 10 per cell and test again. C: t test, ANCOVA with gender, or gender x
# condition (condition OR interaction counts); each person is female with probability .5.
# The paper does not state how gender is coded; in the interaction model this matters:
# effect coding (+-0.5) tests the AVERAGE condition effect, 0/1 coding tests it within
# one gender, a different test. Both are run on the same samples. D: three conditions;
# any of the three pairwise tests or the linear trend (-1, 0, 1). A result "counts" if
# ANY analysis allowed in that situation reaches the threshold. Critical t values by the
# Cornish-Fisher expansion (accurate to about 1e-4 for df > 30).
import math, random
from statistics import NormalDist

def t_crit(alpha, df):
    z = NormalDist().inv_cdf(1 - alpha / 2)
    return (z + (z ** 3 + z) / (4 * df)
            + (5 * z ** 5 + 16 * z ** 3 + 3 * z) / (96 * df ** 2)
            + (3 * z ** 7 + 19 * z ** 5 + 17 * z ** 3 - 15 * z) / (384 * df ** 3))

CRIT = {df: [t_crit(a, df) for a in (0.10, 0.05, 0.01)] for df in range(30, 90)}

def ols_level(cells, tests):
    # cells: (n, sum y, sum y^2, x); level = thresholds (.10/.05/.01) passed by any
    # tested beta
    p = len(cells[0][3])
    A = [[sum(n * x[i] * x[j] for n, _, _, x in cells) for j in range(p)]
         for i in range(p)]
    v = [sum(s * x[i] for _, s, _, x in cells) for i in range(p)]
    M = [row + [float(i == j) for j in range(p)] for i, row in enumerate(A)]
    for i in range(p):                      # Gauss-Jordan inverse of X'X
        if abs(M[i][i]) < 1e-12:
            return 0
        M[i] = [e / M[i][i] for e in M[i]]
        for k in range(p):
            if k != i:
                M[k] = [a - M[k][i] * b for a, b in zip(M[k], M[i])]
    inv = [row[p:] for row in M]
    b = [sum(inv[i][j] * v[j] for j in range(p)) for i in range(p)]
    df = sum(c[0] for c in cells) - p
    s2 = (sum(c[2] for c in cells) - sum(bi * vi for bi, vi in zip(b, v))) / df
    return max(sum(abs(b[j]) / math.sqrt(s2 * inv[j][j]) > c for c in CRIT[df])
               for j in tests)

DESIGNS = {  # condition codes used by each comparison
    "01": {0: -0.5, 1: 0.5}, "02": {0: -0.5, 2: 0.5}, "12": {1: -0.5, 2: 0.5},
    "trend": {0: -1.0, 1: 0.0, 2: 1.0}}

def one_sample(rng):
    data = []  # per condition: list of (gender, y1, y2)
    for _ in range(3):
        rows = []
        for _ in range(30):
            z1, z2 = rng.gauss(0, 1), rng.gauss(0, 1)
            g = 0.5 if rng.random() < 0.5 else -0.5
            rows.append((g, z1, 0.5 * z1 + math.sqrt(0.75) * z2))
        data.append(rows)
    levels = {}
    for stage in (20, 30):
        for dv in (0, 1, 2):
            cell = {}
            for k in range(3):
                for g, y1, y2 in data[k][:stage]:
                    y = (y1, y2, (y1 + y2) / 2)[dv]
                    c = cell.setdefault((k, g), [0, 0.0, 0.0])
                    c[0] += 1; c[1] += y; c[2] += y * y
            for dname, code in DESIGNS.items():
                cs = [(c[0], c[1], c[2], code[k], g) for (k, g), c in cell.items()
                      if k in code]
                models = {
                    "plain": ([(n, s, q, (1.0, x)) for n, s, q, x, g in cs], (1,)),
                    "cov": ([(n, s, q, (1.0, x, g)) for n, s, q, x, g in cs], (1,)),
                    "int": ([(n, s, q, (1.0, x, g, x * g)) for n, s, q, x, g in cs],
                            (1, 3)),
                    "int01": ([(n, s, q, (1.0, x, g + 0.5, x * (g + 0.5)))
                               for n, s, q, x, g in cs], (1, 3))}
                for mname, (cells, tests) in models.items():
                    levels[(stage, dv, dname, mname)] = ols_level(cells, tests)
    return levels

D3 = tuple(DESIGNS)
S2, DV3, C3, C01 = (20, 30), (0, 1, 2), ("plain", "cov", "int"), ("plain", "cov", "int01")
ROWS = [  # (label, stages, DVs, comparisons, models, Table 1: p < .10 / .05 / .01)
    ("baseline: one DV, one test", (20,), (0,), ("01",), ("plain",),
     "  (10.0, 5.0, 1.0 expected)"),
    ("A: two DVs (r = .5) or their mean", (20,), DV3, ("01",), ("plain",),
     "17.8    9.5    2.2"),
    ("B: add 10 per cell if not significant", S2, (0,), ("01",), ("plain",),
     "14.5    7.7    1.6"),
    ("C: gender covariate or interaction", (20,), (0,), ("01",), C3,
     "21.6   11.7    2.7"),
    ("D: drop (or keep) one of 3 conditions", (20,), (0,), D3, ("plain",),
     "23.2   12.6    2.8"),
    ("A + B", S2, DV3, ("01",), ("plain",), "26.0   14.4    3.3"),
    ("A + B + C", S2, DV3, ("01",), C3, "50.9   30.9    8.4"),
    ("A + B + C + D", S2, DV3, D3, C3, "81.5   60.7   21.5"),
    ("C, gender coded 0/1", (20,), (0,), ("01",), C01, "21.6   11.7    2.7"),
    ("A + B + C, gender coded 0/1", S2, DV3, ("01",), C01, "50.9   30.9    8.4"),
    ("A + B + C + D, gender coded 0/1", S2, DV3, D3, C01, "81.5   60.7   21.5")]

def run(n_sims=15000, seed=2011):
    rng = random.Random(seed)
    hits = [[0, 0, 0] for _ in ROWS]
    for _ in range(n_sims):
        lv = one_sample(rng)
        for r, (_, st, dvs, ds, ms, _) in enumerate(ROWS):
            best = max(lv[(s, v, d, m)] for s in st for v in dvs
                       for d in ds for m in ms)
            for k in range(best):
                hits[r][k] += 1
    return [[100 * h / n_sims for h in row] for row in hits]

if __name__ == "__main__":
    res = run()
    print("15,000 simulated studies, no true effect   this run: p<.10  p<.05  p<.01"
          " | paper: .10   .05   .01")
    for k, (row, r) in enumerate(zip(ROWS, res)):
        if k == 8:
            print("same samples, gender coded 0/1 instead of +-0.5:")
        print(f"{row[0]:40s} {r[0]:5.1f}  {r[1]:5.1f}  {r[2]:5.1f} | {row[5]}")
# 15,000 simulated studies, no true effect   this run: p<.10  p<.05  p<.01 | paper: .10   .05   .01
# baseline: one DV, one test                10.1    5.1    0.9 |   (10.0, 5.0, 1.0 expected)
# A: two DVs (r = .5) or their mean         18.7    9.8    2.1 | 17.8    9.5    2.2
# B: add 10 per cell if not significant     14.9    7.9    1.5 | 14.5    7.7    1.6
# C: gender covariate or interaction        21.0   11.1    2.4 | 21.6   11.7    2.7
# D: drop (or keep) one of 3 conditions     23.1   12.4    2.7 | 23.2   12.6    2.8
# A + B                                     26.9   14.6    3.3 | 26.0   14.4    3.3
# A + B + C                                 48.7   29.1    7.5 | 50.9   30.9    8.4
# A + B + C + D                             79.7   57.9   19.6 | 81.5   60.7   21.5
# same samples, gender coded 0/1 instead of +-0.5:
# C, gender coded 0/1                       22.6   12.4    2.7 | 21.6   11.7    2.7
# A + B + C, gender coded 0/1               51.4   31.6    8.6 | 50.9   30.9    8.4
# A + B + C + D, gender coded 0/1           81.9   61.2   21.8 | 81.5   60.7   21.5

The re-run comes very close to the paper. The baseline is 5.1 %, matching the nominal 5 %; situations A, B, C and D alone give 9.8 %, 7.9 %, 11.1 % and 12.4 % (paper 9.5 %, 7.7 %, 11.7 %, 12.6 %); A + B gives 14.6 %. With effect coding, A + B + C gives 29.1 % and all four together 57.9 %, about 3 percentage points below the paper’s 60.7 %, more than Monte Carlo error (about 0.4 points) can explain. On the same samples with 0/1 coding, C gives 12.4 %, A + B + C 31.6 % and all four together 61.2 %, only 0.5 points from the paper and within Monte Carlo error. So the gap comes mainly from the coding of gender: the paper probably used 0/1 coding (a guess; the paper does not say). This is itself an example: an inconspicuous coding choice in a report can move the false-positive rate by 3 points. The conclusion stands: a nominal 5 % false-positive rate rises to about sixty per cent once four innocent-looking choices are stacked.

Left: false-positive rates for the situations in Table 1 of Simmons et al. (grey) and this book’s re-run (purple: gender coded \pm 0.5; blue: coded 0/1, which affects only the three situations involving C); dashed line, the nominal 5 %. Right: p-curves, the distribution of significant p-values: roughly flat with no effect and one test; piled up just below 0.05 with no effect but optional stopping; concentrated at very small p with a true effect d = 0.5.

Optional stopping

Situation B is the mildest form of optional stopping. Figure 1 of Simmons et al. shows more extreme versions: start with 10 or 20 per group, test after every 1, 5, 10 or 20 additional per group, stop as soon as p<0.05p < 0.05, up to a maximum of 50 per group. Starting at 10 and testing after every observation gives a false-positive rate of about 22 %.

A re-run of Figure 1 of the paper on 40,000 simulated data paths with no true effect, so that Monte Carlo error is about 0.2 points; the last test is always at 50 per group. The second part gives pp-curves: all significant results sorted into five bins by pp-value (0 to 0.01, 0.01 to 0.02, and so on), with the share in each bin.

Calculation

# Optional stopping (Figure 1 of Simmons et al. 2011) and the shape of the "p-curve"
# (Simonsohn, Nelson & Simmons 2014). Two groups, true effect ZERO unless stated;
# two-sided t tests. The researcher starts with 10 or 20 per group, tests after every 1,
# 5, 10 or 20 more per group, and stops at p < .05 or at 50 per group. 40,000 simulated
# paths per case (Monte Carlo SE about 0.2 points). Critical t by Cornish-Fisher.
import math, random
from statistics import NormalDist

N_PATHS = 40000

def t_crit(alpha, df):
    z = NormalDist().inv_cdf(1 - alpha / 2)
    return (z + (z ** 3 + z) / (4 * df)
            + (5 * z ** 5 + 16 * z ** 3 + 3 * z) / (96 * df ** 2)
            + (3 * z ** 7 + 19 * z ** 5 + 17 * z ** 3 - 15 * z) / (384 * df ** 3))

BINS = (0.01, 0.02, 0.03, 0.04, 0.05)
CRIT = {df: [t_crit(a, df) for a in BINS] for df in range(10, 100)}

def t_stat(s1, q1, s2, q2, n):
    m1, m2 = s1 / n, s2 / n
    sp2 = (q1 - n * m1 * m1 + q2 - n * m2 * m2) / (2 * n - 2)
    return abs(m1 - m2) / math.sqrt(2 * sp2 / n)

def p_bin(t, df):
    # index of the p-value bin (0: p < .01 ... 4: .04-.05); None if p >= .05
    return next((k for k in range(5) if t > CRIT[df][k]), None)

def path(rng, d, nmax=50):
    s1 = q1 = s2 = q2 = 0.0; out = []
    for n in range(1, nmax + 1):
        a, b = rng.gauss(d, 1), rng.gauss(0, 1)
        s1 += a; q1 += a * a; s2 += b; q2 += b * b
        out.append((s1, q1, s2, q2))
    return out

def first_hit(p, start, step, final=True):
    looks = list(range(start, 51, step))
    if final and looks[-1] != 50:
        looks.append(50)          # the last look is always at the 50-per-group limit
    return next((k for k in (p_bin(t_stat(*p[n - 1], n), 2 * n - 2) for n in looks)
                 if k is not None), None)

if __name__ == "__main__":
    rng = random.Random(1359)
    paths = [path(rng, 0.0) for _ in range(N_PATHS)]
    print("false-positive rate (%), stop at p < .05 or n = 50;"
          " test after every k more per group")
    for start in (10, 20):
        rates = [100 * sum(first_hit(p, start, k) is not None for p in paths) / N_PATHS
                 for k in (1, 5, 10, 20)]
        print(f"   start at n = {start}:  k = 1: {rates[0]:.1f}   k = 5: {rates[1]:.1f}"
              f"   k = 10: {rates[2]:.1f}   k = 20: {rates[3]:.1f}")
    print("p-curve: share of significant results with p in"
          " .00-.01 .01-.02 .02-.03 .03-.04 .04-.05")
    curves = [("no effect, one test at n = 20",
               [first_hit(p, 20, 99, False) for p in paths]),
              ("no effect, optional stopping (10, +1)",
               [first_hit(p, 10, 1) for p in paths])]
    for d in (0.5, 0.8):
        ps = [path(rng, d, 20) for _ in range(N_PATHS)]
        curves.append((f"true d = {d}, one test at n = 20",
                       [first_hit(p, 20, 99, False) for p in ps]))
    for label, ks in curves:
        sig = [k for k in ks if k is not None]
        shares = "   ".join(f"{sig.count(k) / len(sig):.2f}" for k in range(5))
        print(f"   {label:38s} {shares}   (n sig = {len(sig)})")
# false-positive rate (%), stop at p < .05 or n = 50; test after every k more per group
#    start at n = 10:  k = 1: 21.5   k = 5: 16.6   k = 10: 13.9   k = 20: 11.2
#    start at n = 20:  k = 1: 16.2   k = 5: 12.9   k = 10: 11.2   k = 20: 9.9
# p-curve: share of significant results with p in .00-.01 .01-.02 .02-.03 .03-.04 .04-.05
#    no effect, one test at n = 20          0.21   0.19   0.20   0.21   0.20   (n sig = 1908)
#    no effect, optional stopping (10, +1)  0.05   0.05   0.13   0.29   0.48   (n sig = 8615)
#    true d = 0.5, one test at n = 20       0.43   0.20   0.14   0.12   0.10   (n sig = 13464)
#    true d = 0.8, one test at n = 20       0.63   0.15   0.10   0.07   0.05   (n sig = 27840)

Starting at 10 per group and testing after every observation gives 21.5 %; testing after every 5, 10 or 20 gives 16.6 %, 13.9 % and 11.2 %; starting at 20 gives 16.2 %, 12.9 %, 11.2 % and 9.9 %. The paper’s text states about 22 %; the values marked in its figure are about 22.1 %, 17.0 %, 14.3 %, 12.7 % and 16.3 %, 13.3 %, 11.5 %, 10.4 % (as read from the figure; this book could not obtain the full text to check each one). The re-run lies 0.1 to 0.6 points below them, except “start at 10, test after every 20”, which is about 1.5 points lower, perhaps because the paper’s test points for that case differ from this book’s (10, 30 and 50 per group). Direction and size agree: the more often you look, the more false positives you get.

The pp-curve: clues in the distribution of significant results

Simonsohn, Nelson and Simmons went on to propose pp-curve analysis (Simonsohn et al. 2014). If a set of studies rests on a real effect, the significant pp-values should lean towards very small values (right skew); with no effect and no manipulation, they are roughly uniform between 0 and 0.05; with no effect but manipulation just past significance, they pile up near 0.05 (left skew).

The second part of the program shows all three shapes. With no effect and a single test, each of the five bins holds about 0.19 to 0.21, roughly uniform. With no effect but optional stopping, the 0.04-to-0.05 bin holds 48 % and the 0-to-0.01 bin only 5 %. With a true effect d=0.5d = 0.5 (20 per group), the 0-to-0.01 bin holds 43 %; with d=0.8d = 0.8, 63 %. So the pp-curve can to some extent separate “a literature with evidential value” from “a literature produced by manipulation”. It has limits: it assumes the results analysed are independent and selected by a clear rule, and when real effects and manipulation coexist the two shapes can cancel.

The garden of forking paths: no deliberate fishing needed

The simulations of Simmons et al. are easily read as “researchers deliberately trying many analyses”. The statisticians Gelman and Loken pointed out that the problem is more general (Gelman and Loken 2014). Even if researchers run only one analysis, if that analysis was settled after seeing the data (whom to exclude, how to code variables, which groups to compare, which subgroup to look at), and with other data they would have made other, equally reasonable choices, the premises of the pp-value calculation already fail. Borrowing the title of a story by Borges, they called this the garden of forking paths: at each fork only one path is taken, but which one depends on the data in hand.

So pp-hacking is often not fraud but a chain of “reasonable” responses to data by researchers without a prior plan; the remedy cannot rest on exhortation alone but needs procedural constraint: write down the choices at the forks before seeing the data (Chapter 6).

Evidence

The American Statistical Association’s statement on pp-values. In 2016 the ASA published six principles on pp-values, among them: pp-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by chance alone; scientific conclusions and business or policy decisions should not be based only on whether a pp-value passes a threshold; proper inference requires full reporting and transparency; a pp-value does not measure the size of an effect or the importance of a result (Wasserstein and Lazar 2016).

What Simmons et al. proposed

Simmons et al. set six requirements for authors: decide the rule for terminating data collection before it begins and report it; collect at least 20 observations per cell (or justify otherwise); list all variables collected; report all experimental conditions, including failed manipulations; if observations are eliminated, also report the results with them included; if an analysis includes a covariate, also report the results without it (Simmons et al. 2011). What these share is that they let readers see every fork in the garden, not only the path the researcher finally took.

Note

These numbers are not estimates for all research. The 60.7 % (or the re-run’s 57.9 % and 61.2 %) is a worst-case demonstration, with all four degrees of freedom exploited and a true effect of zero; it shows the mechanism, not the actual false-positive rate of any field.

Hypothesis · not established

Not yet established: degrees of freedom and failed replication. If researcher degrees of freedom are among the main causes of failed replication, then within a set of original studies, the more dependent variables, conditions or covariates the original paper reports (without prior registration), the lower the success rate of independent replication should be. If, after controlling for the original sample size and pp-value, there is no such association, the hypothesis is false.

Chapter summary

In data with no true effect, choosing among dependent variables, adding observations, adding covariates and dropping conditions each raise the false-positive rate from 5 % to roughly 8 % to 13 %, and together to about sixty per cent (60.7 % in the paper; in this book’s re-run 57.9 % with gender effect-coded and 61.2 % with 0/1 coding). The more often one looks at the data, the more false positives. The shape of the pp-curve can hint whether a literature has been manipulated. Gelman and Loken stressed that even without deliberate fishing, data-dependent choices of analysis rob pp-values of their meaning. The next chapter looks at another layer of selection: which results get published.

References

Gelman, Andrew, and Eric Loken. 2014. “The Statistical Crisis in Science.” American Scientist 102 (6): 460–65. https://doi.org/10.1511/2014.111.460.
Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22 (11): 1359–66. https://doi.org/10.1177/0956797611417632.
Simonsohn, Uri, Leif D. Nelson, and Joseph P. Simmons. 2014. “P-Curve: A Key to the File-Drawer.” Journal of Experimental Psychology: General 143 (2): 534–47.
Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA’s Statement on p-Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33.

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).