5D Library › 科学方法论、可证伪性与可重复性危机的数理统计 · English

第三章 p 值操纵、研究者自由度与分岔花园

听一首歌,年轻一岁半

2011 年,心理学家西蒙斯、纳尔逊与西蒙森发表了一个刻意荒唐的实验 (Simmons et al. 2011):听了披头士《当我六十四岁》的大学生,在控制父亲年龄之后,比听对照曲目的人「年轻」将近一岁半,p=0.040p = 0.040。听歌当然不能改变出生日期。这个结果是用当时完全合乎惯例的方法得到的:测了好几个变量只报告一个,边收集边看结果再决定是否继续,在几种分析中挑一种协变量。他们把这些选择称为研究者自由度。

四种自由度

西蒙斯等人用模拟量化了四种常见做法的后果。所有数据都来自真实效应为零的正态噪声,基线设计是两组、每组 20 人、一次 tt 检验。

  • A. 两个因变量: 测量两个相关系数为 0.5 的因变量,分别检验每一个以及二者的平均值,哪个显著就报告哪个。
  • B. 追加样本: 每组 20 人时检验一次;如果不显著,每组再加 10 人,再检验一次。
  • C. 协变量: 做普通 tt 检验、加入性别作为协变量、或检验性别与处理的交互作用,任何一种显著都算。
  • D. 条件取舍: 实验有三个条件(低、中、高),可以只比较其中任意两个,也可以检验线性趋势。

单独使用时,每一种都把名义上 5% 的假阳性率抬高到 8% 到 13%;四种合在一起,假阳性率达到 60.7%。这些数字见原文表 1,经核对无误:A 为 9.5%,B 为 7.7%,C 为 11.7%,D 为 12.6%,A + B 为 14.4%,A + B + C 为 30.9%,A + B + C + D 为 60.7%(p<0.05p < 0.05 一栏)。

按原文表 1 的说明重做这项模拟:15,000 个模拟样本,所有观察值独立地取自标准正态分布;每人以 0.5 的概率被设为女性;两个因变量的相关为 0.5。一个样本只要在该情形允许的任何一种分析中达到显著,就算一次「假阳性」。原文没有说明性别怎样编码,而在含交互作用的模型里这一点有影响:按效应编码(±0.5\pm 0.5)时,「条件」系数检验的是两种性别的平均效应;按 0/1 虚拟编码时,它检验的是参照性别之内的效应,相当于多了一种不同的检验。程序在同一批样本上两种都算。

推导

# Researcher degrees of freedom: a Monte Carlo re-run of Table 1 of Simmons, Nelson &
# Simonsohn (2011). Every effect is truly ZERO: all observations are independent N(0, 1)
# noise. Baseline: two conditions, 20 per cell, one t test. A: two dependent variables
# correlated r = .5; test DV1, DV2 or their mean. B: test at 20 per cell; if not
# significant add 10 per cell and test again. C: t test, ANCOVA with gender, or gender x
# condition (condition OR interaction counts); each person is female with probability .5.
# The paper does not state how gender is coded; in the interaction model this matters:
# effect coding (+-0.5) tests the AVERAGE condition effect, 0/1 coding tests it within
# one gender, a different test. Both are run on the same samples. D: three conditions;
# any of the three pairwise tests or the linear trend (-1, 0, 1). A result "counts" if
# ANY analysis allowed in that situation reaches the threshold. Critical t values by the
# Cornish-Fisher expansion (accurate to about 1e-4 for df > 30).
import math, random
from statistics import NormalDist

def t_crit(alpha, df):
    z = NormalDist().inv_cdf(1 - alpha / 2)
    return (z + (z ** 3 + z) / (4 * df)
            + (5 * z ** 5 + 16 * z ** 3 + 3 * z) / (96 * df ** 2)
            + (3 * z ** 7 + 19 * z ** 5 + 17 * z ** 3 - 15 * z) / (384 * df ** 3))

CRIT = {df: [t_crit(a, df) for a in (0.10, 0.05, 0.01)] for df in range(30, 90)}

def ols_level(cells, tests):
    # cells: (n, sum y, sum y^2, x); level = thresholds (.10/.05/.01) passed by any
    # tested beta
    p = len(cells[0][3])
    A = [[sum(n * x[i] * x[j] for n, _, _, x in cells) for j in range(p)]
         for i in range(p)]
    v = [sum(s * x[i] for _, s, _, x in cells) for i in range(p)]
    M = [row + [float(i == j) for j in range(p)] for i, row in enumerate(A)]
    for i in range(p):                      # Gauss-Jordan inverse of X'X
        if abs(M[i][i]) < 1e-12:
            return 0
        M[i] = [e / M[i][i] for e in M[i]]
        for k in range(p):
            if k != i:
                M[k] = [a - M[k][i] * b for a, b in zip(M[k], M[i])]
    inv = [row[p:] for row in M]
    b = [sum(inv[i][j] * v[j] for j in range(p)) for i in range(p)]
    df = sum(c[0] for c in cells) - p
    s2 = (sum(c[2] for c in cells) - sum(bi * vi for bi, vi in zip(b, v))) / df
    return max(sum(abs(b[j]) / math.sqrt(s2 * inv[j][j]) > c for c in CRIT[df])
               for j in tests)

DESIGNS = {  # condition codes used by each comparison
    "01": {0: -0.5, 1: 0.5}, "02": {0: -0.5, 2: 0.5}, "12": {1: -0.5, 2: 0.5},
    "trend": {0: -1.0, 1: 0.0, 2: 1.0}}

def one_sample(rng):
    data = []  # per condition: list of (gender, y1, y2)
    for _ in range(3):
        rows = []
        for _ in range(30):
            z1, z2 = rng.gauss(0, 1), rng.gauss(0, 1)
            g = 0.5 if rng.random() < 0.5 else -0.5
            rows.append((g, z1, 0.5 * z1 + math.sqrt(0.75) * z2))
        data.append(rows)
    levels = {}
    for stage in (20, 30):
        for dv in (0, 1, 2):
            cell = {}
            for k in range(3):
                for g, y1, y2 in data[k][:stage]:
                    y = (y1, y2, (y1 + y2) / 2)[dv]
                    c = cell.setdefault((k, g), [0, 0.0, 0.0])
                    c[0] += 1; c[1] += y; c[2] += y * y
            for dname, code in DESIGNS.items():
                cs = [(c[0], c[1], c[2], code[k], g) for (k, g), c in cell.items()
                      if k in code]
                models = {
                    "plain": ([(n, s, q, (1.0, x)) for n, s, q, x, g in cs], (1,)),
                    "cov": ([(n, s, q, (1.0, x, g)) for n, s, q, x, g in cs], (1,)),
                    "int": ([(n, s, q, (1.0, x, g, x * g)) for n, s, q, x, g in cs],
                            (1, 3)),
                    "int01": ([(n, s, q, (1.0, x, g + 0.5, x * (g + 0.5)))
                               for n, s, q, x, g in cs], (1, 3))}
                for mname, (cells, tests) in models.items():
                    levels[(stage, dv, dname, mname)] = ols_level(cells, tests)
    return levels

D3 = tuple(DESIGNS)
S2, DV3, C3, C01 = (20, 30), (0, 1, 2), ("plain", "cov", "int"), ("plain", "cov", "int01")
ROWS = [  # (label, stages, DVs, comparisons, models, Table 1: p < .10 / .05 / .01)
    ("baseline: one DV, one test", (20,), (0,), ("01",), ("plain",),
     "  (10.0, 5.0, 1.0 expected)"),
    ("A: two DVs (r = .5) or their mean", (20,), DV3, ("01",), ("plain",),
     "17.8    9.5    2.2"),
    ("B: add 10 per cell if not significant", S2, (0,), ("01",), ("plain",),
     "14.5    7.7    1.6"),
    ("C: gender covariate or interaction", (20,), (0,), ("01",), C3,
     "21.6   11.7    2.7"),
    ("D: drop (or keep) one of 3 conditions", (20,), (0,), D3, ("plain",),
     "23.2   12.6    2.8"),
    ("A + B", S2, DV3, ("01",), ("plain",), "26.0   14.4    3.3"),
    ("A + B + C", S2, DV3, ("01",), C3, "50.9   30.9    8.4"),
    ("A + B + C + D", S2, DV3, D3, C3, "81.5   60.7   21.5"),
    ("C, gender coded 0/1", (20,), (0,), ("01",), C01, "21.6   11.7    2.7"),
    ("A + B + C, gender coded 0/1", S2, DV3, ("01",), C01, "50.9   30.9    8.4"),
    ("A + B + C + D, gender coded 0/1", S2, DV3, D3, C01, "81.5   60.7   21.5")]

def run(n_sims=15000, seed=2011):
    rng = random.Random(seed)
    hits = [[0, 0, 0] for _ in ROWS]
    for _ in range(n_sims):
        lv = one_sample(rng)
        for r, (_, st, dvs, ds, ms, _) in enumerate(ROWS):
            best = max(lv[(s, v, d, m)] for s in st for v in dvs
                       for d in ds for m in ms)
            for k in range(best):
                hits[r][k] += 1
    return [[100 * h / n_sims for h in row] for row in hits]

if __name__ == "__main__":
    res = run()
    print("15,000 simulated studies, no true effect   this run: p<.10  p<.05  p<.01"
          " | paper: .10   .05   .01")
    for k, (row, r) in enumerate(zip(ROWS, res)):
        if k == 8:
            print("same samples, gender coded 0/1 instead of +-0.5:")
        print(f"{row[0]:40s} {r[0]:5.1f}  {r[1]:5.1f}  {r[2]:5.1f} | {row[5]}")
# 15,000 simulated studies, no true effect   this run: p<.10  p<.05  p<.01 | paper: .10   .05   .01
# baseline: one DV, one test                10.1    5.1    0.9 |   (10.0, 5.0, 1.0 expected)
# A: two DVs (r = .5) or their mean         18.7    9.8    2.1 | 17.8    9.5    2.2
# B: add 10 per cell if not significant     14.9    7.9    1.5 | 14.5    7.7    1.6
# C: gender covariate or interaction        21.0   11.1    2.4 | 21.6   11.7    2.7
# D: drop (or keep) one of 3 conditions     23.1   12.4    2.7 | 23.2   12.6    2.8
# A + B                                     26.9   14.6    3.3 | 26.0   14.4    3.3
# A + B + C                                 48.7   29.1    7.5 | 50.9   30.9    8.4
# A + B + C + D                             79.7   57.9   19.6 | 81.5   60.7   21.5
# same samples, gender coded 0/1 instead of +-0.5:
# C, gender coded 0/1                       22.6   12.4    2.7 | 21.6   11.7    2.7
# A + B + C, gender coded 0/1               51.4   31.6    8.6 | 50.9   30.9    8.4
# A + B + C + D, gender coded 0/1           81.9   61.2   21.8 | 81.5   60.7   21.5

重做的结果与原文非常接近。基线 5.1%,与名义上的 5% 一致;单项情形 A、B、C、D 分别为 9.8%、7.9%、11.1%、12.4%(原文 9.5%、7.7%、11.7%、12.6%);A + B 为 14.6%。按效应编码,A + B + C 为 29.1%,四种合在一起为 57.9%,比原文的 60.7% 低约 3 个百分点,蒙特卡洛误差(约 0.4 个百分点)不足以解释。同一批样本改用 0/1 编码后,C 为 12.4%,A + B + C 为 31.6%,四种合在一起为 61.2%,与原文相差只有 0.5 个百分点,在蒙特卡洛误差的范围之内。所以与原文的差距主要来自性别的编码方式:原文很可能用的是 0/1 编码(这只是推测,原文没有写明)。这件事本身也是一个例子:一个报告里不起眼的编码选择,就能让假阳性率相差 3 个百分点。结论不变:原本 5% 的假阳性率,在四种看似无害的选择叠加之后,升到了大约六成。

左:西蒙斯等人表 1 各情形的假阳性率(灰)与本书重做(紫:性别按 \pm 0.5 编码;蓝:按 0/1 编码,只影响含 C 的三项),虚线为名义 5%。右:p 曲线,即显著结果的 p 值分布:没有效应、只检验一次时大致均匀;没有效应但采用可选停止时堆积在 0.05 附近;真实效应 d = 0.5 时集中在很小的 p 值。

可选停止

情形 B 只是可选停止最温和的版本。西蒙斯等人的图 1 显示了更极端的做法:从每组 10 人或 20 人开始,每增加 1、5、10 或 20 人就检验一次,一旦 p<0.05p < 0.05 就停止,最多到每组 50 人。从 10 人开始、每加一人看一次时,假阳性率约为 22%。

用 40,000 条模拟数据路径(真实效应为零)重做原文图 1,蒙特卡洛误差约 0.2 个百分点;最后一次检验总在每组 50 人时进行。下半部分是 pp 曲线:把所有显著结果按 pp 值分成五档(0 到 0.01、0.01 到 0.02……),看各档所占的比例。

推导

# Optional stopping (Figure 1 of Simmons et al. 2011) and the shape of the "p-curve"
# (Simonsohn, Nelson & Simmons 2014). Two groups, true effect ZERO unless stated;
# two-sided t tests. The researcher starts with 10 or 20 per group, tests after every 1,
# 5, 10 or 20 more per group, and stops at p < .05 or at 50 per group. 40,000 simulated
# paths per case (Monte Carlo SE about 0.2 points). Critical t by Cornish-Fisher.
import math, random
from statistics import NormalDist

N_PATHS = 40000

def t_crit(alpha, df):
    z = NormalDist().inv_cdf(1 - alpha / 2)
    return (z + (z ** 3 + z) / (4 * df)
            + (5 * z ** 5 + 16 * z ** 3 + 3 * z) / (96 * df ** 2)
            + (3 * z ** 7 + 19 * z ** 5 + 17 * z ** 3 - 15 * z) / (384 * df ** 3))

BINS = (0.01, 0.02, 0.03, 0.04, 0.05)
CRIT = {df: [t_crit(a, df) for a in BINS] for df in range(10, 100)}

def t_stat(s1, q1, s2, q2, n):
    m1, m2 = s1 / n, s2 / n
    sp2 = (q1 - n * m1 * m1 + q2 - n * m2 * m2) / (2 * n - 2)
    return abs(m1 - m2) / math.sqrt(2 * sp2 / n)

def p_bin(t, df):
    # index of the p-value bin (0: p < .01 ... 4: .04-.05); None if p >= .05
    return next((k for k in range(5) if t > CRIT[df][k]), None)

def path(rng, d, nmax=50):
    s1 = q1 = s2 = q2 = 0.0; out = []
    for n in range(1, nmax + 1):
        a, b = rng.gauss(d, 1), rng.gauss(0, 1)
        s1 += a; q1 += a * a; s2 += b; q2 += b * b
        out.append((s1, q1, s2, q2))
    return out

def first_hit(p, start, step, final=True):
    looks = list(range(start, 51, step))
    if final and looks[-1] != 50:
        looks.append(50)          # the last look is always at the 50-per-group limit
    return next((k for k in (p_bin(t_stat(*p[n - 1], n), 2 * n - 2) for n in looks)
                 if k is not None), None)

if __name__ == "__main__":
    rng = random.Random(1359)
    paths = [path(rng, 0.0) for _ in range(N_PATHS)]
    print("false-positive rate (%), stop at p < .05 or n = 50;"
          " test after every k more per group")
    for start in (10, 20):
        rates = [100 * sum(first_hit(p, start, k) is not None for p in paths) / N_PATHS
                 for k in (1, 5, 10, 20)]
        print(f"   start at n = {start}:  k = 1: {rates[0]:.1f}   k = 5: {rates[1]:.1f}"
              f"   k = 10: {rates[2]:.1f}   k = 20: {rates[3]:.1f}")
    print("p-curve: share of significant results with p in"
          " .00-.01 .01-.02 .02-.03 .03-.04 .04-.05")
    curves = [("no effect, one test at n = 20",
               [first_hit(p, 20, 99, False) for p in paths]),
              ("no effect, optional stopping (10, +1)",
               [first_hit(p, 10, 1) for p in paths])]
    for d in (0.5, 0.8):
        ps = [path(rng, d, 20) for _ in range(N_PATHS)]
        curves.append((f"true d = {d}, one test at n = 20",
                       [first_hit(p, 20, 99, False) for p in ps]))
    for label, ks in curves:
        sig = [k for k in ks if k is not None]
        shares = "   ".join(f"{sig.count(k) / len(sig):.2f}" for k in range(5))
        print(f"   {label:38s} {shares}   (n sig = {len(sig)})")
# false-positive rate (%), stop at p < .05 or n = 50; test after every k more per group
#    start at n = 10:  k = 1: 21.5   k = 5: 16.6   k = 10: 13.9   k = 20: 11.2
#    start at n = 20:  k = 1: 16.2   k = 5: 12.9   k = 10: 11.2   k = 20: 9.9
# p-curve: share of significant results with p in .00-.01 .01-.02 .02-.03 .03-.04 .04-.05
#    no effect, one test at n = 20          0.21   0.19   0.20   0.21   0.20   (n sig = 1908)
#    no effect, optional stopping (10, +1)  0.05   0.05   0.13   0.29   0.48   (n sig = 8615)
#    true d = 0.5, one test at n = 20       0.43   0.20   0.14   0.12   0.10   (n sig = 13464)
#    true d = 0.8, one test at n = 20       0.63   0.15   0.10   0.07   0.05   (n sig = 27840)

从每组 10 人开始、每加一人看一次,假阳性率为 21.5%;每 5、10、20 人看一次时为 16.6%、13.9%、11.2%;从 20 人开始时分别为 16.2%、12.9%、11.2%、9.9%。原文正文明确写的是约 22%;原文图中标出的数值约为 22.1%、17.0%、14.3%、12.7% 与 16.3%、13.3%、11.5%、10.4%(据图读出,本书未能取得原文全文逐一核对)。重做的数字比它们低 0.1 到 0.6 个百分点,只有「从 10 人开始、每 20 人看一次」低约 1.5 个百分点,可能是原文这一档的检验时点与本书(10、30、50 人)不同。方向与大小都一致:看得越频繁,假阳性越多。

pp 曲线:从显著结果的分布中看出端倪

西蒙森、纳尔逊与西蒙斯随后提出了 pp 曲线分析 (Simonsohn et al. 2014):如果一组研究背后确实有真实效应,显著结果的 pp 值应当偏向很小的值(右偏);如果没有效应、也没有操纵,pp 值在 0 到 0.05 之间大致均匀;如果没有效应、却被操纵到刚好显著,pp 值就会堆积在 0.05 附近(左偏)。

程序的后半部分显示了这三种形状。没有效应、只检验一次时,五档各占约 0.19 到 0.21,大致均匀。没有效应但采用可选停止时,0.04 到 0.05 一档占 48%,0 到 0.01 一档只占 5%。真实效应 d=0.5d = 0.5(每组 20 人)时,0 到 0.01 一档占 43%;d=0.8d = 0.8 时占 63%。所以 pp 曲线能在一定程度上区分「有证据价值的文献」与「被操纵出来的文献」。它也有局限:它假设被分析的结果彼此独立、选取规则明确,而且当真实效应与操纵同时存在时,两种形状会互相抵消。

分岔花园:不需要故意钓鱼

西蒙斯等人的模拟容易被理解为「研究者故意尝试了很多种分析」。统计学家格尔曼与洛肯指出,问题比这更普遍 (Gelman and Loken 2014)。即使研究者只做了一种分析,如果这种分析是在看到数据之后才确定的(排除哪些人、怎样给变量编码、比较哪两组、看哪个子群),而换一批数据他们就会作出别的同样合理的选择,那么 pp 值的计算前提就已经不成立了。他们借用博尔赫斯的小说名,把这种情形称为分岔小径的花园:每一个岔口都只走了一条路,但走哪一条取决于眼前的数据。

所以 pp 值操纵常常不是欺诈,而是没有事先计划的研究者对数据作出的一连串「合理」反应;解决办法不能只靠道德劝诫,而要靠程序约束:在看到数据之前写下岔口上的选择(第六章)。

证据

美国统计学会关于 pp 值的声明。 2016 年,美国统计学会发表了关于 pp 值的六条原则,其中包括:pp 值不衡量所研究的假说为真的概率,也不衡量数据只是偶然产生的概率;科学结论与商业或政策决定不应只根据 pp 值是否越过某个门槛;正确的推断需要完整的报告与透明度;pp 值不衡量效应的大小或结果的重要性 (Wasserstein and Lazar 2016)。

西蒙斯等人的建议

西蒙斯等人在论文中给作者提出了六条要求:在开始收集数据之前确定停止收集的规则并在论文中报告;每个条件至少 20 个观察值(或说明理由);列出所有测量的变量;报告所有实验条件,包括失败的操纵;如果剔除了观察值,同时报告不剔除时的结果;如果分析中使用了协变量,同时报告不使用协变量时的结果 (Simmons et al. 2011)。这些要求的共同点是:让读者看见花园里所有的岔口,而不只是研究者最后走的那一条。

注

这些数字不是对所有研究的估计。 60.7%(或本书重做的 57.9%、61.2%)是四种自由度全部被利用、真实效应为零时的「最坏情形」示范,说明的是机制,不是任何领域的实际假阳性率。

假说 · 尚未验证

尚未确立:自由度与重复失败。 如果研究者自由度是重复失败的主要原因之一,那么在同一批原研究中,原文报告的因变量、条件或协变量越多(而且没有事先注册),独立重复的成功率应当越低。如果在控制原研究样本量与 pp 值之后,二者没有关联,这个假说就是错的。

本章小结

在真实效应为零的数据中,挑选因变量、追加样本、加入协变量与取舍条件四种做法,各自把假阳性率从 5% 抬高到约 8% 到 13%,合在一起达到约六成(原文 60.7%;本书重做,性别按效应编码为 57.9%,按 0/1 编码为 61.2%)。可选停止越频繁,假阳性越多。pp 曲线的形状可以提示一组文献是否被操纵。格尔曼与洛肯强调,即使没有故意钓鱼,依赖数据的分析选择也会让 pp 值失去意义。下一章讨论另一层筛选:什么样的结果会被发表。

参考文献

Gelman, Andrew, and Eric Loken. 2014. “The Statistical Crisis in Science.” American Scientist 102 (6): 460–65. https://doi.org/10.1511/2014.111.460.
Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22 (11): 1359–66. https://doi.org/10.1177/0956797611417632.
Simonsohn, Uri, Leif D. Nelson, and Joseph P. Simmons. 2014. “P-Curve: A Key to the File-Drawer.” Journal of Experimental Psychology: General 143 (2): 534–47.
Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA’s Statement on p-Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33.

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).