关于「四把尺子:邦费罗尼、霍尔姆、BH 阶梯与 BY 修正」,《伪发现率与文案规则检验》给出的定量结论是什么?

结论:看第五、六行就能明白多出来的两条从哪里来。R18 的 p 值是 0.001665,高于邦费罗尼门槛 0.00125,所以邦费罗尼不要它;但它是排第五的规则,BH 给第五级的门槛是 0.00625,它轻松过线。R33 的 p 值是 0.002019,第六级门槛是 0.00750,同样过线。排第七的 R03 是一条真规则,p 值 0.019500,高于第七级门槛 0.00875,四把尺子都没有拒绝它。真规则也会因为当周的随机波动而落在门槛之外,这是任何方法都躲不开的。

数据口径:书中转化率、访客数与规则条数均为示意参数;全部计算由书中程序生成;本书属于非个人化经营建议。 本书用精确概率推翻这条常识:四十条全部无效的规则同时检验,平均每周冒出两条假冠军,至少一条的概率为 0.8715;模拟中每周最佳假冠军宣称的相对提升平均为 34.2 %,而真实提升为零。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.00125, 0.001665, 0.002019, 0.00625, 0.00750, 0.00875

rank ruletruthp-valuek*q/mBonf HolmBHBY
1R38real0.0000480.00125xxxx
2R12real0.0000740.00250xxxx
3R26real0.0002800.00375xxxx
4R07real0.0006590.00500xxxx
5R18real0.0016650.00625..x.
6R33real0.0020190.00750..x.
7R03real0.0195000.00875....
8R19null0.0381900.01000....
9R02null0.0438170.01125....
10R24null0.0805890.01250....
11R21real0.0950150.01375....
12R32null0.1387300.01500....
raw :9 rejected, 2 false, 7 of 8 real found
Bonf:4 rejected, 0 false, 4 of 8 real found
Holm:4 rejected, 0 false, 4 of 8 real found
BH:6 rejected, 0 false, 6 of 8 real found
BY:4 rejected, 0 false, 4 of 8 real found
查看完整验算程序
# Chapter 2. p21: four rulers applied to one week of forty copy rules.
# Model: 40 rules R01-R40; eight (R03 R07 R12 R18 R21 R26 R33 R38)
# carry a real lift, a one-sided z-score with mean 3.0; the other 32
# are null with mean 0. Seed 20261113. Level 0.05 for every ruler:
# Bonferroni (p <= a/m), Holm step-down (Holm 1979), Benjamini-
# Hochberg step-up (BH 1995), Benjamini-Yekutieli (q / H_m, 2001).
# The data are illustrative; the four procedures are exact.
import math
import random

M, Q, MU, SEED = 40, 0.05, 3.0, 20261113
TRUE = (3, 7, 12, 18, 21, 26, 33, 38)


def week_pvalues():
    rng = random.Random(20261113)
    out = []
    for i in range(1, M + 1):
        mu = MU if i in TRUE else 0.0
        z = rng.gauss(mu, 1.0)
        out.append(("R{:02d}".format(i), 0.5 * math.erfc(z / math.sqrt(2))))
    return out


def order(ps):
    return sorted(range(len(ps)), key=lambda i: (ps[i], i))


def bonferroni(ps, a):
    return {i for i, p in enumerate(ps) if p <= a / len(ps)}


def holm(ps, a):
    m, rej = len(ps), set()
    for k, i in enumerate(order(ps)):
        if ps[i] > a / (m - k):
            break
        rej.add(i)
    return rej


def bh(ps, q):
    m, idx, kmax = len(ps), order(ps), 0
    for k, i in enumerate(idx, 1):
        if ps[i] <= k * q / m:
            kmax = k
    return set(idx[:kmax])


def by(ps, q):
    hm = math.fsum(1.0 / j for j in range(1, len(ps) + 1))
    return bh(ps, q / hm)


if __name__ == "__main__":
    data = week_pvalues()
    ps = [p for _, p in data]
    sets = [("Bonf", bonferroni(ps, Q)), ("Holm", holm(ps, Q)),
            ("BH", bh(ps, Q)), ("BY", by(ps, Q))]
    print("rank rule  truth   p-value   k*q/m   Bonf Holm  BH   BY")
    for k, i in enumerate(order(ps)[:12], 1):
        tag = "real" if i + 1 in TRUE else "null"
        marks = "".join(" {:>4}".format("x" if i in s else ".")
                        for _, s in sets)
        print(" {:2d}   {}   {}  {:.6f}  {:.5f} {}".format(
            k, data[i][0], tag, ps[i], k * Q / M, marks))
    raw = {i for i, p in enumerate(ps) if p < Q}
    for name, s in [("raw", raw)] + sets:
        fd = len([i for i in s if i + 1 not in TRUE])
        print("{:4s}: {:2d} rejected, {} false, {} of 8 real found".format(
            name, len(s), fd, len(s) - fd))

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《多重比较与伪发现率控制、同时检验数十条文案规则的本杰明尼霍赫伯格阶梯母典》第 2 章「四把尺子:邦费罗尼、霍尔姆、BH 阶梯与 BY 修正」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026307,
  author    = {Fan, Yafu},
  title     = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论