关于「四把尺子:邦费罗尼、霍尔姆、BH 阶梯与 BY 修正」,《伪发现率与文案规则检验》给出的定量结论是什么?
结论:看第五、六行就能明白多出来的两条从哪里来。R18 的 p 值是 0.001665,高于邦费罗尼门槛 0.00125,所以邦费罗尼不要它;但它是排第五的规则,BH 给第五级的门槛是 0.00625,它轻松过线。R33 的 p 值是 0.002019,第六级门槛是 0.00750,同样过线。排第七的 R03 是一条真规则,p 值 0.019500,高于第七级门槛 0.00875,四把尺子都没有拒绝它。真规则也会因为当周的随机波动而落在门槛之外,这是任何方法都躲不开的。
数据口径:书中转化率、访客数与规则条数均为示意参数;全部计算由书中程序生成;本书属于非个人化经营建议。 本书用精确概率推翻这条常识:四十条全部无效的规则同时检验,平均每周冒出两条假冠军,至少一条的概率为 0.8715;模拟中每周最佳假冠军宣称的相对提升平均为 34.2 %,而真实提升为零。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.00125, 0.001665, 0.002019, 0.00625, 0.00750, 0.00875
| rank rule | truth | p-value | k*q/m | Bonf Holm | BH | BY | ||
| 1 | R38 | real | 0.000048 | 0.00125 | x | x | x | x |
| 2 | R12 | real | 0.000074 | 0.00250 | x | x | x | x |
| 3 | R26 | real | 0.000280 | 0.00375 | x | x | x | x |
| 4 | R07 | real | 0.000659 | 0.00500 | x | x | x | x |
| 5 | R18 | real | 0.001665 | 0.00625 | . | . | x | . |
| 6 | R33 | real | 0.002019 | 0.00750 | . | . | x | . |
| 7 | R03 | real | 0.019500 | 0.00875 | . | . | . | . |
| 8 | R19 | null | 0.038190 | 0.01000 | . | . | . | . |
| 9 | R02 | null | 0.043817 | 0.01125 | . | . | . | . |
| 10 | R24 | null | 0.080589 | 0.01250 | . | . | . | . |
| 11 | R21 | real | 0.095015 | 0.01375 | . | . | . | . |
| 12 | R32 | null | 0.138730 | 0.01500 | . | . | . | . |
| raw : | 9 rejected, 2 false, 7 of 8 real found | |||||||
| Bonf: | 4 rejected, 0 false, 4 of 8 real found | |||||||
| Holm: | 4 rejected, 0 false, 4 of 8 real found | |||||||
| BH | : | 6 rejected, 0 false, 6 of 8 real found | ||||||
| BY | : | 4 rejected, 0 false, 4 of 8 real found |
查看完整验算程序
# Chapter 2. p21: four rulers applied to one week of forty copy rules.
# Model: 40 rules R01-R40; eight (R03 R07 R12 R18 R21 R26 R33 R38)
# carry a real lift, a one-sided z-score with mean 3.0; the other 32
# are null with mean 0. Seed 20261113. Level 0.05 for every ruler:
# Bonferroni (p <= a/m), Holm step-down (Holm 1979), Benjamini-
# Hochberg step-up (BH 1995), Benjamini-Yekutieli (q / H_m, 2001).
# The data are illustrative; the four procedures are exact.
import math
import random
M, Q, MU, SEED = 40, 0.05, 3.0, 20261113
TRUE = (3, 7, 12, 18, 21, 26, 33, 38)
def week_pvalues():
rng = random.Random(20261113)
out = []
for i in range(1, M + 1):
mu = MU if i in TRUE else 0.0
z = rng.gauss(mu, 1.0)
out.append(("R{:02d}".format(i), 0.5 * math.erfc(z / math.sqrt(2))))
return out
def order(ps):
return sorted(range(len(ps)), key=lambda i: (ps[i], i))
def bonferroni(ps, a):
return {i for i, p in enumerate(ps) if p <= a / len(ps)}
def holm(ps, a):
m, rej = len(ps), set()
for k, i in enumerate(order(ps)):
if ps[i] > a / (m - k):
break
rej.add(i)
return rej
def bh(ps, q):
m, idx, kmax = len(ps), order(ps), 0
for k, i in enumerate(idx, 1):
if ps[i] <= k * q / m:
kmax = k
return set(idx[:kmax])
def by(ps, q):
hm = math.fsum(1.0 / j for j in range(1, len(ps) + 1))
return bh(ps, q / hm)
if __name__ == "__main__":
data = week_pvalues()
ps = [p for _, p in data]
sets = [("Bonf", bonferroni(ps, Q)), ("Holm", holm(ps, Q)),
("BH", bh(ps, Q)), ("BY", by(ps, Q))]
print("rank rule truth p-value k*q/m Bonf Holm BH BY")
for k, i in enumerate(order(ps)[:12], 1):
tag = "real" if i + 1 in TRUE else "null"
marks = "".join(" {:>4}".format("x" if i in s else ".")
for _, s in sets)
print(" {:2d} {} {} {:.6f} {:.5f} {}".format(
k, data[i][0], tag, ps[i], k * Q / M, marks))
raw = {i for i, p in enumerate(ps) if p < Q}
for name, s in [("raw", raw)] + sets:
fd = len([i for i in s if i + 1 not in TRUE])
print("{:4s}: {:2d} rejected, {} false, {} of 8 real found".format(
name, len(s), fd, len(s) - fd))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《多重比较与伪发现率控制、同时检验数十条文案规则的本杰明尼霍赫伯格阶梯母典》第 2 章「四把尺子:邦费罗尼、霍尔姆、BH 阶梯与 BY 修正」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}