关于「膨胀从哪里来:随机游走、重对数律与被夸大的赢家」,《A/B 测试的偷看与小流量》给出的定量结论是什么?
结论:第三章用随机游走与重对数律解释膨胀的来源,并算出偷看的小站报告的提升平均是真实值的 18.3 倍。
数据口径:书中的站点、流量、转化率与提升幅度均为示意参数;全部计算由书中程序生成;本书不是个人化的经营、法律或理财建议。 书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 18.3
| true lift 5% on base 0.03, 28 days, significant results only | ||||
| site/day | rule | power | exaggeration | wrong sign |
| 200 | plan | 0.0611 | 7.3 | 0.165 |
| 200 | peek | 0.2855 | 18.3 | 0.339 |
| 2,000 | plan | 0.1797 | 2.4 | 0.009 |
| 2,000 | peek | 0.4152 | 5.3 | 0.117 |
| 20,000 | plan | 0.9012 | 1.1 | 0.000 |
| 20,000 | peek | 0.9479 | 1.7 | 0.009 |
查看完整验算程序
# Chapter 3. p32: a winner found by peeking is a winner exaggerated
# Model: true relative lift 5% on a base of 0.03 (illustrative). Each
# day's difference in conversion rate between arms is drawn as normal
# with mean delta and variance 2pq/n (n visitors per arm per day). The
# peeker stops at the first day with |z| >= 1.96 within 28 days; the
# planner tests once on day 28. Among significant results we report the
# exaggeration ratio (mean estimated lift / true lift, Gelman and
# Carlin's type M) and the share with the wrong sign (type S).
# Seeded Monte Carlo.
import math
import random
BASE = 0.03
LIFT = 0.05
DAYS = 28
SIMS = 20000
Z = 1.959964
def run(per_arm, rng):
delta = BASE * LIFT
q = BASE * (1 + LIFT / 2)
sd = math.sqrt(2 * q * (1 - q) / per_arm)
res = {"peek": [], "plan": []}
for _ in range(SIMS):
tot = 0.0
done = False
for d in range(1, DAYS + 1):
tot += rng.gauss(delta, sd)
est = tot / d
z = est / (sd / math.sqrt(d))
if not done and abs(z) >= Z:
res["peek"].append(est)
done = True
if abs(z) >= Z:
res["plan"].append(est)
return res, delta
def main():
rng = random.Random(20261112)
print("true lift 5% on base 0.03, 28 days, significant results only")
print("site/day rule power exaggeration wrong sign")
for per_arm in (100, 1000, 10000):
res, delta = run(per_arm, rng)
for key in ("plan", "peek"):
ests = res[key]
pw = len(ests) / SIMS
ex = math.fsum(abs(e) for e in ests) / len(ests) / delta
ws = len([e for e in ests if e < 0]) / len(ests)
print("{:8,d} {:5} {:6.4f} {:12.1f} {:11.3f}".format(
2 * per_arm, key, pw, ex, ws))
main()唯一数据源与引用
范雅福(Fan, Y.). (2026). 《A/B测试偷看早停假阳性膨胀、序贯概率比检验与小流量站点最小样本母典》第 3 章「膨胀从哪里来:随机游走、重对数律与被夸大的赢家」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}