关于「膨胀从哪里来:随机游走、重对数律与被夸大的赢家」,《A/B 测试的偷看与小流量》给出的定量结论是什么?

结论:第三章用随机游走与重对数律解释膨胀的来源,并算出偷看的小站报告的提升平均是真实值的 18.3 倍。

数据口径:书中的站点、流量、转化率与提升幅度均为示意参数;全部计算由书中程序生成;本书不是个人化的经营、法律或理财建议。 书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 18.3

true lift 5% on base 0.03, 28 days, significant results only
site/dayrulepowerexaggerationwrong sign
200plan0.06117.30.165
200peek0.285518.30.339
2,000plan0.17972.40.009
2,000peek0.41525.30.117
20,000plan0.90121.10.000
20,000peek0.94791.70.009
查看完整验算程序
# Chapter 3. p32: a winner found by peeking is a winner exaggerated
# Model: true relative lift 5% on a base of 0.03 (illustrative). Each
# day's difference in conversion rate between arms is drawn as normal
# with mean delta and variance 2pq/n (n visitors per arm per day). The
# peeker stops at the first day with |z| >= 1.96 within 28 days; the
# planner tests once on day 28. Among significant results we report the
# exaggeration ratio (mean estimated lift / true lift, Gelman and
# Carlin's type M) and the share with the wrong sign (type S).
# Seeded Monte Carlo.
import math
import random

BASE = 0.03
LIFT = 0.05
DAYS = 28
SIMS = 20000
Z = 1.959964


def run(per_arm, rng):
    delta = BASE * LIFT
    q = BASE * (1 + LIFT / 2)
    sd = math.sqrt(2 * q * (1 - q) / per_arm)
    res = {"peek": [], "plan": []}
    for _ in range(SIMS):
        tot = 0.0
        done = False
        for d in range(1, DAYS + 1):
            tot += rng.gauss(delta, sd)
            est = tot / d
            z = est / (sd / math.sqrt(d))
            if not done and abs(z) >= Z:
                res["peek"].append(est)
                done = True
        if abs(z) >= Z:
            res["plan"].append(est)
    return res, delta


def main():
    rng = random.Random(20261112)
    print("true lift 5% on base 0.03, 28 days, significant results only")
    print("site/day  rule   power   exaggeration   wrong sign")
    for per_arm in (100, 1000, 10000):
        res, delta = run(per_arm, rng)
        for key in ("plan", "peek"):
            ests = res[key]
            pw = len(ests) / SIMS
            ex = math.fsum(abs(e) for e in ests) / len(ests) / delta
            ws = len([e for e in ests if e < 0]) / len(ests)
            print("{:8,d}  {:5}  {:6.4f}  {:12.1f}  {:11.3f}".format(
                2 * per_arm, key, pw, ex, ws))


main()

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《A/B测试偷看早停假阳性膨胀、序贯概率比检验与小流量站点最小样本母典》第 3 章「膨胀从哪里来:随机游走、重对数律与被夸大的赢家」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论