关于「小站的天花板:功效、最小可检测效应与五年半的等待」,《A/B 测试的偷看与小流量》给出的定量结论是什么?

结论:第二章说明基线 3%、检出 5% 的相对提升每组需要 207,938 个访客,日均 200 访客要 5.7 年,一个月测试的功效只有 0.0609。

数据口径:书中的站点、流量、转化率与提升幅度均为示意参数;全部计算由书中程序生成;本书不是个人化的经营、法律或理财建议。 书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 200, 207938, 5.7

visitors per arm, alpha 0.05 two-sided, power 0.80
baseliftn per armLehr 16days at 200/day
0.015%637,010649,2766,371
0.0110%163,095166,2361,631
0.0120%42,69343,516427
0.0150%7,7507,90178
0.035%207,938211,9432,080
0.0310%53,21154,236533
0.0320%13,91414,183140
0.0350%2,5182,56726
0.055%122,124124,4761,222
0.0510%31,23431,836313
0.0520%8,1588,31782
0.0550%1,4711,50015
z_0.975 + z_0.80 = 1.96 + 0.84 = 2.80; 2 x 2.80^2 = 15.7
3% base, 5% lift: 5.7 years of 200 visitors a day
查看完整验算程序
# Chapter 2. p21: visitors per arm to detect a relative lift in conversion
# Model: two-sided two-proportion z test, alpha 0.05, power 0.80, normal
# approximation with the pooled variance under H0 and the unpooled one
# under H1 (the textbook formula, no continuity correction). Lehr's
# rule n = 16 * pbar * (1 - pbar) / delta^2 is printed for comparison.
# Baselines and lifts are illustrative; traffic is 200 visitors a day.
import math
from statistics import NormalDist

N01 = NormalDist()


def n_per_arm(p1, rel, alpha=0.05, power=0.80):
    p2 = p1 * (1 + rel)
    za = N01.inv_cdf(1 - alpha / 2)
    zb = N01.inv_cdf(power)
    pb = (p1 + p2) / 2
    num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(
        p1 * (1 - p1) + p2 * (1 - p2))
    return math.ceil((num / (p2 - p1)) ** 2)


def lehr(p1, rel):
    p2 = p1 * (1 + rel)
    pb = (p1 + p2) / 2
    return math.ceil(16 * pb * (1 - pb) / (p2 - p1) ** 2)


def main():
    print("visitors per arm, alpha 0.05 two-sided, power 0.80")
    print("base  lift    n per arm    Lehr 16    days at 200/day")
    for p1 in (0.01, 0.03, 0.05):
        for rel in (0.05, 0.10, 0.20, 0.50):
            n = n_per_arm(p1, rel)
            days = math.ceil(2 * n / 200)
            print("{:.2f}  {:4.0%}  {:11,d}  {:9,d}  {:15,d}".format(
                p1, rel, n, lehr(p1, rel), days))
    za, zb = N01.inv_cdf(0.975), N01.inv_cdf(0.80)
    print("z_0.975 + z_0.80 = {:.2f} + {:.2f} = {:.2f};".format(
        za, zb, za + zb), "2 x {:.2f}^2 = {:.1f}".format(
            za + zb, 2 * (za + zb) ** 2))
    n = n_per_arm(0.03, 0.05)
    print("3% base, 5% lift: {:.1f} years of 200 visitors a day".format(
        2 * n / 200 / 365))


main()

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《A/B测试偷看早停假阳性膨胀、序贯概率比检验与小流量站点最小样本母典》第 2 章「小站的天花板:功效、最小可检测效应与五年半的等待」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论