关于「小站的天花板:功效、最小可检测效应与五年半的等待」,《A/B 测试的偷看与小流量》给出的定量结论是什么?
结论:第二章说明基线 3%、检出 5% 的相对提升每组需要 207,938 个访客,日均 200 访客要 5.7 年,一个月测试的功效只有 0.0609。
数据口径:书中的站点、流量、转化率与提升幅度均为示意参数;全部计算由书中程序生成;本书不是个人化的经营、法律或理财建议。 书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 200, 207938, 5.7
| visitors per arm, alpha 0.05 two-sided, power 0.80 | ||||
| base | lift | n per arm | Lehr 16 | days at 200/day |
| 0.01 | 5% | 637,010 | 649,276 | 6,371 |
| 0.01 | 10% | 163,095 | 166,236 | 1,631 |
| 0.01 | 20% | 42,693 | 43,516 | 427 |
| 0.01 | 50% | 7,750 | 7,901 | 78 |
| 0.03 | 5% | 207,938 | 211,943 | 2,080 |
| 0.03 | 10% | 53,211 | 54,236 | 533 |
| 0.03 | 20% | 13,914 | 14,183 | 140 |
| 0.03 | 50% | 2,518 | 2,567 | 26 |
| 0.05 | 5% | 122,124 | 124,476 | 1,222 |
| 0.05 | 10% | 31,234 | 31,836 | 313 |
| 0.05 | 20% | 8,158 | 8,317 | 82 |
| 0.05 | 50% | 1,471 | 1,500 | 15 |
| z_0.975 + z_0.80 = 1.96 + 0.84 = 2.80; 2 x 2.80^2 = 15.7 | ||||
| 3% base, 5% lift: 5.7 years of 200 visitors a day |
查看完整验算程序
# Chapter 2. p21: visitors per arm to detect a relative lift in conversion
# Model: two-sided two-proportion z test, alpha 0.05, power 0.80, normal
# approximation with the pooled variance under H0 and the unpooled one
# under H1 (the textbook formula, no continuity correction). Lehr's
# rule n = 16 * pbar * (1 - pbar) / delta^2 is printed for comparison.
# Baselines and lifts are illustrative; traffic is 200 visitors a day.
import math
from statistics import NormalDist
N01 = NormalDist()
def n_per_arm(p1, rel, alpha=0.05, power=0.80):
p2 = p1 * (1 + rel)
za = N01.inv_cdf(1 - alpha / 2)
zb = N01.inv_cdf(power)
pb = (p1 + p2) / 2
num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(
p1 * (1 - p1) + p2 * (1 - p2))
return math.ceil((num / (p2 - p1)) ** 2)
def lehr(p1, rel):
p2 = p1 * (1 + rel)
pb = (p1 + p2) / 2
return math.ceil(16 * pb * (1 - pb) / (p2 - p1) ** 2)
def main():
print("visitors per arm, alpha 0.05 two-sided, power 0.80")
print("base lift n per arm Lehr 16 days at 200/day")
for p1 in (0.01, 0.03, 0.05):
for rel in (0.05, 0.10, 0.20, 0.50):
n = n_per_arm(p1, rel)
days = math.ceil(2 * n / 200)
print("{:.2f} {:4.0%} {:11,d} {:9,d} {:15,d}".format(
p1, rel, n, lehr(p1, rel), days))
za, zb = N01.inv_cdf(0.975), N01.inv_cdf(0.80)
print("z_0.975 + z_0.80 = {:.2f} + {:.2f} = {:.2f};".format(
za, zb, za + zb), "2 x {:.2f}^2 = {:.1f}".format(
za + zb, 2 * (za + zb) ** 2))
n = n_per_arm(0.03, 0.05)
print("3% base, 5% lift: {:.1f} years of 200 visitors a day".format(
2 * n / 200 / 365))
main()唯一数据源与引用
范雅福(Fan, Y.). (2026). 《A/B测试偷看早停假阳性膨胀、序贯概率比检验与小流量站点最小样本母典》第 2 章「小站的天花板:功效、最小可检测效应与五年半的等待」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}