关于「随时可看的推断、贝叶斯「胜出概率」与事前数据的极限」,《A/B 测试的偷看与小流量》给出的定量结论是什么?
结论:第六章说明混合序贯检验每天看一年误报率只有 0.0280;默认先验下的贝叶斯「胜出概率」在多数改动无效的世界里只有 0.2158 为真;事前数据要让小站一个月看见 5% 的提升,需要 0.9932 的相关系数。
数据口径:书中的站点、流量、转化率与提升幅度均为示意参数;全部计算由书中程序生成;本书不是个人化的经营、法律或理财建议。 书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.0280
| daily looks for up to 365 days, nominal alpha 0.05 | ||||
| rule | truth | by day 28 | by day 90 | by day 365 |
| naive | H0 | 0.2782 | 0.3668 | 0.4758 |
| naive | H1 | 0.8925 | 0.9992 | 1.0000 |
| mSPRT | H0 | 0.0107 | 0.0203 | 0.0280 |
| mSPRT | H1 | 0.5095 | 0.9858 | 1.0000 |
| mSPRT under H1: median stop day 28 (fixed test: 28) |
查看完整验算程序
# Chapter 6. p61: a mixture SPRT can be watched every day without cost
# Model: standardized daily increments x_i ~ N(theta, 1); S_n = sum x_i.
# Mixing the likelihood ratio over theta ~ N(0, tau^2) gives
# Lambda_n = (1 + n tau^2)^(-1/2) exp(tau^2 S_n^2 / (2 (1 + n tau^2)))
# (the mSPRT of Robbins 1970, used by Johari et al. 2022). Rejecting
# when Lambda_n >= 1 / alpha keeps P(ever reject | H0) <= alpha for all
# n by Ville's inequality. The naive rule rejects when |S_n|/sqrt(n) >=
# 1.96. Under H1 theta = 2.8 / sqrt(28), i.e. a fixed 28-day test would
# have power 0.80. tau = 0.5 and the 365-day horizon are illustrative.
# Seeded Monte Carlo.
import math
import random
ALPHA, TAU2, DAYS, SIMS = 0.05, 0.25, 365, 4000
Z = 1.959964
def run(theta, rng):
lim = math.log(1 / ALPHA)
m_stop, n_stop = [], []
for _ in range(SIMS):
s = 0.0
m = n = 0
for d in range(1, DAYS + 1):
s += rng.gauss(theta, 1.0)
v = 1 + d * TAU2
if not m and TAU2 * s * s / (2 * v) - 0.5 * math.log(v) >= lim:
m = d
if not n and s * s >= Z * Z * d:
n = d
if m and n:
break
m_stop.append(m)
n_stop.append(n)
return m_stop, n_stop
def share(stops, h):
return len([d for d in stops if 0 < d <= h]) / SIMS
def main():
rng = random.Random(20261112)
m0, n0 = run(0.0, rng)
th = 2.8 / math.sqrt(28)
m1, n1 = run(th, rng)
print("daily looks for up to 365 days, nominal alpha 0.05")
print("rule truth by day 28 by day 90 by day 365")
for name, st0, st1 in (("naive", n0, n1), ("mSPRT", m0, m1)):
for truth, st in (("H0", st0), ("H1", st1)):
print("{:6} {:5} {:9.4f} {:9.4f} {:10.4f}".format(
name, truth, share(st, 28), share(st, 90),
share(st, 365)))
hit = sorted(d for d in m1 if d > 0)
print("mSPRT under H1: median stop day {} (fixed test: 28)".format(
hit[len(hit) // 2]))
main()唯一数据源与引用
范雅福(Fan, Y.). (2026). 《A/B测试偷看早停假阳性膨胀、序贯概率比检验与小流量站点最小样本母典》第 6 章「随时可看的推断、贝叶斯「胜出概率」与事前数据的极限」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}