关于「实证闭环:三十站停止规则仪表、最小样本计算器与证伪方案」,《A/B 测试的偷看与小流量》给出的定量结论是什么?

结论:第八章给出三十站停止规则仪表、A/B 停止规则器的用法与预注册证伪方案。书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。

数据口径:书中的站点、流量、转化率与提升幅度均为示意参数;全部计算由书中程序生成;本书不是个人化的经营、法律或理财建议。 书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 19

sitevis/d base mde28 d@10% rule | same, next column
S0193 2.5%80.4% 1,368 NOAB | S161,094 3.3%17.9%87 SEQ
S02103 4.5%54.0%682 NOAB | S171,197 1.9%23.3%144 SEQ
S03121 4.3%50.5%606 NOAB | S181,385 5.5%12.1%41 SEQ
S04141 4.4%45.6%503 NOAB | S191,524 2.0%19.7%104 SEQ
S05164 4.8%40.1%398 NOAB | S202,252 4.0%11.2%35 SEQ
S06194 5.0%35.9%324 NOAB | S212,592 4.9%9.3%25 FIX
S07296 3.5%34.8%305 NOAB | S223,832 2.6%10.7%32 SEQ
S08333 2.1%43.6%462 NOAB | S234,540 3.8%8.1%19 FIX
S09337 3.2%34.5%300 NOAB | S244,767 1.9%11.5%37 SEQ
S10 345 4.4% 28.3% 208 SEQ | S25 8,395 4.1% 5.6% 10 FIX
S11377 4.0%28.4%209 SEQ| S26 15,636 3.1%4.8%7 FIX
S12502 1.3%46.1%510 NOAB | S27 16,934 4.4%3.8%5 FIX
S13648 3.1%24.7%161 SEQ| S28 21,122 1.6%5.7%10 FIX
S14667 5.2%18.2%90 SEQ| S29 22,825 1.6%5.5%9 FIX
S15861 4.7%16.9%78 SEQ| S30 27,642 1.1%6.2%11 FIX
FIX 8SEQ 12NOAB 10
查看完整验算程序
# Chapter 8. p81: a stopping-rule dashboard for thirty sites
# Model: thirty synthetic sites (seeded) with daily visitors log-uniform
# between 80 and 40,000 and base conversion uniform in 0.01 to 0.06.
# For each: relative MDE in 28 days (p22), days a fixed test needs for a
# 10% lift (p21), and the rule the book recommends:
#   FIX  : 10% lift is visible within 28 days -> fixed horizon, at most
#          5 O'Brien-Fleming looks (p51)
#   SEQ  : a 20% lift is visible within 56 days -> test only bold
#          changes, with an SPRT or mSPRT (p41, p61)
#   NOAB : neither -> no A/B test; pool traffic or change the metric
# All sites and thresholds are illustrative.
import math
import random
from statistics import NormalDist

N01 = NormalDist()


def n_per_arm(p1, rel, alpha=0.05, power=0.80):
    p2 = p1 * (1 + rel)
    za = N01.inv_cdf(1 - alpha / 2)
    zb = N01.inv_cdf(power)
    pb = (p1 + p2) / 2
    num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(
        p1 * (1 - p1) + p2 * (1 - p2))
    return (num / (p2 - p1)) ** 2


def mde(p1, n, lo=1e-4, hi=20.0):
    for _ in range(80):
        mid = (lo + hi) / 2
        if n_per_arm(p1, mid) > n:
            lo = mid
        else:
            hi = mid
    return hi


def rule(t, p):
    if 2 * n_per_arm(p, 0.10) / t <= 28:
        return "FIX"
    if 2 * n_per_arm(p, 0.20) / t <= 56:
        return "SEQ"
    return "NOAB"


def main():
    rng = random.Random(20261112)
    sites = []
    for i in range(30):
        t = int(math.exp(rng.uniform(math.log(80), math.log(40000))))
        p = rng.uniform(0.01, 0.06)
        sites.append((t, p))
    sites.sort()
    rows = []
    for i, (t, p) in enumerate(sites, 1):
        d10 = math.ceil(2 * n_per_arm(p, 0.10) / t)
        m28 = mde(p, t / 2 * 28)
        rows.append("S{:02d} {:6,d} {:4.1%} {:6.1%} {:5,d} {:4}".format(
            i, t, p, m28, d10, rule(t, p)))
    print("site  vis/d base mde28 d@10% rule | same, next column")
    for a, b in zip(rows[:15], rows[15:]):
        print((a + " | " + b).rstrip())
    tally = {}
    for t, p in sites:
        r = rule(t, p)
        tally[r] = tally.get(r, 0) + 1
    print("FIX {}  SEQ {}  NOAB {}".format(
        tally.get("FIX", 0), tally.get("SEQ", 0), tally.get("NOAB", 0)))


main()

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《A/B测试偷看早停假阳性膨胀、序贯概率比检验与小流量站点最小样本母典》第 8 章「实证闭环:三十站停止规则仪表、最小样本计算器与证伪方案」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论