关于「偷看的价格:每天一看、显著即停的假阳性精确账」,《A/B 测试的偷看与小流量》给出的定量结论是什么?

结论:第一章用格点递推精确计算:每天看一次、显著即停,两周后的假阳性率是 0.2197,一个月后是 0.2802,并修正了立项时「两到三成」的说法。

数据口径:书中的站点、流量、转化率与提升幅度均为示意参数;全部计算由书中程序生成;本书不是个人化的经营、法律或理财建议。 书中以标注框区分已发表的证据、可运行的 Python 计算(19 段程序)、原典与作者的假说。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.2197, 0.2802

A/A test, two-sided z at every look, nominal alpha 0.05
looksP(at least one 'significant')x nominal
10.05001.00
20.08311.66
30.10732.15
50.14172.83
70.16633.33
100.19343.87
140.21974.39
200.24804.96
300.28025.60
600.33476.69
daily for 2 weeks : 0.2197
daily for 30 days : 0.2802
daily for 60 days : 0.3347
查看完整验算程序
# Chapter 1. p11: exact false-positive rate of "look daily, stop if p<0.05"
# Model: an A/A test (no true difference); at every look a two-sided z
# test, stop as soon as |z| >= 1.96. Equal information arrives between
# looks, so S_k is a sum of k iid N(0,1) and z_k = S_k / sqrt(k).
# A grid recursion (the numerical-integration idea of Armitage, McPherson
# and Rowe 1969) gives the probability of at least one 'significant'
# look; grid width 0.05 sd, no Monte Carlo.
# Parameters: alpha 0.05 by convention; number of looks illustrative.
import math
from statistics import NormalDist

N01 = NormalDist()
H = 0.05          # grid width in standard deviations
WIDTH = 7.0       # kernel cut at 7 sd


def cross_curve(bounds, h=H):
    """Cumulative P(|S_k| >= b_k*sqrt(k) at some look <= k)."""
    cdf = N01.cdf
    m = int(WIDTH / h) + 1
    kern = [cdf((d + .5) * h) - cdf((d - .5) * h)
            for d in range(-m, m + 1)]
    mass, n0 = [1.0], 0
    out = []
    for k, b in enumerate(bounds, 1):
        lim = b * math.sqrt(k)
        n = int(lim / h + 0.5)
        new = []
        for i in range(-n, n + 1):
            lo = max((i - .5) * h, -lim)
            hi = min((i + .5) * h, lim)
            if hi <= lo:
                new.append(0.0)
                continue
            full = lo == (i - .5) * h and hi == (i + .5) * h
            acc = []
            for jj, w in enumerate(mass):
                d = i - (jj - n0)
                if w == 0.0 or d < -m or d > m:
                    continue
                if full:
                    acc.append(w * kern[d + m])
                else:
                    u = (jj - n0) * h
                    acc.append(w * (cdf(hi - u) - cdf(lo - u)))
            new.append(math.fsum(acc))
        mass, n0 = new, n
        out.append(1.0 - math.fsum(mass))
    return out


def main():
    z = N01.inv_cdf(0.975)
    looks = [1, 2, 3, 5, 7, 10, 14, 20, 30, 60]
    curve = cross_curve([z] * looks[-1])
    print("A/A test, two-sided z at every look, nominal alpha 0.05")
    print("looks  P(at least one 'significant')  x nominal")
    for k in looks:
        p = curve[k - 1]
        print("{:5d}  {:29.4f}  {:9.2f}".format(k, p, p / 0.05))
    print("daily for 2 weeks : {:.4f}".format(curve[13]))
    print("daily for 30 days : {:.4f}".format(curve[29]))
    print("daily for 60 days : {:.4f}".format(curve[59]))


main()

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《A/B测试偷看早停假阳性膨胀、序贯概率比检验与小流量站点最小样本母典》第 1 章「偷看的价格:每天一看、显著即停的假阳性精确账」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论