What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant"”?

Answer: Chapter 1 computes exactly, by grid recursion, that looking once a day and stopping when significant gives a false-positive rate of 0.2197 after two weeks and 0.2802 after a month, correcting the founding claim of "a fifth to nearly a third".

Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.2197, 0.2802

A/A test, two-sided z at every look, nominal alpha 0.05
looksP(at least one 'significant')x nominal
10.05001.00
20.08311.66
30.10732.15
50.14172.83
70.16633.33
100.19343.87
140.21974.39
200.24804.96
300.28025.60
600.33476.69
daily for 2 weeks : 0.2197
daily for 30 days : 0.2802
daily for 60 days : 0.3347
Show the full program
# Chapter 1. p11: exact false-positive rate of "look daily, stop if p<0.05"
# Model: an A/A test (no true difference); at every look a two-sided z
# test, stop as soon as |z| >= 1.96. Equal information arrives between
# looks, so S_k is a sum of k iid N(0,1) and z_k = S_k / sqrt(k).
# A grid recursion (the numerical-integration idea of Armitage, McPherson
# and Rowe 1969) gives the probability of at least one 'significant'
# look; grid width 0.05 sd, no Monte Carlo.
# Parameters: alpha 0.05 by convention; number of looks illustrative.
import math
from statistics import NormalDist

N01 = NormalDist()
H = 0.05          # grid width in standard deviations
WIDTH = 7.0       # kernel cut at 7 sd


def cross_curve(bounds, h=H):
    """Cumulative P(|S_k| >= b_k*sqrt(k) at some look <= k)."""
    cdf = N01.cdf
    m = int(WIDTH / h) + 1
    kern = [cdf((d + .5) * h) - cdf((d - .5) * h)
            for d in range(-m, m + 1)]
    mass, n0 = [1.0], 0
    out = []
    for k, b in enumerate(bounds, 1):
        lim = b * math.sqrt(k)
        n = int(lim / h + 0.5)
        new = []
        for i in range(-n, n + 1):
            lo = max((i - .5) * h, -lim)
            hi = min((i + .5) * h, lim)
            if hi <= lo:
                new.append(0.0)
                continue
            full = lo == (i - .5) * h and hi == (i + .5) * h
            acc = []
            for jj, w in enumerate(mass):
                d = i - (jj - n0)
                if w == 0.0 or d < -m or d > m:
                    continue
                if full:
                    acc.append(w * kern[d + m])
                else:
                    u = (jj - n0) * h
                    acc.append(w * (cdf(hi - u) - cdf(lo - u)))
            new.append(math.fsum(acc))
        mass, n0 = new, n
        out.append(1.0 - math.fsum(mass))
    return out


def main():
    z = N01.inv_cdf(0.975)
    looks = [1, 2, 3, 5, 7, 10, 14, 20, 30, 60]
    curve = cross_curve([z] * looks[-1])
    print("A/A test, two-sided z at every look, nominal alpha 0.05")
    print("looks  P(at least one 'significant')  x nominal")
    for k in looks:
        p = curve[k - 1]
        print("{:5d}  {:29.4f}  {:9.2f}".format(k, p, p / 0.05))
    print("daily for 2 weeks : {:.4f}".format(curve[13]))
    print("daily for 30 days : {:.4f}".format(curve[29]))
    print("daily for 60 days : {:.4f}".format(curve[59]))


main()

Single source and citation

Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 1: The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant". 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book