What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data”?

Answer: Chapter 6 shows that a mixture sequential test watched daily for a year keeps its false-alarm rate at 0.0280; that a Bayesian "chance to beat control" under a default prior is true only 0.2158 of the time in a world where most changes do nothing; and that pre-experiment data would need a correlation of 0.9932 for a small site to see a 5% lift in a month.

Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0280

daily looks for up to 365 days, nominal alpha 0.05
ruletruthby day 28by day 90by day 365
naiveH00.27820.36680.4758
naiveH10.89250.99921.0000
mSPRTH00.01070.02030.0280
mSPRTH10.50950.98581.0000
mSPRT under H1: median stop day 28 (fixed test: 28)
Show the full program
# Chapter 6. p61: a mixture SPRT can be watched every day without cost
# Model: standardized daily increments x_i ~ N(theta, 1); S_n = sum x_i.
# Mixing the likelihood ratio over theta ~ N(0, tau^2) gives
#   Lambda_n = (1 + n tau^2)^(-1/2) exp(tau^2 S_n^2 / (2 (1 + n tau^2)))
# (the mSPRT of Robbins 1970, used by Johari et al. 2022). Rejecting
# when Lambda_n >= 1 / alpha keeps P(ever reject | H0) <= alpha for all
# n by Ville's inequality. The naive rule rejects when |S_n|/sqrt(n) >=
# 1.96. Under H1 theta = 2.8 / sqrt(28), i.e. a fixed 28-day test would
# have power 0.80. tau = 0.5 and the 365-day horizon are illustrative.
# Seeded Monte Carlo.
import math
import random

ALPHA, TAU2, DAYS, SIMS = 0.05, 0.25, 365, 4000
Z = 1.959964


def run(theta, rng):
    lim = math.log(1 / ALPHA)
    m_stop, n_stop = [], []
    for _ in range(SIMS):
        s = 0.0
        m = n = 0
        for d in range(1, DAYS + 1):
            s += rng.gauss(theta, 1.0)
            v = 1 + d * TAU2
            if not m and TAU2 * s * s / (2 * v) - 0.5 * math.log(v) >= lim:
                m = d
            if not n and s * s >= Z * Z * d:
                n = d
            if m and n:
                break
        m_stop.append(m)
        n_stop.append(n)
    return m_stop, n_stop


def share(stops, h):
    return len([d for d in stops if 0 < d <= h]) / SIMS


def main():
    rng = random.Random(20261112)
    m0, n0 = run(0.0, rng)
    th = 2.8 / math.sqrt(28)
    m1, n1 = run(th, rng)
    print("daily looks for up to 365 days, nominal alpha 0.05")
    print("rule    truth   by day 28   by day 90   by day 365")
    for name, st0, st1 in (("naive", n0, n1), ("mSPRT", m0, m1)):
        for truth, st in (("H0", st0), ("H1", st1)):
            print("{:6}  {:5}   {:9.4f}   {:9.4f}   {:10.4f}".format(
                name, truth, share(st, 28), share(st, 90),
                share(st, 365)))
    hit = sorted(d for d in m1 if d > 0)
    print("mSPRT under H1: median stop day {} (fixed test: 28)".format(
        hit[len(hit) // 2]))


main()

Single source and citation

Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 6: Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book