What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions”?

Answer: Chapter 5 gives group-sequential bars: with five looks, 2.4133 for Pocock and a final 2.0402 for O'Brien-Fleming.

Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 2.0402, 2.4133

two-sided alpha 0.05 with K equally spaced looks
KPocock cp per lookOBF Cfirst-look plast-look p
11.96000.05001.96005.00e-020.0500
22.17830.02941.97755.16e-030.0480
32.28960.02202.00425.18e-040.0451
42.36140.01822.02455.15e-050.0429
52.41330.01582.04025.06e-060.0413
102.55520.01062.08674.15e-110.0369
Show the full program
# Chapter 5. p51: Pocock and O'Brien-Fleming bars that keep alpha at 0.05
# Model: K equally spaced looks at accumulating normal data, two-sided.
# Pocock (1977): the same bar c at every look. O'Brien and Fleming
# (1979): bar C * sqrt(K / k) at look k, very strict early. Both
# constants are found by bisection so that the exact crossing
# probability under H0 (p11's grid recursion) equals 0.05.
import math
from statistics import NormalDist

N01 = NormalDist()
H = 0.05          # grid width in standard deviations
WIDTH = 7.0       # kernel cut at 7 sd


def cross_curve(bounds, h=H):
    """Cumulative P(|S_k| >= b_k*sqrt(k) at some look <= k)."""
    cdf = N01.cdf
    m = int(WIDTH / h) + 1
    kern = [cdf((d + .5) * h) - cdf((d - .5) * h)
            for d in range(-m, m + 1)]
    mass, n0 = [1.0], 0
    out = []
    for k, b in enumerate(bounds, 1):
        lim = b * math.sqrt(k)
        n = int(lim / h + 0.5)
        new = []
        for i in range(-n, n + 1):
            lo = max((i - .5) * h, -lim)
            hi = min((i + .5) * h, lim)
            if hi <= lo:
                new.append(0.0)
                continue
            full = lo == (i - .5) * h and hi == (i + .5) * h
            acc = []
            for jj, w in enumerate(mass):
                d = i - (jj - n0)
                if w == 0.0 or d < -m or d > m:
                    continue
                if full:
                    acc.append(w * kern[d + m])
                else:
                    u = (jj - n0) * h
                    acc.append(w * (cdf(hi - u) - cdf(lo - u)))
            new.append(math.fsum(acc))
        mass, n0 = new, n
        out.append(1.0 - math.fsum(mass))
    return out


def bisect(f, lo, hi, target=0.05, it=28):
    for _ in range(it):
        mid = (lo + hi) / 2
        if f(mid) > target:
            lo = mid
        else:
            hi = mid
    return (lo + hi) / 2


def main():
    print("two-sided alpha 0.05 with K equally spaced looks")
    print(" K   Pocock c  p per look   OBF C  first-look p  last-look p")
    for k in (1, 2, 3, 4, 5, 10):
        c = bisect(lambda x: cross_curve([x] * k)[-1], 1.5, 3.5)
        obf = bisect(lambda x: cross_curve(
            [x * math.sqrt(k / j) for j in range(1, k + 1)])[-1],
            1.5, 3.5)
        pp = 2 * (1 - N01.cdf(c))
        p1 = 2 * (1 - N01.cdf(obf * math.sqrt(k)))
        pk = 2 * (1 - N01.cdf(obf))
        print("{:2d}   {:8.4f}  {:10.4f}  {:6.4f}  {:12.2e}  {:11.4f}".format(
            k, c, pp, obf, p1, pk))


main()

Single source and citation

Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 5: Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book