What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book”?

Answer: Chapter 8 offers a stopping-rule dashboard for thirty sites, the use of the A/B Stopping-Rule Calculator and a pre-registered plan for refuting the book. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 19

sitevis/d base mde28 d@10% rule | same, next column
S0193 2.5%80.4% 1,368 NOAB | S161,094 3.3%17.9%87 SEQ
S02103 4.5%54.0%682 NOAB | S171,197 1.9%23.3%144 SEQ
S03121 4.3%50.5%606 NOAB | S181,385 5.5%12.1%41 SEQ
S04141 4.4%45.6%503 NOAB | S191,524 2.0%19.7%104 SEQ
S05164 4.8%40.1%398 NOAB | S202,252 4.0%11.2%35 SEQ
S06194 5.0%35.9%324 NOAB | S212,592 4.9%9.3%25 FIX
S07296 3.5%34.8%305 NOAB | S223,832 2.6%10.7%32 SEQ
S08333 2.1%43.6%462 NOAB | S234,540 3.8%8.1%19 FIX
S09337 3.2%34.5%300 NOAB | S244,767 1.9%11.5%37 SEQ
S10 345 4.4% 28.3% 208 SEQ | S25 8,395 4.1% 5.6% 10 FIX
S11377 4.0%28.4%209 SEQ| S26 15,636 3.1%4.8%7 FIX
S12502 1.3%46.1%510 NOAB | S27 16,934 4.4%3.8%5 FIX
S13648 3.1%24.7%161 SEQ| S28 21,122 1.6%5.7%10 FIX
S14667 5.2%18.2%90 SEQ| S29 22,825 1.6%5.5%9 FIX
S15861 4.7%16.9%78 SEQ| S30 27,642 1.1%6.2%11 FIX
FIX 8SEQ 12NOAB 10
Show the full program
# Chapter 8. p81: a stopping-rule dashboard for thirty sites
# Model: thirty synthetic sites (seeded) with daily visitors log-uniform
# between 80 and 40,000 and base conversion uniform in 0.01 to 0.06.
# For each: relative MDE in 28 days (p22), days a fixed test needs for a
# 10% lift (p21), and the rule the book recommends:
#   FIX  : 10% lift is visible within 28 days -> fixed horizon, at most
#          5 O'Brien-Fleming looks (p51)
#   SEQ  : a 20% lift is visible within 56 days -> test only bold
#          changes, with an SPRT or mSPRT (p41, p61)
#   NOAB : neither -> no A/B test; pool traffic or change the metric
# All sites and thresholds are illustrative.
import math
import random
from statistics import NormalDist

N01 = NormalDist()


def n_per_arm(p1, rel, alpha=0.05, power=0.80):
    p2 = p1 * (1 + rel)
    za = N01.inv_cdf(1 - alpha / 2)
    zb = N01.inv_cdf(power)
    pb = (p1 + p2) / 2
    num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(
        p1 * (1 - p1) + p2 * (1 - p2))
    return (num / (p2 - p1)) ** 2


def mde(p1, n, lo=1e-4, hi=20.0):
    for _ in range(80):
        mid = (lo + hi) / 2
        if n_per_arm(p1, mid) > n:
            lo = mid
        else:
            hi = mid
    return hi


def rule(t, p):
    if 2 * n_per_arm(p, 0.10) / t <= 28:
        return "FIX"
    if 2 * n_per_arm(p, 0.20) / t <= 56:
        return "SEQ"
    return "NOAB"


def main():
    rng = random.Random(20261112)
    sites = []
    for i in range(30):
        t = int(math.exp(rng.uniform(math.log(80), math.log(40000))))
        p = rng.uniform(0.01, 0.06)
        sites.append((t, p))
    sites.sort()
    rows = []
    for i, (t, p) in enumerate(sites, 1):
        d10 = math.ceil(2 * n_per_arm(p, 0.10) / t)
        m28 = mde(p, t / 2 * 28)
        rows.append("S{:02d} {:6,d} {:4.1%} {:6.1%} {:5,d} {:4}".format(
            i, t, p, m28, d10, rule(t, p)))
    print("site  vis/d base mde28 d@10% rule | same, next column")
    for a, b in zip(rows[:15], rows[15:]):
        print((a + " | " + b).rstrip())
    tally = {}
    for t, p in sites:
        r = rule(t, p)
        tally[r] = tally.get(r, 0) + 1
    print("FIX {}  SEQ {}  NOAB {}".format(
        tally.get("FIX", 0), tally.get("SEQ", 0), tally.get("NOAB", 0)))


main()

Single source and citation

Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 8: Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book