What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book”?
Answer: Chapter 8 offers a stopping-rule dashboard for thirty sites, the use of the A/B Stopping-Rule Calculator and a pre-registered plan for refuting the book. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.
Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 19
| site | vis/d base mde28 d@10% rule | same, next column | |||||
| S01 | 93 2.5% | 80.4% 1,368 NOAB | S16 | 1,094 3.3% | 17.9% | 87 SEQ | |
| S02 | 103 4.5% | 54.0% | 682 NOAB | S17 | 1,197 1.9% | 23.3% | 144 SEQ |
| S03 | 121 4.3% | 50.5% | 606 NOAB | S18 | 1,385 5.5% | 12.1% | 41 SEQ |
| S04 | 141 4.4% | 45.6% | 503 NOAB | S19 | 1,524 2.0% | 19.7% | 104 SEQ |
| S05 | 164 4.8% | 40.1% | 398 NOAB | S20 | 2,252 4.0% | 11.2% | 35 SEQ |
| S06 | 194 5.0% | 35.9% | 324 NOAB | S21 | 2,592 4.9% | 9.3% | 25 FIX |
| S07 | 296 3.5% | 34.8% | 305 NOAB | S22 | 3,832 2.6% | 10.7% | 32 SEQ |
| S08 | 333 2.1% | 43.6% | 462 NOAB | S23 | 4,540 3.8% | 8.1% | 19 FIX |
| S09 | 337 3.2% | 34.5% | 300 NOAB | S24 | 4,767 1.9% | 11.5% | 37 SEQ |
| S10 345 4.4% 28.3% 208 SEQ | S25 8,395 4.1% 5.6% 10 FIX | ||||||
| S11 | 377 4.0% | 28.4% | 209 SEQ | | S26 15,636 3.1% | 4.8% | 7 FIX |
| S12 | 502 1.3% | 46.1% | 510 NOAB | S27 16,934 4.4% | 3.8% | 5 FIX | |
| S13 | 648 3.1% | 24.7% | 161 SEQ | | S28 21,122 1.6% | 5.7% | 10 FIX |
| S14 | 667 5.2% | 18.2% | 90 SEQ | | S29 22,825 1.6% | 5.5% | 9 FIX |
| S15 | 861 4.7% | 16.9% | 78 SEQ | | S30 27,642 1.1% | 6.2% | 11 FIX |
| FIX 8 | SEQ 12 | NOAB 10 | ||||
Show the full program
# Chapter 8. p81: a stopping-rule dashboard for thirty sites
# Model: thirty synthetic sites (seeded) with daily visitors log-uniform
# between 80 and 40,000 and base conversion uniform in 0.01 to 0.06.
# For each: relative MDE in 28 days (p22), days a fixed test needs for a
# 10% lift (p21), and the rule the book recommends:
# FIX : 10% lift is visible within 28 days -> fixed horizon, at most
# 5 O'Brien-Fleming looks (p51)
# SEQ : a 20% lift is visible within 56 days -> test only bold
# changes, with an SPRT or mSPRT (p41, p61)
# NOAB : neither -> no A/B test; pool traffic or change the metric
# All sites and thresholds are illustrative.
import math
import random
from statistics import NormalDist
N01 = NormalDist()
def n_per_arm(p1, rel, alpha=0.05, power=0.80):
p2 = p1 * (1 + rel)
za = N01.inv_cdf(1 - alpha / 2)
zb = N01.inv_cdf(power)
pb = (p1 + p2) / 2
num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(
p1 * (1 - p1) + p2 * (1 - p2))
return (num / (p2 - p1)) ** 2
def mde(p1, n, lo=1e-4, hi=20.0):
for _ in range(80):
mid = (lo + hi) / 2
if n_per_arm(p1, mid) > n:
lo = mid
else:
hi = mid
return hi
def rule(t, p):
if 2 * n_per_arm(p, 0.10) / t <= 28:
return "FIX"
if 2 * n_per_arm(p, 0.20) / t <= 56:
return "SEQ"
return "NOAB"
def main():
rng = random.Random(20261112)
sites = []
for i in range(30):
t = int(math.exp(rng.uniform(math.log(80), math.log(40000))))
p = rng.uniform(0.01, 0.06)
sites.append((t, p))
sites.sort()
rows = []
for i, (t, p) in enumerate(sites, 1):
d10 = math.ceil(2 * n_per_arm(p, 0.10) / t)
m28 = mde(p, t / 2 * 28)
rows.append("S{:02d} {:6,d} {:4.1%} {:6.1%} {:5,d} {:4}".format(
i, t, p, m28, d10, rule(t, p)))
print("site vis/d base mde28 d@10% rule | same, next column")
for a, b in zip(rows[:15], rows[15:]):
print((a + " | " + b).rstrip())
tally = {}
for t, p in sites:
r = rule(t, p)
tally[r] = tally.get(r, 0) + 1
print("FIX {} SEQ {} NOAB {}".format(
tally.get("FIX", 0), tally.get("SEQ", 0), tally.get("NOAB", 0)))
main()Single source and citation
Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 8: Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant"
- Chapter 2 · The Ceiling of a Small Site: Power, Minimum Detectable Effects and a Five-and-a-Half-Year Wait
- Chapter 3 · Where the Inflation Comes From: Random Walks, the Iterated Logarithm and Exaggerated Winners
- Chapter 4 · Wald's Sequential Probability Ratio Test: a Look at Every Single Conversion
- Chapter 5 · Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions
- Chapter 6 · Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data
- Chapter 7 · Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea