What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “The Ceiling of a Small Site: Power, Minimum Detectable Effects and a Five-and-a-Half-Year Wait”?
Answer: Chapter 2 shows that at a 3% base a 5% relative lift needs 207,938 visitors per arm, 5.7 years at 200 visitors a day, and that a one-month test has power of 0.0609.
Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 200, 207938, 5.7
| visitors per arm, alpha 0.05 two-sided, power 0.80 | ||||
| base | lift | n per arm | Lehr 16 | days at 200/day |
| 0.01 | 5% | 637,010 | 649,276 | 6,371 |
| 0.01 | 10% | 163,095 | 166,236 | 1,631 |
| 0.01 | 20% | 42,693 | 43,516 | 427 |
| 0.01 | 50% | 7,750 | 7,901 | 78 |
| 0.03 | 5% | 207,938 | 211,943 | 2,080 |
| 0.03 | 10% | 53,211 | 54,236 | 533 |
| 0.03 | 20% | 13,914 | 14,183 | 140 |
| 0.03 | 50% | 2,518 | 2,567 | 26 |
| 0.05 | 5% | 122,124 | 124,476 | 1,222 |
| 0.05 | 10% | 31,234 | 31,836 | 313 |
| 0.05 | 20% | 8,158 | 8,317 | 82 |
| 0.05 | 50% | 1,471 | 1,500 | 15 |
| z_0.975 + z_0.80 = 1.96 + 0.84 = 2.80; 2 x 2.80^2 = 15.7 | ||||
| 3% base, 5% lift: 5.7 years of 200 visitors a day |
Show the full program
# Chapter 2. p21: visitors per arm to detect a relative lift in conversion
# Model: two-sided two-proportion z test, alpha 0.05, power 0.80, normal
# approximation with the pooled variance under H0 and the unpooled one
# under H1 (the textbook formula, no continuity correction). Lehr's
# rule n = 16 * pbar * (1 - pbar) / delta^2 is printed for comparison.
# Baselines and lifts are illustrative; traffic is 200 visitors a day.
import math
from statistics import NormalDist
N01 = NormalDist()
def n_per_arm(p1, rel, alpha=0.05, power=0.80):
p2 = p1 * (1 + rel)
za = N01.inv_cdf(1 - alpha / 2)
zb = N01.inv_cdf(power)
pb = (p1 + p2) / 2
num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(
p1 * (1 - p1) + p2 * (1 - p2))
return math.ceil((num / (p2 - p1)) ** 2)
def lehr(p1, rel):
p2 = p1 * (1 + rel)
pb = (p1 + p2) / 2
return math.ceil(16 * pb * (1 - pb) / (p2 - p1) ** 2)
def main():
print("visitors per arm, alpha 0.05 two-sided, power 0.80")
print("base lift n per arm Lehr 16 days at 200/day")
for p1 in (0.01, 0.03, 0.05):
for rel in (0.05, 0.10, 0.20, 0.50):
n = n_per_arm(p1, rel)
days = math.ceil(2 * n / 200)
print("{:.2f} {:4.0%} {:11,d} {:9,d} {:15,d}".format(
p1, rel, n, lehr(p1, rel), days))
za, zb = N01.inv_cdf(0.975), N01.inv_cdf(0.80)
print("z_0.975 + z_0.80 = {:.2f} + {:.2f} = {:.2f};".format(
za, zb, za + zb), "2 x {:.2f}^2 = {:.1f}".format(
za + zb, 2 * (za + zb) ** 2))
n = n_per_arm(0.03, 0.05)
print("3% base, 5% lift: {:.1f} years of 200 visitors a day".format(
2 * n / 200 / 365))
main()Single source and citation
Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 2: The Ceiling of a Small Site: Power, Minimum Detectable Effects and a Five-and-a-Half-Year Wait. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant"
- Chapter 3 · Where the Inflation Comes From: Random Walks, the Iterated Logarithm and Exaggerated Winners
- Chapter 4 · Wald's Sequential Probability Ratio Test: a Look at Every Single Conversion
- Chapter 5 · Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions
- Chapter 6 · Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data
- Chapter 7 · Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea
- Chapter 8 · Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book