What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Where the Inflation Comes From: Random Walks, the Iterated Logarithm and Exaggerated Winners”?
Answer: Chapter 3 explains the inflation with random walks and the law of the iterated logarithm and finds that a peeking small site reports on average 18.3 times the true lift.
Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 18.3
| true lift 5% on base 0.03, 28 days, significant results only | ||||
| site/day | rule | power | exaggeration | wrong sign |
| 200 | plan | 0.0611 | 7.3 | 0.165 |
| 200 | peek | 0.2855 | 18.3 | 0.339 |
| 2,000 | plan | 0.1797 | 2.4 | 0.009 |
| 2,000 | peek | 0.4152 | 5.3 | 0.117 |
| 20,000 | plan | 0.9012 | 1.1 | 0.000 |
| 20,000 | peek | 0.9479 | 1.7 | 0.009 |
Show the full program
# Chapter 3. p32: a winner found by peeking is a winner exaggerated
# Model: true relative lift 5% on a base of 0.03 (illustrative). Each
# day's difference in conversion rate between arms is drawn as normal
# with mean delta and variance 2pq/n (n visitors per arm per day). The
# peeker stops at the first day with |z| >= 1.96 within 28 days; the
# planner tests once on day 28. Among significant results we report the
# exaggeration ratio (mean estimated lift / true lift, Gelman and
# Carlin's type M) and the share with the wrong sign (type S).
# Seeded Monte Carlo.
import math
import random
BASE = 0.03
LIFT = 0.05
DAYS = 28
SIMS = 20000
Z = 1.959964
def run(per_arm, rng):
delta = BASE * LIFT
q = BASE * (1 + LIFT / 2)
sd = math.sqrt(2 * q * (1 - q) / per_arm)
res = {"peek": [], "plan": []}
for _ in range(SIMS):
tot = 0.0
done = False
for d in range(1, DAYS + 1):
tot += rng.gauss(delta, sd)
est = tot / d
z = est / (sd / math.sqrt(d))
if not done and abs(z) >= Z:
res["peek"].append(est)
done = True
if abs(z) >= Z:
res["plan"].append(est)
return res, delta
def main():
rng = random.Random(20261112)
print("true lift 5% on base 0.03, 28 days, significant results only")
print("site/day rule power exaggeration wrong sign")
for per_arm in (100, 1000, 10000):
res, delta = run(per_arm, rng)
for key in ("plan", "peek"):
ests = res[key]
pw = len(ests) / SIMS
ex = math.fsum(abs(e) for e in ests) / len(ests) / delta
ws = len([e for e in ests if e < 0]) / len(ests)
print("{:8,d} {:5} {:6.4f} {:12.1f} {:11.3f}".format(
2 * per_arm, key, pw, ex, ws))
main()Single source and citation
Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 3: Where the Inflation Comes From: Random Walks, the Iterated Logarithm and Exaggerated Winners. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant"
- Chapter 2 · The Ceiling of a Small Site: Power, Minimum Detectable Effects and a Five-and-a-Half-Year Wait
- Chapter 4 · Wald's Sequential Probability Ratio Test: a Look at Every Single Conversion
- Chapter 5 · Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions
- Chapter 6 · Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data
- Chapter 7 · Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea
- Chapter 8 · Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book