What does Martingales, Stopping Times and Stop-Loss Bounds conclude, in numbers, about “Peeking and Early Stopping: Optional Stopping and Test Martingales in Online Experiments”?
Answer: Chapter 6 shows that peeking a hundred times in an online experiment raises false alarms to 0.3726, and presents test martingales and sequential tests.
Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs. It is not individual financial advice, does not encourage gambling of any kind, and analyses doubling only as a counter-example.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.3726
| A/A test, nominal two-sided level 0.05, 20000 tests | ||
| looks false alarm rate | x 0.05 | |
| 1 | 0.0498 | 1.00 |
| 2 | 0.0830 | 1.66 |
| 5 | 0.1409 | 2.82 |
| 10 | 0.1919 | 3.84 |
| 20 | 0.2469 | 4.94 |
| 50 | 0.3214 | 6.43 |
| 100 | 0.3726 | 7.45 |
| Monte Carlo standard error at rate 0.2: 0.0028 |
Key formula
Show the full program
# Chapter 6. p61: Peeking at an A/A test and stopping at the first
# "significant" look inflates false alarms. Model: two identical
# arms; the per-visitor difference is N(0, 1), so the running total
# S_n is a fair (Gaussian) walk. At each look the two-sided z test
# |S_n| / sqrt(n) > 1.96 is applied; the test "wins" at the first
# look that crosses. Looks are evenly spaced over 10000 visitors.
# Fixed-seed Monte Carlo, 20000 tests; the plan is illustrative.
import math
import random
rng = random.Random(20261113)
TOTAL, GRID, REPS = 10000, 100, 20000
STEP = TOTAL // GRID
LOOKS = (1, 2, 5, 10, 20, 50, 100)
first = {k: 0 for k in LOOKS}
for _ in range(REPS):
s = 0.0
path = []
for i in range(GRID):
s += rng.gauss(0.0, math.sqrt(STEP))
path.append(s)
for k in LOOKS:
every = GRID // k
for j in range(every - 1, GRID, every):
n = (j + 1) * STEP
if abs(path[j]) / math.sqrt(n) > 1.96:
first[k] += 1
break
print("A/A test, nominal two-sided level 0.05, %d tests" % REPS)
print("%6s %16s %8s" % ("looks", "false alarm rate", "x 0.05"))
for k in LOOKS:
r = first[k] / REPS
print("%6d %16.4f %8.2f" % (k, r, r / 0.05))
print("Monte Carlo standard error at rate 0.2: %.4f" % math.sqrt(
0.2 * 0.8 / REPS))Single source and citation
Fan, Y. (2026). Doob's Martingales and the Optional Stopping Theorem: the Doubling Fallacy and Rigorous Stop-Loss Bounds for Business Trials (Version 1.0.0), Chapter 6: Peeking and Early Stopping: Optional Stopping and Test Martingales in Online Experiments. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-249 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026249,
author = {Fan, Yafu},
title = {Doob's Martingales and the Optional Stopping Theorem: the Doubling Fallacy and Rigorous Stop-Loss Bounds for Business Trials},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · A Conservation Law for Fair Games: Martingales, Bounded Stopping Times and the Optional Stopping Theorem
- Chapter 2 · The Exact Accounts of the Doubling Ladder: Finite Capital, Tail Losses and Doubling After a Loss
- Chapter 3 · Wald's Identities: a Stop Line Sells Time
- Chapter 4 · Quitting While Ahead Changes Only the Shape: the Win-Rate Illusion and Selling Winners While Holding Losers
- Chapter 5 · The Learning Value of a Stop Line: Lines of Belief, the Gap Between Good and Bad, and Escalating Sunk Costs
- Chapter 7 · Eastern and Western Sources Examined: Laozi on Knowing Enough and Knowing When to Stop, and Bernoulli's Logarithm
- Chapter 8 · Closing the Loop: the Three-Line Register of a Trial Budget, the Stop-Loss Bound Calculator and a Pre-Registered Protocol