What does Martingales, Stopping Times and Stop-Loss Bounds conclude, in numbers, about “Peeking and Early Stopping: Optional Stopping and Test Martingales in Online Experiments”?

Answer: Chapter 6 shows that peeking a hundred times in an online experiment raises false alarms to 0.3726, and presents test martingales and sequential tests.

Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs. It is not individual financial advice, does not encourage gambling of any kind, and analyses doubling only as a counter-example.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.3726

A/A test, nominal two-sided level 0.05, 20000 tests
looks false alarm ratex 0.05
10.04981.00
20.08301.66
50.14092.82
100.19193.84
200.24694.94
500.32146.43
1000.37267.45
Monte Carlo standard error at rate 0.2: 0.0028

Key formula

\[L_n=\sqrt{\frac{1}{1+n\tau^2}}\exp\!\left(\frac{\tau^2S_n^2}{2(1+n\tau^2)}\right).\]
Show the full program
# Chapter 6. p61: Peeking at an A/A test and stopping at the first
# "significant" look inflates false alarms. Model: two identical
# arms; the per-visitor difference is N(0, 1), so the running total
# S_n is a fair (Gaussian) walk. At each look the two-sided z test
# |S_n| / sqrt(n) > 1.96 is applied; the test "wins" at the first
# look that crosses. Looks are evenly spaced over 10000 visitors.
# Fixed-seed Monte Carlo, 20000 tests; the plan is illustrative.
import math
import random

rng = random.Random(20261113)
TOTAL, GRID, REPS = 10000, 100, 20000
STEP = TOTAL // GRID
LOOKS = (1, 2, 5, 10, 20, 50, 100)
first = {k: 0 for k in LOOKS}
for _ in range(REPS):
    s = 0.0
    path = []
    for i in range(GRID):
        s += rng.gauss(0.0, math.sqrt(STEP))
        path.append(s)
    for k in LOOKS:
        every = GRID // k
        for j in range(every - 1, GRID, every):
            n = (j + 1) * STEP
            if abs(path[j]) / math.sqrt(n) > 1.96:
                first[k] += 1
                break
print("A/A test, nominal two-sided level 0.05, %d tests" % REPS)
print("%6s %16s %8s" % ("looks", "false alarm rate", "x 0.05"))
for k in LOOKS:
    r = first[k] / REPS
    print("%6d %16.4f %8.2f" % (k, r, r / 0.05))
print("Monte Carlo standard error at rate 0.2: %.4f" % math.sqrt(
    0.2 * 0.8 / REPS))

Single source and citation

Fan, Y. (2026). Doob's Martingales and the Optional Stopping Theorem: the Doubling Fallacy and Rigorous Stop-Loss Bounds for Business Trials (Version 1.0.0), Chapter 6: Peeking and Early Stopping: Optional Stopping and Test Martingales in Online Experiments. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-249 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026249,
  author    = {Fan, Yafu},
  title     = {Doob's Martingales and the Optional Stopping Theorem: the Doubling Fallacy and Rigorous Stop-Loss Bounds for Business Trials},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book