What does Attribution, Bandits and Customer Lifetime Value conclude, in numbers, about “The Price of Peeking: Optional Stopping, Group-Sequential Designs and Always-Valid Inference”?
Answer: With equally spaced looks and the unadjusted 1.96 at each, a nominal 5 % test has an actual type I error of 14.2 % with 5 looks, 24.8 % with 20 and 37.4 % with 100; "soaring to 25 % or more" holds only from about 20 looks. The problem is not looking but failing to fix a stopping rule in advance. Group-sequential designs adjust the critical values for a fixed number of looks (with 5 looks, Pocock 2.413, O'Brien–Fleming from 4.562 down to 2.040) and save a quarter to a third of the sample on average; always-valid inference allows looking and stopping at any time, at the cost of somewhat larger samples for small effects.
Scope of the numbers: Which advertising channel should a sale be credited to? When may an A/B test stop? What is a customer still worth? This book presents mathematical models for teaching and research and is not marketing, investment or legal advice. Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's models and hypotheses; all data in the book are fictional or simulated.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 2.040, 2.413, 4.562
A. K = 5 looks, two-sided alpha 0.05; effect with 90 % power for one test
Pocock z-bounds 2.413 2.413 2.413 2.413 2.413
power 0.835; expected looks 3.13 of 5 (null: 4.88)
max sample to keep 90 % power: 1.206 x fixed design
O'Brien-Fleming z-bounds 4.562 3.226 2.634 2.281 2.040
power 0.892; expected looks 3.69 of 5 (null: 4.96)
max sample to keep 90 % power: 1.026 x fixed design
fixed sample, one look at the end: power 0.900, looks 5 of 5
B. mixture SPRT, checked after every pair, alpha 0.05, tau^2 = 1
A/A, 5,000 pairs each, 400 runs: rejected 0.022 (any time; bound 0.05)
effect 0.3 sd: fixed design 234 pairs; mSPRT stopped in 200/200, median 194
effect 0.1 sd: fixed design 2,102 pairs; mSPRT stopped in 200/200, median 2,193Key formula
Show the full program
# Ways to look at a running test without inflating the error rate.
# A: group-sequential boundaries for K = 5 equally spaced looks, two-sided alpha 0.05:
# Pocock (one constant z) and O'Brien-Fleming (z_k = c sqrt(K / k)), constants found
# by bisection on the exact crossing probability of p04; power at the drift that
# gives a fixed-sample test 90 % power, and the expected number of looks used.
# B: mixture SPRT (normal mixture, Johari et al.) monitored after every observation:
# always-valid type I error, and how long it takes to stop under an effect.
import math, random
# --- helpers repeated from earlier programs, so that this one runs on its own ---
def stay_probs(bounds, h=0.1, drift=0.0):
"""P(|S_j| < bounds[j-1] for all j <= k), k = 1..K, S a Gaussian random walk with
N(drift, 1) steps: Simpson's rule on a grid spanning each continuation interval."""
phi = lambda x: math.exp(-x * x / 2) / math.sqrt(2 * math.pi)
def grid(b):
n = 2 * max(1, math.ceil(b / h)); d = 2 * b / n
w = [(1 if i in (0, n) else 4 if i % 2 else 2) * d / 3 for i in range(n + 1)]
return [-b + i * d for i in range(n + 1)], w
xs, w = grid(bounds[0]); f = [phi(x - drift) for x in xs]
out = [sum(a * b for a, b in zip(w, f))]
for b in bounds[1:]:
ys, v = grid(b)
g = [sum(wi * fi * phi(y - x - drift) for x, wi, fi in zip(xs, w, f)
if abs(y - x - drift) < 8.5)
for y in ys]
xs, w, f = ys, v, g
out.append(sum(a * b for a, b in zip(w, f)))
return out
# --- end of repeated helpers ---
def calibrate(shape, K=5, alpha=0.05):
lo, hi = 1.5, 5.0
for _ in range(40):
c = (lo + hi) / 2
b = [c * shape(k) * math.sqrt(k) for k in range(1, K + 1)]
if 1 - stay_probs(b)[-1] > alpha:
lo = c
else:
hi = c
return (lo + hi) / 2
def operating(c, shape, drift, K=5):
b = [c * shape(k) * math.sqrt(k) for k in range(1, K + 1)]
st = stay_probs(b, drift=drift) # ignores the (tiny) wrong-side crossing
return 1 - st[-1], 1 + sum(st[:-1])
def pw_drift(c, shape, K, target=0.90):
lo, hi = 0.5, 3.0 # drift per look giving target power
for _ in range(30):
d = (lo + hi) / 2
if operating(c, shape, d, K)[0] < target:
lo = d
else:
hi = d
return (lo + hi) / 2 * math.sqrt(K)
def msprt_stop(rng, delta, tau2=1.0, alpha=0.05, nmax=20_000):
"""Paired differences x_i ~ N(delta, 2); stop when the mixture likelihood ratio
exceeds 1/alpha. Returns the stopping time or None."""
s2, s = 2.0, 0.0
for n in range(1, nmax + 1):
s += rng.gauss(delta, math.sqrt(s2))
lr = math.sqrt(s2 / (s2 + n * tau2)) * math.exp(
tau2 * s * s / (2 * s2 * (s2 + n * tau2)))
if lr >= 1 / alpha:
return n
return None
if __name__ == "__main__":
K = 5; z = 1.959964; zb = 1.281552
drift = (z + zb) / math.sqrt(K) # per-look drift: 90 % power at look K
print(f"A. K = {K} looks, two-sided alpha 0.05; effect with 90 % power for one test")
for name, shape in (("Pocock", lambda k: 1.0),
("O'Brien-Fleming", lambda k: math.sqrt(K / k))):
c = calibrate(shape, K)
zs = [c * shape(k) for k in range(1, K + 1)]
pw, el = operating(c, shape, drift, K)
print(f" {name:16s} z-bounds " + " ".join(f"{x:.3f}" for x in zs))
print(f" {'':16s} power {pw:.3f}; expected looks {el:.2f} of {K}"
f" (null: {operating(c, shape, 0.0, K)[1]:.2f})")
n_inf = (pw_drift(c, shape, K) / (z + zb)) ** 2
print(f" {'':16s} max sample to keep 90 % power: {n_inf:.3f} x fixed design")
print(" fixed sample, one look at the end: power 0.900, looks 5 of 5")
print("B. mixture SPRT, checked after every pair, alpha 0.05, tau^2 = 1")
rng = random.Random(2017); n0 = 400
fp = sum(msprt_stop(rng, 0.0, nmax=5000) is not None for _ in range(n0))
print(f" A/A, 5,000 pairs each, {n0} runs: rejected {fp / n0:.3f}"
f" (any time; bound 0.05)")
for delta in (0.3, 0.1):
n_fix = math.ceil((z + zb) ** 2 * 2 / delta ** 2)
t = [msprt_stop(rng, delta) for _ in range(200)]
ok = sorted(x for x in t if x is not None)
print(f" effect {delta} sd: fixed design {n_fix:,} pairs; mSPRT stopped in"
f" {len(ok)}/200, median {ok[len(ok) // 2]:,}")Single source and citation
Fan, Y. (2026). Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels (Version 1.0.0), Chapter 3: The Price of Peeking: Optional Stopping, Group-Sequential Designs and Always-Valid Inference. 5D Library. https://doi.org/10.5281/zenodo.23836264
DOI: 10.5281/zenodo.23836264 · Read the monograph · 5DL-B-048 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026048,
author = {Fan, Yafu},
title = {Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836264},
url = {https://doi.org/10.5281/zenodo.23836264}
}Other chapter answers in this book
- Chapter 1 · Attribution Is Not Incrementality: Last Click, Potential Outcomes and Randomised Holdouts
- Chapter 2 · Shapley Values and Markov Removal Effects: Exact Allocations of an Inexact Question
- Chapter 4 · Learning While Earning: Thompson Sampling and the Bias of Adaptive Experiments
- Chapter 5 · What Is a Customer Still Worth? BG/NBD, Gamma-Gamma and the Illusion of Rising Retention
- Chapter 6 · Diffusion and Spend: the Bass Model, the Dorfman–Steiner Condition and Whole-Life Budgets
- Chapter 7 · Reading the Classics: *Shi* and *Jie* in the *Sunzi*
- Chapter 8 · Practice and a Pre-Registered Test