5D Library › Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels · 中文
Chapter 3 The Price of Peeking: Optional Stopping, Group-Sequential Designs and Always-Valid Inference
A look every morning
Once an A/B test goes live, its results update every day. A common practice is to check the p-value each morning, declare a winner and stop the moment it falls below 0.05, and otherwise wait until the planned end date. This is optional stopping: when to stop depends on the data themselves. The trouble is that a fixed-sample p-value guarantees a 5 % type I error only for the one planned test; each additional look is another chance for random fluctuation to cross the threshold, and once it does the test stops, so any later fall back is never seen.
Book 33 has already shown, with the researcher-degrees-of-freedom simulation of Simmons and co-authors, how optional stopping contributes to false positives, and that is not repeated here. This chapter does three things: computes exactly what the type I error is after looks, to check the framework’s “soaring to 25 %+”; introduces two families of methods that allow interim looks while controlling the error rate; and explains what each costs.
The exact computation
Under the null hypothesis (two identical variants), treat the accumulating standardised difference as a random walk: at the -th look the partial sum is , with independent increments between looks. Equally spaced looks with the criterion at each amount to stopping when . The type I error is the probability that this walk leaves the boundary within steps. Armitage, McPherson and Rowe studied this problem of “repeated significance tests on accumulating data” in 1969 (Armitage et al. 1969). The computation is recursive: the sub-density of “not yet crossed” at step is discretised on , convolved with the normal density of the next increment, and truncated to the next interval.
Calculation
# Optional stopping: an A/A test is checked K times at equally spaced looks, and the
# experimenter stops at the first look with |z| >= 1.96 (nominal two-sided 5 %).
# Exact probability of a false positive by recursive numerical integration of the
# sub-density of the partial sum S_k ~ N(0, k) inside the continuation region
# (Armitage, McPherson & Rowe 1969), checked by a Monte Carlo of Bernoulli A/A tests.
import math, random
def stay_probs(bounds, h=0.1, drift=0.0):
"""P(|S_j| < bounds[j-1] for all j <= k), k = 1..K, S a Gaussian random walk with
N(drift, 1) steps: Simpson's rule on a grid spanning each continuation interval."""
phi = lambda x: math.exp(-x * x / 2) / math.sqrt(2 * math.pi)
def grid(b):
n = 2 * max(1, math.ceil(b / h)); d = 2 * b / n
w = [(1 if i in (0, n) else 4 if i % 2 else 2) * d / 3 for i in range(n + 1)]
return [-b + i * d for i in range(n + 1)], w
xs, w = grid(bounds[0]); f = [phi(x - drift) for x in xs]
out = [sum(a * b for a, b in zip(w, f))]
for b in bounds[1:]:
ys, v = grid(b)
g = [sum(wi * fi * phi(y - x - drift) for x, wi, fi in zip(xs, w, f)
if abs(y - x - drift) < 8.5)
for y in ys]
xs, w, f = ys, v, g
out.append(sum(a * b for a, b in zip(w, f)))
return out
if __name__ == "__main__":
z = 1.959964
print("looks K false-positive rate (exact) 1 - 0.95^K (independent looks)")
stay = stay_probs([z * math.sqrt(k) for k in range(1, 101)])
rates = {K: 1 - stay[K - 1] for K in (1, 2, 3, 5, 10, 20, 50, 100)}
for K in rates:
print(f" {K:4d} {rates[K]:6.3f} "
f"{1 - 0.95 ** K:6.3f}")
print("Monte Carlo check: A/A test, conversion 5 %, 400 users per arm per look,")
print(" two-proportion z-test at every look, 2,000 simulated experiments")
rng = random.Random(1969)
for K in (5, 20):
hits = 0
for _ in range(2000):
a = b = n = 0
for _ in range(K):
a += sum(rng.random() < 0.05 for _ in range(400))
b += sum(rng.random() < 0.05 for _ in range(400)); n += 400
p = (a + b) / (2 * n); se = math.sqrt(2 * p * (1 - p) / n)
if se > 0 and abs(a - b) / n / se >= z:
hits += 1; break
print(f" K = {K:2d}: {hits / 2000:.3f} (exact Gaussian value {rates[K]:.3f})")
# looks K false-positive rate (exact) 1 - 0.95^K (independent looks)
# 1 0.050 0.050
# 2 0.083 0.098
# 3 0.107 0.143
# 5 0.142 0.226
# 10 0.193 0.401
# 20 0.248 0.642
# 50 0.320 0.923
# 100 0.374 0.994
# Monte Carlo check: A/A test, conversion 5 %, 400 users per arm per look,
# two-proportion z-test at every look, 2,000 simulated experiments
# K = 5: 0.149 (exact Gaussian value 0.142)
# K = 20: 0.237 (exact Gaussian value 0.248)With one look the rate is 5.0 %; with 2 looks 8.3 %, 3 looks 10.7 %, 5 looks 14.2 %, 10 looks 19.3 %, 20 looks 24.8 %, 50 looks 32.0 % and 100 looks 37.4 %. These rates tend to 1 as the number of looks grows without bound (by the law of the iterated logarithm the walk eventually crosses ), but slowly. They are far below , the value if each look were an independent test (64.2 % for 20 looks), because successive looks share most of their data and the test statistics are highly correlated. A Monte Carlo check with binomial conversion data (5 % conversion, 400 new users per arm per look) gives 14.9 % and 23.7 %, within simulation error and discreteness of the exact Gaussian values. So “soaring to 25 % or more” in the author’s framework should be qualified: it holds with about 20 or more equally spaced looks, each using the unadjusted 1.96; a test checked daily for three weeks is roughly of that order. The left panel of Figure 2 shows the full curve.
Group-sequential designs: deciding in advance how often and how to look
If it is known in advance that there will be looks, the critical value at each look can be raised so that the overall type I error stays at 5 %. Pocock (1977) proposed the same critical value at every look (Pocock 1977); O’Brien and Fleming (1979) proposed boundaries that are very high early and fall later, of the form (O’Brien and Fleming 1979). The program uses the exact computation above to find the constants for , and computes, for an effect that gives a fixed-sample test with a single final analysis 90 % power, each design’s power, the expected number of looks, and how much the maximum sample must be inflated to keep 90 % power.
Calculation
# Ways to look at a running test without inflating the error rate.
# A: group-sequential boundaries for K = 5 equally spaced looks, two-sided alpha 0.05:
# Pocock (one constant z) and O'Brien-Fleming (z_k = c sqrt(K / k)), constants found
# by bisection on the exact crossing probability of p04; power at the drift that
# gives a fixed-sample test 90 % power, and the expected number of looks used.
# B: mixture SPRT (normal mixture, Johari et al.) monitored after every observation:
# always-valid type I error, and how long it takes to stop under an effect.
import math, random
# --- helpers repeated from earlier programs, so that this one runs on its own ---
def stay_probs(bounds, h=0.1, drift=0.0):
"""P(|S_j| < bounds[j-1] for all j <= k), k = 1..K, S a Gaussian random walk with
N(drift, 1) steps: Simpson's rule on a grid spanning each continuation interval."""
phi = lambda x: math.exp(-x * x / 2) / math.sqrt(2 * math.pi)
def grid(b):
n = 2 * max(1, math.ceil(b / h)); d = 2 * b / n
w = [(1 if i in (0, n) else 4 if i % 2 else 2) * d / 3 for i in range(n + 1)]
return [-b + i * d for i in range(n + 1)], w
xs, w = grid(bounds[0]); f = [phi(x - drift) for x in xs]
out = [sum(a * b for a, b in zip(w, f))]
for b in bounds[1:]:
ys, v = grid(b)
g = [sum(wi * fi * phi(y - x - drift) for x, wi, fi in zip(xs, w, f)
if abs(y - x - drift) < 8.5)
for y in ys]
xs, w, f = ys, v, g
out.append(sum(a * b for a, b in zip(w, f)))
return out
# --- end of repeated helpers ---
def calibrate(shape, K=5, alpha=0.05):
lo, hi = 1.5, 5.0
for _ in range(40):
c = (lo + hi) / 2
b = [c * shape(k) * math.sqrt(k) for k in range(1, K + 1)]
if 1 - stay_probs(b)[-1] > alpha:
lo = c
else:
hi = c
return (lo + hi) / 2
def operating(c, shape, drift, K=5):
b = [c * shape(k) * math.sqrt(k) for k in range(1, K + 1)]
st = stay_probs(b, drift=drift) # ignores the (tiny) wrong-side crossing
return 1 - st[-1], 1 + sum(st[:-1])
def pw_drift(c, shape, K, target=0.90):
lo, hi = 0.5, 3.0 # drift per look giving target power
for _ in range(30):
d = (lo + hi) / 2
if operating(c, shape, d, K)[0] < target:
lo = d
else:
hi = d
return (lo + hi) / 2 * math.sqrt(K)
def msprt_stop(rng, delta, tau2=1.0, alpha=0.05, nmax=20_000):
"""Paired differences x_i ~ N(delta, 2); stop when the mixture likelihood ratio
exceeds 1/alpha. Returns the stopping time or None."""
s2, s = 2.0, 0.0
for n in range(1, nmax + 1):
s += rng.gauss(delta, math.sqrt(s2))
lr = math.sqrt(s2 / (s2 + n * tau2)) * math.exp(
tau2 * s * s / (2 * s2 * (s2 + n * tau2)))
if lr >= 1 / alpha:
return n
return None
if __name__ == "__main__":
K = 5; z = 1.959964; zb = 1.281552
drift = (z + zb) / math.sqrt(K) # per-look drift: 90 % power at look K
print(f"A. K = {K} looks, two-sided alpha 0.05; effect with 90 % power for one test")
for name, shape in (("Pocock", lambda k: 1.0),
("O'Brien-Fleming", lambda k: math.sqrt(K / k))):
c = calibrate(shape, K)
zs = [c * shape(k) for k in range(1, K + 1)]
pw, el = operating(c, shape, drift, K)
print(f" {name:16s} z-bounds " + " ".join(f"{x:.3f}" for x in zs))
print(f" {'':16s} power {pw:.3f}; expected looks {el:.2f} of {K}"
f" (null: {operating(c, shape, 0.0, K)[1]:.2f})")
n_inf = (pw_drift(c, shape, K) / (z + zb)) ** 2
print(f" {'':16s} max sample to keep 90 % power: {n_inf:.3f} x fixed design")
print(" fixed sample, one look at the end: power 0.900, looks 5 of 5")
print("B. mixture SPRT, checked after every pair, alpha 0.05, tau^2 = 1")
rng = random.Random(2017); n0 = 400
fp = sum(msprt_stop(rng, 0.0, nmax=5000) is not None for _ in range(n0))
print(f" A/A, 5,000 pairs each, {n0} runs: rejected {fp / n0:.3f}"
f" (any time; bound 0.05)")
for delta in (0.3, 0.1):
n_fix = math.ceil((z + zb) ** 2 * 2 / delta ** 2)
t = [msprt_stop(rng, delta) for _ in range(200)]
ok = sorted(x for x in t if x is not None)
print(f" effect {delta} sd: fixed design {n_fix:,} pairs; mSPRT stopped in"
f" {len(ok)}/200, median {ok[len(ok) // 2]:,}")
# A. K = 5 looks, two-sided alpha 0.05; effect with 90 % power for one test
# Pocock z-bounds 2.413 2.413 2.413 2.413 2.413
# power 0.835; expected looks 3.13 of 5 (null: 4.88)
# max sample to keep 90 % power: 1.206 x fixed design
# O'Brien-Fleming z-bounds 4.562 3.226 2.634 2.281 2.040
# power 0.892; expected looks 3.69 of 5 (null: 4.96)
# max sample to keep 90 % power: 1.026 x fixed design
# fixed sample, one look at the end: power 0.900, looks 5 of 5
# B. mixture SPRT, checked after every pair, alpha 0.05, tau^2 = 1
# A/A, 5,000 pairs each, 400 runs: rejected 0.022 (any time; bound 0.05)
# effect 0.3 sd: fixed design 234 pairs; mSPRT stopped in 200/200, median 194
# effect 0.1 sd: fixed design 2,102 pairs; mSPRT stopped in 200/200, median 2,193The Pocock boundary is 2.413 at every look; the O’Brien–Fleming boundary falls from 4.562 to 2.040 at the last look (close to the fixed-sample 1.96). With the same maximum sample, both have slightly less power than the fixed design (0.835 and 0.892 against 0.900), but they stop after 3.13 and 3.69 looks on average, saving about a third and a quarter of the sample; under the null they almost always run all 5 looks. To keep 90 % power, the Pocock design’s maximum sample must be 1.206 times the fixed design’s, O’Brien–Fleming’s only 1.026 times. This is the trade-off: Pocock stops early easily but, if the effect is small, needs more sample in the end; O’Brien–Fleming almost never stops early on a lucky early result, and its final test is almost the same as the fixed design’s. Marketing experiments can use the latter, or a more general α-spending function.
Always-valid inference: look whenever you like
Group-sequential designs require the number and timing of looks to be fixed in advance. On online experimentation platforms, users often want to look and stop at any time. Johari, Koomen, Pekelis and Walsh packaged sequential tests as always-valid p-values and confidence intervals: whenever, and for whatever reason, a user stops, the reported p-value guarantees a type I error no larger than the nominal level (Johari et al. 2017, 2022). They use the mixture sequential probability ratio test (mSPRT): take a mixing distribution over the effect (for example ), compute the mixture likelihood ratio
and reject the null as soon as . Because is a non-negative martingale under the null, Ville’s inequality guarantees that the probability it ever exceeds is at most , whatever the stopping rule. The idea goes back to Wald’s sequential probability ratio test and to Robbins’s statistical methods related to the law of the iterated logarithm (Wald 1945; Robbins 1970); Howard and co-authors later gave general, non-asymptotic, time-uniform confidence sequences (Howard et al. 2021).
Part B of the program checks after every pair of observations. In A/A tests, 2.2 % of 400 runs (5,000 pairs each) ever rejected the null, below the 5 % bound (the martingale inequality is conservative). The cost depends on the size of the effect: with an effect of 0.3 standard deviations the median stopping time of the mSPRT is 194 pairs, fewer than the fixed design’s 234; with 0.1 standard deviations the median is 2,193 pairs, slightly more than the fixed design’s 2,102. The choice of mixing variance decides which effect sizes it handles most efficiently; Johari and co-authors show that it still balances sample size and power efficiently without knowing a user’s preference between the two (Johari et al. 2022).
Evidence
Practice on large platforms. Johari and co-authors report that their method has been deployed on a large commercial A/B-testing platform and applied to hundreds of thousands of experiments (Johari et al. 2022). The handbook of online controlled experiments by Kohavi, Tang and Xu systematically discusses common pitfalls of online experiments such as peeking, sample-ratio mismatch, novelty effects and multiple comparisons (Kohavi et al. 2020).
Summary
With equally spaced looks and the unadjusted 1.96 at each, a nominal 5 % test has an actual type I error of 14.2 % with 5 looks, 24.8 % with 20 and 37.4 % with 100; “soaring to 25 % or more” holds only from about 20 looks. The problem is not looking but failing to fix a stopping rule in advance. Group-sequential designs adjust the critical values for a fixed number of looks (with 5 looks, Pocock 2.413, O’Brien–Fleming from 4.562 down to 2.040) and save a quarter to a third of the sample on average; always-valid inference allows looking and stopping at any time, at the cost of somewhat larger samples for small effects.
Hypothesis · not established · Peeking and reported lifts
The author’s hypothesis (not established): on online experimentation platforms without sequential correction, the lifts reported by experiments stopped early because they became significant are systematically larger than the lifts measured when the same variants are retested in later, independent randomised experiments (the joint effect of the winner’s curse and optional stopping). What result would show it to be wrong: in a pre-registered set of early-stopped experiments (sampled at random from platform logs and retested by independent researchers), the 95 % confidence interval of the geometric mean of the ratio of reported to retested lift contains 1.
References
Fan, Y. (2026). Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels: From Thompson Sampling Bandits and Absorbing Markov Chains to BG/NBD Customer Lifetime Value, Read Beside the Strategic Dispositions of the Sunzi [算法营销、多触点沙普利归因与获客漏斗的随机控制] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836264 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).