What does Attribution, Bandits and Customer Lifetime Value conclude, in numbers, about “The Price of Peeking: Optional Stopping, Group-Sequential Designs and Always-Valid Inference”?

Answer: With equally spaced looks and the unadjusted 1.96 at each, a nominal 5 % test has an actual type I error of 14.2 % with 5 looks, 24.8 % with 20 and 37.4 % with 100; "soaring to 25 % or more" holds only from about 20 looks. The problem is not looking but failing to fix a stopping rule in advance. Group-sequential designs adjust the critical values for a fixed number of looks (with 5 looks, Pocock 2.413, O'Brien–Fleming from 4.562 down to 2.040) and save a quarter to a third of the sample on average; always-valid inference allows looking and stopping at any time, at the cost of somewhat larger samples for small effects.

Scope of the numbers: Which advertising channel should a sale be credited to? When may an A/B test stop? What is a customer still worth? This book presents mathematical models for teaching and research and is not marketing, investment or legal advice. Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's models and hypotheses; all data in the book are fictional or simulated.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 2.040, 2.413, 4.562

A. K = 5 looks, two-sided alpha 0.05; effect with 90 % power for one test
   Pocock           z-bounds 2.413 2.413 2.413 2.413 2.413
                    power 0.835; expected looks 3.13 of 5 (null: 4.88)
                    max sample to keep 90 % power: 1.206 x fixed design
   O'Brien-Fleming  z-bounds 4.562 3.226 2.634 2.281 2.040
                    power 0.892; expected looks 3.69 of 5 (null: 4.96)
                    max sample to keep 90 % power: 1.026 x fixed design
   fixed sample, one look at the end: power 0.900, looks 5 of 5
B. mixture SPRT, checked after every pair, alpha 0.05, tau^2 = 1
   A/A, 5,000 pairs each, 400 runs: rejected 0.022 (any time; bound 0.05)
   effect 0.3 sd: fixed design 234 pairs; mSPRT stopped in 200/200, median 194
   effect 0.1 sd: fixed design 2,102 pairs; mSPRT stopped in 200/200, median 2,193

Key formula

\[\Lambda_n = \int \prod_{i=1}^n \frac{f_\theta(x_i)}{f_0(x_i)}\,dH(\theta),\]
Show the full program
# Ways to look at a running test without inflating the error rate.
# A: group-sequential boundaries for K = 5 equally spaced looks, two-sided alpha 0.05:
#    Pocock (one constant z) and O'Brien-Fleming (z_k = c sqrt(K / k)), constants found
#    by bisection on the exact crossing probability of p04; power at the drift that
#    gives a fixed-sample test 90 % power, and the expected number of looks used.
# B: mixture SPRT (normal mixture, Johari et al.) monitored after every observation:
#    always-valid type I error, and how long it takes to stop under an effect.
import math, random

# --- helpers repeated from earlier programs, so that this one runs on its own ---
def stay_probs(bounds, h=0.1, drift=0.0):
    """P(|S_j| < bounds[j-1] for all j <= k), k = 1..K, S a Gaussian random walk with
    N(drift, 1) steps: Simpson's rule on a grid spanning each continuation interval."""
    phi = lambda x: math.exp(-x * x / 2) / math.sqrt(2 * math.pi)
    def grid(b):
        n = 2 * max(1, math.ceil(b / h)); d = 2 * b / n
        w = [(1 if i in (0, n) else 4 if i % 2 else 2) * d / 3 for i in range(n + 1)]
        return [-b + i * d for i in range(n + 1)], w
    xs, w = grid(bounds[0]); f = [phi(x - drift) for x in xs]
    out = [sum(a * b for a, b in zip(w, f))]
    for b in bounds[1:]:
        ys, v = grid(b)
        g = [sum(wi * fi * phi(y - x - drift) for x, wi, fi in zip(xs, w, f)
                 if abs(y - x - drift) < 8.5)
             for y in ys]
        xs, w, f = ys, v, g
        out.append(sum(a * b for a, b in zip(w, f)))
    return out
# --- end of repeated helpers ---

def calibrate(shape, K=5, alpha=0.05):
    lo, hi = 1.5, 5.0
    for _ in range(40):
        c = (lo + hi) / 2
        b = [c * shape(k) * math.sqrt(k) for k in range(1, K + 1)]
        if 1 - stay_probs(b)[-1] > alpha:
            lo = c
        else:
            hi = c
    return (lo + hi) / 2

def operating(c, shape, drift, K=5):
    b = [c * shape(k) * math.sqrt(k) for k in range(1, K + 1)]
    st = stay_probs(b, drift=drift)               # ignores the (tiny) wrong-side crossing
    return 1 - st[-1], 1 + sum(st[:-1])

def pw_drift(c, shape, K, target=0.90):
    lo, hi = 0.5, 3.0                             # drift per look giving target power
    for _ in range(30):
        d = (lo + hi) / 2
        if operating(c, shape, d, K)[0] < target:
            lo = d
        else:
            hi = d
    return (lo + hi) / 2 * math.sqrt(K)

def msprt_stop(rng, delta, tau2=1.0, alpha=0.05, nmax=20_000):
    """Paired differences x_i ~ N(delta, 2); stop when the mixture likelihood ratio
    exceeds 1/alpha. Returns the stopping time or None."""
    s2, s = 2.0, 0.0
    for n in range(1, nmax + 1):
        s += rng.gauss(delta, math.sqrt(s2))
        lr = math.sqrt(s2 / (s2 + n * tau2)) * math.exp(
            tau2 * s * s / (2 * s2 * (s2 + n * tau2)))
        if lr >= 1 / alpha:
            return n
    return None

if __name__ == "__main__":
    K = 5; z = 1.959964; zb = 1.281552
    drift = (z + zb) / math.sqrt(K)               # per-look drift: 90 % power at look K
    print(f"A. K = {K} looks, two-sided alpha 0.05; effect with 90 % power for one test")
    for name, shape in (("Pocock", lambda k: 1.0),
                        ("O'Brien-Fleming", lambda k: math.sqrt(K / k))):
        c = calibrate(shape, K)
        zs = [c * shape(k) for k in range(1, K + 1)]
        pw, el = operating(c, shape, drift, K)
        print(f"   {name:16s} z-bounds " + " ".join(f"{x:.3f}" for x in zs))
        print(f"   {'':16s} power {pw:.3f}; expected looks {el:.2f} of {K}"
              f" (null: {operating(c, shape, 0.0, K)[1]:.2f})")
        n_inf = (pw_drift(c, shape, K) / (z + zb)) ** 2
        print(f"   {'':16s} max sample to keep 90 % power: {n_inf:.3f} x fixed design")
    print("   fixed sample, one look at the end: power 0.900, looks 5 of 5")
    print("B. mixture SPRT, checked after every pair, alpha 0.05, tau^2 = 1")
    rng = random.Random(2017); n0 = 400
    fp = sum(msprt_stop(rng, 0.0, nmax=5000) is not None for _ in range(n0))
    print(f"   A/A, 5,000 pairs each, {n0} runs: rejected {fp / n0:.3f}"
          f" (any time; bound 0.05)")
    for delta in (0.3, 0.1):
        n_fix = math.ceil((z + zb) ** 2 * 2 / delta ** 2)
        t = [msprt_stop(rng, delta) for _ in range(200)]
        ok = sorted(x for x in t if x is not None)
        print(f"   effect {delta} sd: fixed design {n_fix:,} pairs; mSPRT stopped in"
              f" {len(ok)}/200, median {ok[len(ok) // 2]:,}")

Single source and citation

Fan, Y. (2026). Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels (Version 1.0.0), Chapter 3: The Price of Peeking: Optional Stopping, Group-Sequential Designs and Always-Valid Inference. 5D Library. https://doi.org/10.5281/zenodo.23836264

DOI: 10.5281/zenodo.23836264 · Read the monograph · 5DL-B-048 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026048,
  author    = {Fan, Yafu},
  title     = {Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836264},
  url       = {https://doi.org/10.5281/zenodo.23836264}
}

Other chapter answers in this book