What does The Life of a Rule conclude, in numbers, about “Exact Recursions: the Long Tail of the Sample Size and the Cost of a Cap”?

Answer: Exact recursion adds three things to Wald's formulas. First, the true error rates are 0.0470 and 0.2000, both inside the bounds 0.0625 and 0.2105. Second, the mean sample size is 0.464 and 0.632 of the fixed sample at the two design points and rises to 0.748 for middling rules, nearly a quarter of which run longer than a fixed sample. Third, a cap has a price: removing all open rules at a nine-week cap leaves a power of only 0.6548, while settling them by sign lifts the type I error rate to 0.0876. All of this assumes a look after every visitor and an exactly known baseline. Yet the weekly radar looks once a week, and the baseline must be estimated from the control. Which way do batches and the source of the baseline push these numbers? Chapter 4 connects the test to real conversion streams.

Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0470, 0.0625, 0.2000, 0.2105, 0.464, 0.632

exact SPRT, p0 0.030 vs p1 0.036, alpha 0.05, beta 0.20
truth P(KEEP) P(CUT) open@60k E[N] Wald E[N] median p90
p00.0470 0.9530 5.8e-12 2468 2444 1740 5265
p10.8000 0.2000 1.0e-11 3360 3275 2713 6538
p0: E[N]/fixed n 0.464 P(N > fixed n 5314)0.0977
p1: E[N]/fixed n 0.632 P(N > fixed n 5314)0.1692
Wald bounds: alpha' <0.0625, beta' <= 0.2105; exact: 0.0470, 0.2000
Show the full program
# Chapter 3. p31: exact OC and ASN of the Bernoulli SPRT, visitor by visitor
# Model: SPRT of p0 0.03 against p1 0.036 with alpha 0.05, beta 0.20
# (illustrative p0 and lift, charter error rates). Because the log
# ratio after n visitors depends only on the conversion count k, the
# set of open paths is a short list of k values; we push their exact
# probabilities forward one visitor at a time and record the mass that
# leaves through KEEP or CUT. E[N] is the sum of survival probabilities
# P(N > n). Run to 60,000 visitors; the mass still open is printed.
import math

P0, P1 = 0.03, 0.036
ALPHA, BETA = 0.05, 0.20
A = math.log((1 - BETA) / ALPHA)
B = math.log(BETA / (1 - ALPHA))
LA = math.log(P1 / P0)
LB = math.log((1 - P1) / (1 - P0))
N_MAX = 60000
N_FIX = 5314                    # fixed-sample n from p22


def exact(p, n_max=N_MAX):
    open_k = {0: 1.0}
    keep, cut, surv = [], [], [1.0]
    for n in range(1, n_max + 1):
        nxt = {}
        for k, m in open_k.items():
            nxt[k] = nxt.get(k, 0.0) + m * (1 - p)
            nxt[k + 1] = nxt.get(k + 1, 0.0) + m * p
        open_k = {}
        for k in sorted(nxt):
            llr = k * LA + (n - k) * LB
            if llr >= A:
                keep.append(nxt[k])
            elif llr <= B:
                cut.append(nxt[k])
            else:
                open_k[k] = nxt[k]
        surv.append(math.fsum(open_k.values()))
    return math.fsum(keep), math.fsum(cut), surv


def quantile(surv, q):
    for n, s in enumerate(surv):
        if 1 - s >= q:
            return n
    return None


def wald(p):
    ez = math.fsum([p * LA, (1 - p) * LB])
    oc = 1 - ALPHA if p == P0 else BETA
    return math.fsum([oc * B, (1 - oc) * A]) / ez


print("exact SPRT, p0 %.3f vs p1 %.3f, alpha %.2f, beta %.2f"
      % (P0, P1, ALPHA, BETA))
print("truth  P(KEEP)  P(CUT)  open@60k   E[N]  Wald E[N]  median"
      "  p90")
rows = {}
for lab, p in (("p0", P0), ("p1", P1)):
    kp, cp, surv = exact(p)
    en = math.fsum(surv[:-1])
    rows[lab] = (kp, cp, surv, en)
    print("%-5s  %7.4f  %6.4f  %8.1e  %5.0f  %9.0f  %6d  %5d"
          % (lab, kp, cp, surv[-1], en, wald(p), quantile(surv, 0.5),
             quantile(surv, 0.9)))
for lab in ("p0", "p1"):
    kp, cp, surv, en = rows[lab]
    print("%s: E[N]/fixed n %.3f   P(N > fixed n %d) = %.4f"
          % (lab, en / N_FIX, N_FIX, surv[N_FIX]))
print("Wald bounds: alpha' <= %.4f, beta' <= %.4f; exact: %.4f, %.4f"
      % (ALPHA / (1 - BETA), BETA / (1 - ALPHA), rows["p0"][0],
         rows["p1"][1]))

Single source and citation

Fan, Y. (2026). Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules (Version 1.0.0), Chapter 3: Exact Recursions: the Long Tail of the Sample Size and the Cost of a Cap. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-263 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026263,
  author    = {Fan, Yafu},
  title     = {Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book