What does The Life of a Rule conclude, in numbers, about “Exact Recursions: the Long Tail of the Sample Size and the Cost of a Cap”?
Answer: Exact recursion adds three things to Wald's formulas. First, the true error rates are 0.0470 and 0.2000, both inside the bounds 0.0625 and 0.2105. Second, the mean sample size is 0.464 and 0.632 of the fixed sample at the two design points and rises to 0.748 for middling rules, nearly a quarter of which run longer than a fixed sample. Third, a cap has a price: removing all open rules at a nine-week cap leaves a power of only 0.6548, while settling them by sign lifts the type I error rate to 0.0876. All of this assumes a look after every visitor and an exactly known baseline. Yet the weekly radar looks once a week, and the baseline must be estimated from the control. Which way do batches and the source of the baseline push these numbers? Chapter 4 connects the test to real conversion streams.
Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0470, 0.0625, 0.2000, 0.2105, 0.464, 0.632
| exact SPRT, p0 0.030 vs p1 0.036, alpha 0.05, beta 0.20 | |
| truth P(KEEP) P(CUT) open@60k E[N] Wald E[N] median p90 | |
| p0 | 0.0470 0.9530 5.8e-12 2468 2444 1740 5265 |
| p1 | 0.8000 0.2000 1.0e-11 3360 3275 2713 6538 |
| p0: E[N]/fixed n 0.464 P(N > fixed n 5314) | 0.0977 |
| p1: E[N]/fixed n 0.632 P(N > fixed n 5314) | 0.1692 |
| Wald bounds: alpha' < | 0.0625, beta' <= 0.2105; exact: 0.0470, 0.2000 |
Show the full program
# Chapter 3. p31: exact OC and ASN of the Bernoulli SPRT, visitor by visitor
# Model: SPRT of p0 0.03 against p1 0.036 with alpha 0.05, beta 0.20
# (illustrative p0 and lift, charter error rates). Because the log
# ratio after n visitors depends only on the conversion count k, the
# set of open paths is a short list of k values; we push their exact
# probabilities forward one visitor at a time and record the mass that
# leaves through KEEP or CUT. E[N] is the sum of survival probabilities
# P(N > n). Run to 60,000 visitors; the mass still open is printed.
import math
P0, P1 = 0.03, 0.036
ALPHA, BETA = 0.05, 0.20
A = math.log((1 - BETA) / ALPHA)
B = math.log(BETA / (1 - ALPHA))
LA = math.log(P1 / P0)
LB = math.log((1 - P1) / (1 - P0))
N_MAX = 60000
N_FIX = 5314 # fixed-sample n from p22
def exact(p, n_max=N_MAX):
open_k = {0: 1.0}
keep, cut, surv = [], [], [1.0]
for n in range(1, n_max + 1):
nxt = {}
for k, m in open_k.items():
nxt[k] = nxt.get(k, 0.0) + m * (1 - p)
nxt[k + 1] = nxt.get(k + 1, 0.0) + m * p
open_k = {}
for k in sorted(nxt):
llr = k * LA + (n - k) * LB
if llr >= A:
keep.append(nxt[k])
elif llr <= B:
cut.append(nxt[k])
else:
open_k[k] = nxt[k]
surv.append(math.fsum(open_k.values()))
return math.fsum(keep), math.fsum(cut), surv
def quantile(surv, q):
for n, s in enumerate(surv):
if 1 - s >= q:
return n
return None
def wald(p):
ez = math.fsum([p * LA, (1 - p) * LB])
oc = 1 - ALPHA if p == P0 else BETA
return math.fsum([oc * B, (1 - oc) * A]) / ez
print("exact SPRT, p0 %.3f vs p1 %.3f, alpha %.2f, beta %.2f"
% (P0, P1, ALPHA, BETA))
print("truth P(KEEP) P(CUT) open@60k E[N] Wald E[N] median"
" p90")
rows = {}
for lab, p in (("p0", P0), ("p1", P1)):
kp, cp, surv = exact(p)
en = math.fsum(surv[:-1])
rows[lab] = (kp, cp, surv, en)
print("%-5s %7.4f %6.4f %8.1e %5.0f %9.0f %6d %5d"
% (lab, kp, cp, surv[-1], en, wald(p), quantile(surv, 0.5),
quantile(surv, 0.9)))
for lab in ("p0", "p1"):
kp, cp, surv, en = rows[lab]
print("%s: E[N]/fixed n %.3f P(N > fixed n %d) = %.4f"
% (lab, en / N_FIX, N_FIX, surv[N_FIX]))
print("Wald bounds: alpha' <= %.4f, beta' <= %.4f; exact: %.4f, %.4f"
% (ALPHA / (1 - BETA), BETA / (1 - ALPHA), rows["p0"][0],
rows["p1"][1]))Single source and citation
Fan, Y. (2026). Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules (Version 1.0.0), Chapter 3: Exact Recursions: the Long Tail of the Sample Size and the Cost of a Cap. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-263 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026263,
author = {Fan, Yafu},
title = {Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · A Weekly Glance: How a Mediocre Copy Rule Gets Promoted by Mistake
- Chapter 2 · Wald's Two Straight Lines: Boundaries, the OC Function and the Mean Sample Size
- Chapter 4 · Connecting Real Conversion Streams: Weekly Batches and Where the Baseline Comes From
- Chapter 5 · The Rule Lifecycle as a State Machine: Promotion, Demotion and Retirement
- Chapter 6 · Many Rules in Parallel: Sharing Alpha, and How Many Promoted Rules Are Duds
- Chapter 7 · Classical Sources East and West: Checking by Trial in the Han Feizi, and Wald's Sequential Analysis
- Chapter 8 · Closing the Loop: Wiring the Weekly Radar's Verdicts into the Rule State Machine