What does The Life of a Rule conclude, in numbers, about “Connecting Real Conversion Streams: Weekly Batches and Where the Baseline Comes From”?
Answer: Chapter 4 shows that weekly batches make the test more conservative, while re-estimating the baseline every week lifts false promotions to 0.1138.
Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.1138
| P(KEEP) after up to 12 weeks, 4000 runs per cell | ||||
| world | rule | GIVEN p0 | PLUG-IN p0 | TWO-ARM |
| steady | NULL | 0.0260 | 0.1138 | 0.0147 |
| steady | GOOD | 0.7190 | 0.6450 | 0.4015 |
| drift | NULL | 0.1708 | 0.1202 | 0.0175 |
| drift | GOOD | 0.9240 | 0.6763 | 0.4587 |
Show the full program
# Chapter 4. p42: where the baseline p0 comes from decides the error rate
# Model: control and variant each get 600 visitors a week for up to 12
# weeks. The one-arm SPRT of p31 (p1 = 1.2 p0, alpha 0.05, beta 0.20)
# is run on the variant with p0 either GIVEN (a historical 0.03) or
# PLUG-IN (the control's cumulative rate, re-estimated every week). The
# first crossing is latched. TWO-ARM is the conditional test of p43:
# each conversion is a coin that lands on the variant with chance 1/2
# under H0 and 1.2/2.2 under H1. Worlds: STEADY, both arms at 0.03;
# DRIFT, both arms rise to 0.0345 from week 5 (a season, illustrative).
# NULL means variant = control; GOOD means variant = 1.2 x control.
# Binomial draws by inverse CDF; fixed seed; 4,000 runs per cell.
import bisect
import math
import random
ALPHA, BETA, MDE = 0.05, 0.20, 0.20
A = math.log((1 - BETA) / ALPHA)
B = math.log(BETA / (1 - ALPHA))
M, WEEKS, RUNS = 600, 12, 4000
T1 = (1 + MDE) / (2 + MDE)
CA, CB = math.log(T1 / 0.5), math.log((1 - T1) / 0.5)
_cache = {}
def cdf(p):
if p not in _cache:
hi = min(M, int(M * p + 12 * math.sqrt(M * p * (1 - p))) + 1)
pm = [math.exp(math.lgamma(M + 1) - math.lgamma(k + 1)
- math.lgamma(M - k + 1) + k * math.log(p)
+ (M - k) * math.log(1 - p)) for k in range(hi + 1)]
_cache[p] = [math.fsum(pm[:k + 1]) for k in range(hi + 1)]
return _cache[p]
def draw(rng, p):
c = cdf(p)
return min(bisect.bisect_left(c, rng.random()), len(c) - 1)
def one_arm(nv, kv, p0):
p1 = p0 * (1 + MDE)
return kv * math.log(p1 / p0) + (nv - kv) * math.log((1 - p1)
/ (1 - p0))
def run(rng, world, lift):
verdict = {"given": None, "plugin": None, "two": None}
nc = kc = nv = kv = 0
for w in range(1, WEEKS + 1):
base = 0.03 if (world == "steady" or w < 5) else 0.0345
kc += draw(rng, base)
kv += draw(rng, base * (1 + lift))
nc += M
nv += M
scores = {"given": one_arm(nv, kv, 0.03),
"plugin": one_arm(nv, kv, max(kc, 1) / nc),
"two": kv * CA + kc * CB}
for key in ("given", "plugin", "two"):
if verdict[key] is None:
if scores[key] >= A:
verdict[key] = "keep"
elif scores[key] <= B:
verdict[key] = "cut"
return verdict
rng = random.Random(20261113)
print("P(KEEP) after up to %d weeks, %d runs per cell" % (WEEKS, RUNS))
print("world rule GIVEN p0 PLUG-IN p0 TWO-ARM")
for world in ("steady", "drift"):
for lab, lift in (("NULL", 0.0), ("GOOD", MDE)):
hits = {"given": 0, "plugin": 0, "two": 0}
for _ in range(RUNS):
v = run(rng, world, lift)
for key in hits:
hits[key] += v[key] == "keep"
print("%-6s %-5s %8.4f %10.4f %7.4f"
% (world, lab, hits["given"] / RUNS, hits["plugin"] / RUNS,
hits["two"] / RUNS))Single source and citation
Fan, Y. (2026). Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules (Version 1.0.0), Chapter 4: Connecting Real Conversion Streams: Weekly Batches and Where the Baseline Comes From. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-263 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026263,
author = {Fan, Yafu},
title = {Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · A Weekly Glance: How a Mediocre Copy Rule Gets Promoted by Mistake
- Chapter 2 · Wald's Two Straight Lines: Boundaries, the OC Function and the Mean Sample Size
- Chapter 3 · Exact Recursions: the Long Tail of the Sample Size and the Cost of a Cap
- Chapter 5 · The Rule Lifecycle as a State Machine: Promotion, Demotion and Retirement
- Chapter 6 · Many Rules in Parallel: Sharing Alpha, and How Many Promoted Rules Are Duds
- Chapter 7 · Classical Sources East and West: Checking by Trial in the Han Feizi, and Wald's Sequential Analysis
- Chapter 8 · Closing the Loop: Wiring the Weekly Radar's Verdicts into the Rule State Machine