What does The Life of a Rule conclude, in numbers, about “Connecting Real Conversion Streams: Weekly Batches and Where the Baseline Comes From”?

Answer: Chapter 4 shows that weekly batches make the test more conservative, while re-estimating the baseline every week lifts false promotions to 0.1138.

Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.1138

P(KEEP) after up to 12 weeks, 4000 runs per cell
worldruleGIVEN p0PLUG-IN p0TWO-ARM
steadyNULL0.02600.11380.0147
steadyGOOD0.71900.64500.4015
driftNULL0.17080.12020.0175
driftGOOD0.92400.67630.4587
Show the full program
# Chapter 4. p42: where the baseline p0 comes from decides the error rate
# Model: control and variant each get 600 visitors a week for up to 12
# weeks. The one-arm SPRT of p31 (p1 = 1.2 p0, alpha 0.05, beta 0.20)
# is run on the variant with p0 either GIVEN (a historical 0.03) or
# PLUG-IN (the control's cumulative rate, re-estimated every week). The
# first crossing is latched. TWO-ARM is the conditional test of p43:
# each conversion is a coin that lands on the variant with chance 1/2
# under H0 and 1.2/2.2 under H1. Worlds: STEADY, both arms at 0.03;
# DRIFT, both arms rise to 0.0345 from week 5 (a season, illustrative).
# NULL means variant = control; GOOD means variant = 1.2 x control.
# Binomial draws by inverse CDF; fixed seed; 4,000 runs per cell.
import bisect
import math
import random

ALPHA, BETA, MDE = 0.05, 0.20, 0.20
A = math.log((1 - BETA) / ALPHA)
B = math.log(BETA / (1 - ALPHA))
M, WEEKS, RUNS = 600, 12, 4000
T1 = (1 + MDE) / (2 + MDE)
CA, CB = math.log(T1 / 0.5), math.log((1 - T1) / 0.5)
_cache = {}


def cdf(p):
    if p not in _cache:
        hi = min(M, int(M * p + 12 * math.sqrt(M * p * (1 - p))) + 1)
        pm = [math.exp(math.lgamma(M + 1) - math.lgamma(k + 1)
                       - math.lgamma(M - k + 1) + k * math.log(p)
                       + (M - k) * math.log(1 - p)) for k in range(hi + 1)]
        _cache[p] = [math.fsum(pm[:k + 1]) for k in range(hi + 1)]
    return _cache[p]


def draw(rng, p):
    c = cdf(p)
    return min(bisect.bisect_left(c, rng.random()), len(c) - 1)


def one_arm(nv, kv, p0):
    p1 = p0 * (1 + MDE)
    return kv * math.log(p1 / p0) + (nv - kv) * math.log((1 - p1)
                                                         / (1 - p0))


def run(rng, world, lift):
    verdict = {"given": None, "plugin": None, "two": None}
    nc = kc = nv = kv = 0
    for w in range(1, WEEKS + 1):
        base = 0.03 if (world == "steady" or w < 5) else 0.0345
        kc += draw(rng, base)
        kv += draw(rng, base * (1 + lift))
        nc += M
        nv += M
        scores = {"given": one_arm(nv, kv, 0.03),
                  "plugin": one_arm(nv, kv, max(kc, 1) / nc),
                  "two": kv * CA + kc * CB}
        for key in ("given", "plugin", "two"):
            if verdict[key] is None:
                if scores[key] >= A:
                    verdict[key] = "keep"
                elif scores[key] <= B:
                    verdict[key] = "cut"
    return verdict


rng = random.Random(20261113)
print("P(KEEP) after up to %d weeks, %d runs per cell" % (WEEKS, RUNS))
print("world   rule   GIVEN p0  PLUG-IN p0  TWO-ARM")
for world in ("steady", "drift"):
    for lab, lift in (("NULL", 0.0), ("GOOD", MDE)):
        hits = {"given": 0, "plugin": 0, "two": 0}
        for _ in range(RUNS):
            v = run(rng, world, lift)
            for key in hits:
                hits[key] += v[key] == "keep"
        print("%-6s  %-5s  %8.4f  %10.4f  %7.4f"
              % (world, lab, hits["given"] / RUNS, hits["plugin"] / RUNS,
                 hits["two"] / RUNS))

Single source and citation

Fan, Y. (2026). Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules (Version 1.0.0), Chapter 4: Connecting Real Conversion Streams: Weekly Batches and Where the Baseline Comes From. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-263 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026263,
  author    = {Fan, Yafu},
  title     = {Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book