What does The Life of a Rule conclude, in numbers, about “Closing the Loop: Wiring the Weekly Radar's Verdicts into the Rule State Machine”?

Answer: Chapter 8 wires the weekly radar's verdicts into the state machine and gives the distilled tool, the Rule Promotion and Demotion Judge, and a pre-registered A/A calibration: 100 A/A rules, with 11 or more promotions overturning the book. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 100, 11, 20

A/A rulesreject if KEEPs >=size@0.0625size@0.0332pow@0.10pow@0.15
2040.03300.00380.13300.3523
4060.03670.00190.20630.5675
6080.03260.00080.24840.6953
80100.02730.00030.27660.7789
100110.04800.00050.41680.9006
120130.03750.00020.42390.9254
150150.04910.00020.53980.9717
200190.04620.00000.62760.9918
smallest listed n with power >= 0.80 against 0.15: 100
at 20 A/A slots and 5.83 weeks per null trial: 29.1 weeks
Show the full program
# Chapter 8. p82: how many A/A calibration rules the pre-registration needs
# Model: an A/A rule sends traffic to a variant identical to control, so
# every KEEP it earns is a false promotion. The book's claim is that the
# weekly SPRT pipeline keeps the false-KEEP rate at or below Wald's
# bound 0.0625; its exact value at 600 visitors a week is 0.0332 (p41).
# The rival claim is that it behaves like weekly peeking, near 0.15 or
# more (p11, p12). For n A/A rules we pick the smallest critical count
# c with exact one-sided binomial size <= 0.05 under rate 0.0625, and
# report the power against 0.10 and 0.15. Exact binomial sums.
import math

R0, R_TRUE = 0.0625, 0.0332


def tail(n, c, p):
    terms = [math.exp(math.lgamma(n + 1) - math.lgamma(j + 1)
                      - math.lgamma(n - j + 1) + j * math.log(p)
                      + (n - j) * math.log(1 - p))
             for j in range(c, n + 1)]
    return math.fsum(terms)


def crit(n):
    for c in range(n + 1):
        if tail(n, c, R0) <= 0.05:
            return c
    return n + 1


print("A/A rules  reject if KEEPs >=  size@0.0625  size@0.0332"
      "  pow@0.10  pow@0.15")
first = None
for n in (20, 40, 60, 80, 100, 120, 150, 200):
    c = crit(n)
    pw = tail(n, c, 0.15)
    if first is None and pw >= 0.8:
        first = n
    print("%9d  %18d  %11.4f  %11.4f  %8.4f  %8.4f"
          % (n, c, tail(n, c, R0), tail(n, c, R_TRUE), tail(n, c, 0.10),
             pw))
print("smallest listed n with power >= 0.80 against 0.15: %d" % first)
weeks = first * 5.83 / 20
print("at 20 A/A slots and 5.83 weeks per null trial: %.1f weeks" % weeks)

Single source and citation

Fan, Y. (2026). Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules (Version 1.0.0), Chapter 8: Closing the Loop: Wiring the Weekly Radar's Verdicts into the Rule State Machine. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-263 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026263,
  author    = {Fan, Yafu},
  title     = {Empirical Retirement of Live Rules: Wiring the Sequential Probability Ratio Test into Real Conversion Streams to Promote, Demote and Retire Copy Rules},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book