What does False Discovery Rates for Copy Tests conclude, in numbers, about “Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan”?
Answer: Chapter 8 adds multiple-rule correction to the weekly review radar, where e-BH lowers the false discovery rate from 0.0896 to 0.0023, and presents the distilled FDR Step-Up Ladder and a pre-registered falsification plan. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0023, 0.0896
| A = 2.7726, B = -1.5581, 24 rules, 4 real | ||||
| layer | false/run | P(any false) | FDR | power |
| SPRT KEEP | 0.424 | 0.3540 | 0.0896 | 0.886 |
| e-BH | 0.009 | 0.0095 | 0.0023 | 0.792 |
| Ville+BH | 0.054 | 0.0515 | 0.0120 | 0.833 |
Show the full program
# Chapter 8. p81: a weekly radar that runs one SPRT per copy rule, and
# the multiple-rule layer added on top. Model (illustrative): 24 rules,
# baseline p0 = 0.030 known, mde 20 % (p1 = 0.036), alpha 0.05, beta
# 0.20, Wald bounds A = ln(16), B = ln(0.2/0.95); 1800 visitors per
# rule per week, 8 weeks; four rules truly convert at 0.036, the rest
# at 0.030. Counts use the normal approximation to the binomial.
# Layer 1 (as is): a rule is KEEP once its log-likelihood ratio
# crosses A. Layer 2: at the week-8 review the likelihood ratio of
# each rule is an e-value; e-BH (Wang and Ramdas 2022) rejects the k
# largest with e_(k) >= m / (q k), valid under any dependence.
# Layer 3: Ville p-value 1 / max_t LR_t, then BH. 2000 runs, seed
# 20261113.
import math
import random
M, REAL, P0, MDE, ALPHA, BETA = 24, 4, 0.030, 0.20, 0.05, 0.20
NW, WEEKS, Q, RUNS, SEED = 1800, 8, 0.05, 2000, 20261113
P1 = P0 * (1 + MDE)
A = math.log((1 - BETA) / ALPHA)
B = math.log(BETA / (1 - ALPHA))
UP, DN = math.log(P1 / P0), math.log((1 - P1) / (1 - P0))
def rule_path(rng, p):
terms, top, keep, decided = [], 0.0, False, False
for _ in range(WEEKS):
k = round(NW * p + math.sqrt(NW * p * (1 - p)) * rng.gauss(0, 1))
k = max(0, k)
terms.append(k * UP + (NW - k) * DN)
llr = math.fsum(terms)
top = max(top, llr)
if not decided and llr >= A:
keep, decided = True, True
elif not decided and llr <= B:
decided = True
return keep, math.exp(llr), min(1.0, math.exp(-top))
def ebh(es, q):
m = len(es)
idx = sorted(range(m), key=lambda i: (-es[i], i))
kmax = 0
for k, i in enumerate(idx, 1):
if es[i] >= m / (q * k):
kmax = k
return set(idx[:kmax])
def bh(ps, q):
m = len(ps)
idx = sorted(range(m), key=lambda i: (ps[i], i))
kmax = 0
for k, i in enumerate(idx, 1):
if ps[i] <= k * q / m:
kmax = k
return set(idx[:kmax])
if __name__ == "__main__":
rng = random.Random(20261113)
stats = {n: ([], [], []) for n in ("SPRT KEEP", "e-BH", "Ville+BH")}
for _ in range(RUNS):
res = [rule_path(rng, P1 if i < REAL else P0)
for i in range(M)]
sets = {"SPRT KEEP": {i for i in range(M) if res[i][0]},
"e-BH": ebh([r[1] for r in res], Q),
"Ville+BH": bh([r[2] for r in res], Q)}
for n, s in sets.items():
v = len([i for i in s if i >= REAL])
stats[n][0].append(float(v))
stats[n][1].append(v / max(len(s), 1))
stats[n][2].append((len(s) - v) / REAL)
print("A = {:.4f}, B = {:.4f}, {} rules, {} real".format(A, B, M, REAL))
print("layer false/run P(any false) FDR power")
for n, (fv, fd, pw) in stats.items():
anyf = len([x for x in fv if x > 0]) / RUNS
print("{:10s} {:.3f} {:.4f} {:.4f} {:.3f}".format(
n, math.fsum(fv) / RUNS, anyf, math.fsum(fd) / RUNS,
math.fsum(pw) / RUNS))Single source and citation
Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 8: Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Two False Champions among Forty Useless Rules
- Chapter 2 · Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction
- Chapter 3 · False Discovery Rate and Family-Wise Error Rate: Two Different Targets
- Chapter 4 · Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules
- Chapter 5 · Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction
- Chapter 6 · Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control
- Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities