What does False Discovery Rates for Copy Tests conclude, in numbers, about “Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control”?
Answer: Chapter 6 converts correction into visitors per arm: 10,960 without correction, 26,476 under Bonferroni, and about 18,555 under BH when a fifth of rules are real.
Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 18555, 26476
| raw level 0.05: | n per arm = 10,958 | |||
| Bonferroni 0.05/40: | n per arm = 26,476 | |||
| pi1 | n per arm | BH threshold | equiv. m | vs Bonf |
| 0.05 | 24,321 | 0.002100 | 23.8 | 0.919 |
| 0.10 | 21,441 | 0.004189 | 11.9 | 0.810 |
| 0.20 | 18,555 | 0.008333 | 6.0 | 0.701 |
| 0.30 | 16,869 | 0.012436 | 4.0 | 0.637 |
| 0.50 | 14,751 | 0.020513 | 2.4 | 0.557 |
Show the full program
# Chapter 6. p62: the threshold BH settles on, and the n it implies.
# Model: many rules, a share pi1 real with the same lift; at n per arm
# a real rule's z-drift is d(n) = (p1 - p0) / sqrt((p0 q0 + p1 q1) / n)
# and its power at threshold t is G(t) = 1 - Phi(z_(1-t) - d). In the
# large-m limit BH rejects p <= t*, the largest t with
# t = q (pi0 t + pi1 G(t)),
# found by iterating from t = q. We then search the smallest n with
# G(t*) >= 0.80. p0 = 0.030, lift 20 %, q = 0.05, m = 40 for
# Bonferroni. All parameters illustrative.
from statistics import NormalDist
import math
P0, LIFT, Q, M, TARGET = 0.030, 0.20, 0.05, 40, 0.80
ND = NormalDist()
P1 = P0 * (1 + LIFT)
def drift(n):
return (P1 - P0) / math.sqrt((P0 * (1 - P0) + P1 * (1 - P1)) / n)
def power(t, n):
return 1 - ND.cdf(ND.inv_cdf(1 - t) - drift(n))
def bh_threshold(pi1, n):
t = Q
for _ in range(2000):
nxt = Q * ((1 - pi1) * t + pi1 * power(t, n))
if abs(nxt - t) < 1e-15:
break
t = nxt
return t
def smallest_n(fn):
lo, hi = 10, 10 ** 7
while lo < hi:
mid = (lo + hi) // 2
if fn(mid) >= TARGET:
hi = mid
else:
lo = mid + 1
return lo
if __name__ == "__main__":
n_raw = smallest_n(lambda n: power(Q, n))
n_bon = smallest_n(lambda n: power(Q / M, n))
print("raw level 0.05: n per arm = {:,}".format(n_raw))
print("Bonferroni 0.05/40: n per arm = {:,}".format(n_bon))
print(" pi1 n per arm BH threshold equiv. m vs Bonf")
for pi1 in (0.05, 0.1, 0.2, 0.3, 0.5):
n = smallest_n(lambda k: power(bh_threshold(pi1, k), k))
t = bh_threshold(pi1, n)
print(" {:.2f} {:>9,} {:.6f} {:5.1f} {:.3f}".format(
pi1, n, t, Q / t, n / n_bon))Single source and citation
Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 6: Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Two False Champions among Forty Useless Rules
- Chapter 2 · Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction
- Chapter 3 · False Discovery Rate and Family-Wise Error Rate: Two Different Targets
- Chapter 4 · Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules
- Chapter 5 · Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction
- Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities
- Chapter 8 · Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan