What does False Discovery Rates for Copy Tests conclude, in numbers, about “False Discovery Rate and Family-Wise Error Rate: Two Different Targets”?
Answer: Without correction the share of weeks with a false champion is 0.8722, matching the exact value of 0.8715, with 1.9939 false champions per week on average. Bonferroni and Holm both come in at 0.0469; when everything is useless the two are identical, because Holm's first rung is the Bonferroni bar and a failed first rung stops everything. BH comes in at 0.0481, with 0.0529 false discoveries on average, only slightly above Bonferroni. BY is at just 0.0115, clearly more cautious.
Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0115, 0.0469, 0.0481, 0.0529, 0.8715, 0.8722
| m = 40, all nulls true, 20000 weeks | ||
| rule | P(V>=1)=FWER=FDR | mean V |
| none | 0.8722 | 1.9939 |
| Bonf | 0.0469 | 0.0476 |
| Holm | 0.0469 | 0.0476 |
| BH | 0.0481 | 0.0529 |
| BY | 0.0115 | 0.0116 |
| exact for none: 0.8715; Bonferroni bound 0.05 |
Show the full program
# Chapter 3. p31: when every rule is useless, FDR and FWER coincide.
# Model: m = 40 independent one-sided z-tests, all nulls true; 20000
# simulated weeks, seed 20261113. Level 0.05 for each procedure.
# With no true effect, any rejection is a false one and the false
# discovery proportion is 1, so FDR = P(V >= 1) = FWER.
import math
import random
M, Q, REPS, SEED = 40, 0.05, 20000, 20261113
def bh_count(ps, q):
m, kmax = len(ps), 0
for k, p in enumerate(sorted(ps), 1):
if p <= k * q / m:
kmax = k
return kmax
def holm_count(ps, a):
m = len(ps)
for k, p in enumerate(sorted(ps)):
if p > a / (m - k):
return k
return m
if __name__ == "__main__":
rng = random.Random(20261113)
hm = math.fsum(1.0 / j for j in range(1, M + 1))
names = ("none", "Bonf", "Holm", "BH", "BY")
hit = {n: [] for n in names}
vol = {n: [] for n in names}
for _ in range(REPS):
ps = [0.5 * math.erfc(rng.gauss(0, 1) / math.sqrt(2))
for _ in range(M)]
v = {"none": len([p for p in ps if p < Q]),
"Bonf": len([p for p in ps if p <= Q / M]),
"Holm": holm_count(ps, Q),
"BH": bh_count(ps, Q),
"BY": bh_count(ps, Q / hm)}
for n in names:
hit[n].append(1.0 if v[n] else 0.0)
vol[n].append(float(v[n]))
print("m = 40, all nulls true, {} weeks".format(REPS))
print("rule P(V>=1)=FWER=FDR mean V")
for n in names:
print("{:5s} {:.4f} {:.4f}".format(
n, math.fsum(hit[n]) / REPS, math.fsum(vol[n]) / REPS))
print("exact for none: {:.4f}; Bonferroni bound 0.05".format(
1 - 0.95 ** M))Single source and citation
Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 3: False Discovery Rate and Family-Wise Error Rate: Two Different Targets. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Two False Champions among Forty Useless Rules
- Chapter 2 · Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction
- Chapter 4 · Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules
- Chapter 5 · Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction
- Chapter 6 · Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control
- Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities
- Chapter 8 · Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan