What does False Discovery Rates for Copy Tests conclude, in numbers, about “Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction”?
Answer: Rows five and six show where the extra two come from. R18 has p-value 0.001665, above the Bonferroni bar of 0.00125, so Bonferroni turns it down; but it sits fifth, and BH sets the fifth rung at 0.00625, which it clears easily. R33 has p-value 0.002019 against a sixth rung of 0.00750, and clears too. R03, seventh, is a real rule with p-value 0.019500, above the seventh rung of 0.00875, and none of the four rulers rejects it. Real rules can also land outside the bar because of that week's random fluctuation, and no method escapes this.
Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.00125, 0.001665, 0.002019, 0.00625, 0.00750, 0.00875
| rank rule | truth | p-value | k*q/m | Bonf Holm | BH | BY | ||
| 1 | R38 | real | 0.000048 | 0.00125 | x | x | x | x |
| 2 | R12 | real | 0.000074 | 0.00250 | x | x | x | x |
| 3 | R26 | real | 0.000280 | 0.00375 | x | x | x | x |
| 4 | R07 | real | 0.000659 | 0.00500 | x | x | x | x |
| 5 | R18 | real | 0.001665 | 0.00625 | . | . | x | . |
| 6 | R33 | real | 0.002019 | 0.00750 | . | . | x | . |
| 7 | R03 | real | 0.019500 | 0.00875 | . | . | . | . |
| 8 | R19 | null | 0.038190 | 0.01000 | . | . | . | . |
| 9 | R02 | null | 0.043817 | 0.01125 | . | . | . | . |
| 10 | R24 | null | 0.080589 | 0.01250 | . | . | . | . |
| 11 | R21 | real | 0.095015 | 0.01375 | . | . | . | . |
| 12 | R32 | null | 0.138730 | 0.01500 | . | . | . | . |
| raw : | 9 rejected, 2 false, 7 of 8 real found | |||||||
| Bonf: | 4 rejected, 0 false, 4 of 8 real found | |||||||
| Holm: | 4 rejected, 0 false, 4 of 8 real found | |||||||
| BH | : | 6 rejected, 0 false, 6 of 8 real found | ||||||
| BY | : | 4 rejected, 0 false, 4 of 8 real found |
Show the full program
# Chapter 2. p21: four rulers applied to one week of forty copy rules.
# Model: 40 rules R01-R40; eight (R03 R07 R12 R18 R21 R26 R33 R38)
# carry a real lift, a one-sided z-score with mean 3.0; the other 32
# are null with mean 0. Seed 20261113. Level 0.05 for every ruler:
# Bonferroni (p <= a/m), Holm step-down (Holm 1979), Benjamini-
# Hochberg step-up (BH 1995), Benjamini-Yekutieli (q / H_m, 2001).
# The data are illustrative; the four procedures are exact.
import math
import random
M, Q, MU, SEED = 40, 0.05, 3.0, 20261113
TRUE = (3, 7, 12, 18, 21, 26, 33, 38)
def week_pvalues():
rng = random.Random(20261113)
out = []
for i in range(1, M + 1):
mu = MU if i in TRUE else 0.0
z = rng.gauss(mu, 1.0)
out.append(("R{:02d}".format(i), 0.5 * math.erfc(z / math.sqrt(2))))
return out
def order(ps):
return sorted(range(len(ps)), key=lambda i: (ps[i], i))
def bonferroni(ps, a):
return {i for i, p in enumerate(ps) if p <= a / len(ps)}
def holm(ps, a):
m, rej = len(ps), set()
for k, i in enumerate(order(ps)):
if ps[i] > a / (m - k):
break
rej.add(i)
return rej
def bh(ps, q):
m, idx, kmax = len(ps), order(ps), 0
for k, i in enumerate(idx, 1):
if ps[i] <= k * q / m:
kmax = k
return set(idx[:kmax])
def by(ps, q):
hm = math.fsum(1.0 / j for j in range(1, len(ps) + 1))
return bh(ps, q / hm)
if __name__ == "__main__":
data = week_pvalues()
ps = [p for _, p in data]
sets = [("Bonf", bonferroni(ps, Q)), ("Holm", holm(ps, Q)),
("BH", bh(ps, Q)), ("BY", by(ps, Q))]
print("rank rule truth p-value k*q/m Bonf Holm BH BY")
for k, i in enumerate(order(ps)[:12], 1):
tag = "real" if i + 1 in TRUE else "null"
marks = "".join(" {:>4}".format("x" if i in s else ".")
for _, s in sets)
print(" {:2d} {} {} {:.6f} {:.5f} {}".format(
k, data[i][0], tag, ps[i], k * Q / M, marks))
raw = {i for i, p in enumerate(ps) if p < Q}
for name, s in [("raw", raw)] + sets:
fd = len([i for i in s if i + 1 not in TRUE])
print("{:4s}: {:2d} rejected, {} false, {} of 8 real found".format(
name, len(s), fd, len(s) - fd))Single source and citation
Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 2: Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Two False Champions among Forty Useless Rules
- Chapter 3 · False Discovery Rate and Family-Wise Error Rate: Two Different Targets
- Chapter 4 · Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules
- Chapter 5 · Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction
- Chapter 6 · Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control
- Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities
- Chapter 8 · Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan