What does False Discovery Rates for Copy Tests conclude, in numbers, about “Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules”?
Answer: The Bonferroni column barely moves, staying between 0.298 and 0.309, because its bar depends only on the number of rules and not on how many are real. Holm rises slightly, from 0.309 to 0.325: the more real rules, the more its bar loosens after the first rungs are rejected. BH's power rises fastest with the share of real rules: 0.347 at 0.05, 0.483 at 0.20, 0.647 at 0.50, 0.706 at 0.70. With more real rules the last passing rung moves further up the ladder and every rung's bar is wider. Meanwhile BH's false discovery rate stays below 0.05 throughout, falling from 0.0460 to 0.0145, in line with the previous chapter's "share of useless rules times q".
Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0145, 0.0460, 0.05, 0.20, 0.298, 0.309
| pi1 m1 raw Bonf Holm BH BY FDR BH FDR raw | |
| 0.05 | 2 0.801 0.309 0.309 0.347 0.198 0.0460 0.4777 |
| 0.10 | 4 0.807 0.308 0.310 0.402 0.213 0.0440 0.3243 |
| 0.20 | 8 0.803 0.301 0.306 0.483 0.257 0.0395 0.1850 |
| 0.30 12 | 0.803 0.301 0.311 0.555 0.300 0.0361 0.1188 |
| 0.50 20 | 0.803 0.300 0.318 0.647 0.369 0.0254 0.0564 |
| 0.70 28 | 0.804 0.298 0.325 0.706 0.424 0.0145 0.0246 |
Show the full program
# Chapter 4. p41: power as the share of real rules grows.
# Model: m = 40 independent one-sided z-tests; a share pi1 of rules
# has z-mean 2.5, the rest are null. Average power = expected share
# of real rules found. Level 0.05 for all; 5000 weeks per row, seed
# 20261113. Parameters illustrative.
import math
import random
M, Q, MU, REPS, SEED = 40, 0.05, 2.5, 5000, 20261113
def bh_k(sp, q):
m, kmax = len(sp), 0
for k, p in enumerate(sp, 1):
if p <= k * q / m:
kmax = k
return kmax
def holm_k(sp, a):
m = len(sp)
for k, p in enumerate(sp):
if p > a / (m - k):
return k
return m
def week(rng, m1):
out = []
for i in range(M):
z = rng.gauss(MU if i < m1 else 0.0, 1.0)
out.append((0.5 * math.erfc(z / math.sqrt(2)), i))
out.sort()
return out
if __name__ == "__main__":
rng = random.Random(20261113)
hm = math.fsum(1.0 / j for j in range(1, M + 1))
print(" pi1 m1 raw Bonf Holm BH BY FDR BH FDR raw")
for m1 in (2, 4, 8, 12, 20, 28):
pw = {k: [] for k in ("raw", "bon", "holm", "bh", "by")}
fdr_bh, fdr_raw = [], []
for _ in range(REPS):
w = week(rng, m1)
sp = [p for p, _ in w]
ks = {"raw": len([p for p in sp if p < Q]),
"bon": len([p for p in sp if p <= Q / M]),
"holm": holm_k(sp, Q), "bh": bh_k(sp, Q),
"by": bh_k(sp, Q / hm)}
for name, k in ks.items():
real = len([i for _, i in w[:k] if i < m1])
pw[name].append(real / m1)
if name == "bh":
fdr_bh.append((k - real) / max(k, 1))
if name == "raw":
fdr_raw.append((k - real) / max(k, 1))
cells = " ".join("{:.3f}".format(math.fsum(pw[k]) / REPS)
for k in ("raw", "bon", "holm", "bh", "by"))
print(" {:.2f} {:2d} {} {:.4f} {:.4f}".format(
m1 / M, m1, cells, math.fsum(fdr_bh) / REPS,
math.fsum(fdr_raw) / REPS))Single source and citation
Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 4: Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Two False Champions among Forty Useless Rules
- Chapter 2 · Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction
- Chapter 3 · False Discovery Rate and Family-Wise Error Rate: Two Different Targets
- Chapter 5 · Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction
- Chapter 6 · Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control
- Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities
- Chapter 8 · Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan