What does False Discovery Rates for Copy Tests conclude, in numbers, about “False Discovery Rate and Family-Wise Error Rate: Two Different Targets”?

Answer: Without correction the share of weeks with a false champion is 0.8722, matching the exact value of 0.8715, with 1.9939 false champions per week on average. Bonferroni and Holm both come in at 0.0469; when everything is useless the two are identical, because Holm's first rung is the Bonferroni bar and a failed first rung stops everything. BH comes in at 0.0481, with 0.0529 false discoveries on average, only slightly above Bonferroni. BY is at just 0.0115, clearly more cautious.

Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0115, 0.0469, 0.0481, 0.0529, 0.8715, 0.8722

m = 40, all nulls true, 20000 weeks
ruleP(V>=1)=FWER=FDRmean V
none0.87221.9939
Bonf0.04690.0476
Holm0.04690.0476
BH0.04810.0529
BY0.01150.0116
exact for none: 0.8715; Bonferroni bound 0.05
Show the full program
# Chapter 3. p31: when every rule is useless, FDR and FWER coincide.
# Model: m = 40 independent one-sided z-tests, all nulls true; 20000
# simulated weeks, seed 20261113. Level 0.05 for each procedure.
# With no true effect, any rejection is a false one and the false
# discovery proportion is 1, so FDR = P(V >= 1) = FWER.
import math
import random

M, Q, REPS, SEED = 40, 0.05, 20000, 20261113


def bh_count(ps, q):
    m, kmax = len(ps), 0
    for k, p in enumerate(sorted(ps), 1):
        if p <= k * q / m:
            kmax = k
    return kmax


def holm_count(ps, a):
    m = len(ps)
    for k, p in enumerate(sorted(ps)):
        if p > a / (m - k):
            return k
    return m


if __name__ == "__main__":
    rng = random.Random(20261113)
    hm = math.fsum(1.0 / j for j in range(1, M + 1))
    names = ("none", "Bonf", "Holm", "BH", "BY")
    hit = {n: [] for n in names}
    vol = {n: [] for n in names}
    for _ in range(REPS):
        ps = [0.5 * math.erfc(rng.gauss(0, 1) / math.sqrt(2))
              for _ in range(M)]
        v = {"none": len([p for p in ps if p < Q]),
             "Bonf": len([p for p in ps if p <= Q / M]),
             "Holm": holm_count(ps, Q),
             "BH": bh_count(ps, Q),
             "BY": bh_count(ps, Q / hm)}
        for n in names:
            hit[n].append(1.0 if v[n] else 0.0)
            vol[n].append(float(v[n]))
    print("m = 40, all nulls true, {} weeks".format(REPS))
    print("rule   P(V>=1)=FWER=FDR   mean V")
    for n in names:
        print("{:5s}  {:.4f}             {:.4f}".format(
            n, math.fsum(hit[n]) / REPS, math.fsum(vol[n]) / REPS))
    print("exact for none: {:.4f}; Bonferroni bound 0.05".format(
        1 - 0.95 ** M))

Single source and citation

Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 3: False Discovery Rate and Family-Wise Error Rate: Two Different Targets. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026307,
  author    = {Fan, Yafu},
  title     = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book