What does False Discovery Rates for Copy Tests conclude, in numbers, about “Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules”?

Answer: The Bonferroni column barely moves, staying between 0.298 and 0.309, because its bar depends only on the number of rules and not on how many are real. Holm rises slightly, from 0.309 to 0.325: the more real rules, the more its bar loosens after the first rungs are rejected. BH's power rises fastest with the share of real rules: 0.347 at 0.05, 0.483 at 0.20, 0.647 at 0.50, 0.706 at 0.70. With more real rules the last passing rung moves further up the ladder and every rung's bar is wider. Meanwhile BH's false discovery rate stays below 0.05 throughout, falling from 0.0460 to 0.0145, in line with the previous chapter's "share of useless rules times q".

Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0145, 0.0460, 0.05, 0.20, 0.298, 0.309

pi1 m1 raw Bonf Holm BH BY FDR BH FDR raw
0.052 0.801 0.309 0.309 0.347 0.198 0.0460 0.4777
0.104 0.807 0.308 0.310 0.402 0.213 0.0440 0.3243
0.208 0.803 0.301 0.306 0.483 0.257 0.0395 0.1850
0.30 120.803 0.301 0.311 0.555 0.300 0.0361 0.1188
0.50 200.803 0.300 0.318 0.647 0.369 0.0254 0.0564
0.70 280.804 0.298 0.325 0.706 0.424 0.0145 0.0246
Show the full program
# Chapter 4. p41: power as the share of real rules grows.
# Model: m = 40 independent one-sided z-tests; a share pi1 of rules
# has z-mean 2.5, the rest are null. Average power = expected share
# of real rules found. Level 0.05 for all; 5000 weeks per row, seed
# 20261113. Parameters illustrative.
import math
import random

M, Q, MU, REPS, SEED = 40, 0.05, 2.5, 5000, 20261113


def bh_k(sp, q):
    m, kmax = len(sp), 0
    for k, p in enumerate(sp, 1):
        if p <= k * q / m:
            kmax = k
    return kmax


def holm_k(sp, a):
    m = len(sp)
    for k, p in enumerate(sp):
        if p > a / (m - k):
            return k
    return m


def week(rng, m1):
    out = []
    for i in range(M):
        z = rng.gauss(MU if i < m1 else 0.0, 1.0)
        out.append((0.5 * math.erfc(z / math.sqrt(2)), i))
    out.sort()
    return out


if __name__ == "__main__":
    rng = random.Random(20261113)
    hm = math.fsum(1.0 / j for j in range(1, M + 1))
    print(" pi1  m1   raw    Bonf   Holm   BH     BY     FDR BH  FDR raw")
    for m1 in (2, 4, 8, 12, 20, 28):
        pw = {k: [] for k in ("raw", "bon", "holm", "bh", "by")}
        fdr_bh, fdr_raw = [], []
        for _ in range(REPS):
            w = week(rng, m1)
            sp = [p for p, _ in w]
            ks = {"raw": len([p for p in sp if p < Q]),
                  "bon": len([p for p in sp if p <= Q / M]),
                  "holm": holm_k(sp, Q), "bh": bh_k(sp, Q),
                  "by": bh_k(sp, Q / hm)}
            for name, k in ks.items():
                real = len([i for _, i in w[:k] if i < m1])
                pw[name].append(real / m1)
                if name == "bh":
                    fdr_bh.append((k - real) / max(k, 1))
                if name == "raw":
                    fdr_raw.append((k - real) / max(k, 1))
        cells = " ".join("{:.3f}".format(math.fsum(pw[k]) / REPS)
                         for k in ("raw", "bon", "holm", "bh", "by"))
        print(" {:.2f} {:2d}   {}  {:.4f}  {:.4f}".format(
            m1 / M, m1, cells, math.fsum(fdr_bh) / REPS,
            math.fsum(fdr_raw) / REPS))

Single source and citation

Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 4: Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026307,
  author    = {Fan, Yafu},
  title     = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book