What does False Discovery Rates for Copy Tests conclude, in numbers, about “Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction”?
Answer: Chapter 5 treats positive correlation from a shared control, two-sided tests and negative correlation, and constructs exactly a counterexample in which BH's false discovery rate reaches 0.0750, showing why the BY factor is needed.
Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0750
| check: p1 takes every grid value once (uniform): True | |
| check: p2 takes every grid value once (uniform): True | |
| all nulls true, so FDR = P(any rejection) | |
| BH at q | 0.05: FDR = 3/40 = 0.0750 |
| bound q * H_2 | 0.0750 |
| BY at q / H_2: FDR | 167/4000 = 0.0418 |
| H_m for m | 2, 10, 40, 100: |
| m | 2 H_m = 1.5000 BY level = 0.03333 |
| m | 10 H_m = 2.9290 BY level = 0.01707 |
| m | 40 H_m = 4.2785 BY level = 0.01169 |
| m | 100 H_m = 5.1874 BY level = 0.00964 |
Show the full program
# Chapter 5. p52: under arbitrary dependence BH can exceed q, and the
# Benjamini-Yekutieli factor H_m is exactly what it can cost.
# Model: two true-null rules whose p-values are built from one
# uniform U so that each p is exactly uniform but the pair is
# arranged against BH (negative dependence on the low end):
# U in [0, q/2): p1 = U, p2 = U + q
# U in [q/2, q): p1 = U + q/2, p2 = U - q/2
# U in [q, 3q/2): p1 = p2 = U - q/2
# U in [3q/2, 1): p1 = p2 = U
# The FDR is computed exactly over a grid of U cells with Fraction,
# and the marginals are checked to be uniform. q = 0.05 illustrative.
import operator
from fractions import Fraction
from functools import reduce
Q = Fraction(1, 20)
CELLS = 4000
def pair(u, q):
if u < q / 2:
return u, u + q
if u < q:
return u + q / 2, u - q / 2
if u < 3 * q / 2:
return u - q / 2, u - q / 2
return u, u
def bh_rejects(ps, q):
m = len(ps)
srt = sorted(ps)
kmax = 0
for k, p in enumerate(srt, 1):
if p <= k * q / m:
kmax = k
return kmax
def fdr(level):
hits = 0
for c in range(CELLS):
u = Fraction(2 * c + 1, 2 * CELLS)
if bh_rejects(pair(u, Q), level) > 0:
hits += 1
return Fraction(hits, CELLS)
if __name__ == "__main__":
grid = [Fraction(2 * c + 1, 2 * CELLS) for c in range(CELLS)]
for j in (0, 1):
same = sorted(pair(u, Q)[j] for u in grid) == grid
print("check: p{} takes every grid value once (uniform): {}"
.format(j + 1, same))
h2 = 1 + Fraction(1, 2)
print("all nulls true, so FDR = P(any rejection)")
f = fdr(Q)
print("BH at q = 0.05: FDR = {} = {:.4f}".format(f, float(f)))
print("bound q * H_2 = {:.4f}".format(float(Q * h2)))
f = fdr(Q / h2)
print("BY at q / H_2: FDR = {} = {:.4f}".format(f, float(f)))
print("H_m for m = 2, 10, 40, 100:")
for m in (2, 10, 40, 100):
hm = reduce(operator.add,
(Fraction(1, j) for j in range(1, m + 1)))
print(" m = {:3d} H_m = {:.4f} BY level = {:.5f}".format(
m, float(hm), float(Q / hm)))Single source and citation
Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 5: Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Two False Champions among Forty Useless Rules
- Chapter 2 · Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction
- Chapter 3 · False Discovery Rate and Family-Wise Error Rate: Two Different Targets
- Chapter 4 · Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules
- Chapter 6 · Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control
- Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities
- Chapter 8 · Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan