What does False Discovery Rates for Copy Tests conclude, in numbers, about “Two False Champions among Forty Useless Rules”?
Answer: The shape matters more. The chance of no false champion at all is only 0.1285, so out of a hundred such weeks only about thirteen give a clean leaderboard, and the chance of at least one is 0.8715. Exactly one and exactly two occur with probabilities 0.2706 and 0.2777, and two is the most likely count. Three or more happens with probability 0.3233, and even five or more still has 0.0480, close to one week in twenty.
Scope of the numbers: Conversion rates, visitor counts and numbers of rules are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0480, 0.1285, 0.2706, 0.2777, 0.3233, 0.8715
| m = 40 true nulls, alpha = 0.05 each | ||
| E[false champions] = 2.000 (exact 2) | ||
| k | P(V = k) | P(V >= k) |
| 0 | 0.1285 | 1.0000 |
| 1 | 0.2706 | 0.8715 |
| 2 | 0.2777 | 0.6009 |
| 3 | 0.1851 | 0.3233 |
| 4 | 0.0901 | 0.1381 |
| 5 | 0.0342 | 0.0480 |
| 6 | 0.0105 | 0.0139 |
| 7 | 0.0027 | 0.0034 |
| most likely count = 2 | ||
| P(no false champion) = 0.95^40 = 0.1285 | ||
| P(at least one) | = 0.8715 |
Show the full program
# Chapter 1. p11: forty useless copy rules, each tested at p < 0.05.
# Model: m = 40 independent one-sided tests whose null hypotheses are
# all true; each p-value is uniform, so the number of "champions" V
# (p < alpha) is Binomial(40, 0.05). Exact arithmetic with Fraction.
# m and alpha are illustrative; the binomial law is exact.
import math
import operator
from fractions import Fraction
from functools import reduce
M = 40
ALPHA = Fraction(1, 20)
def pmf(k):
return math.comb(M, k) * ALPHA ** k * (1 - ALPHA) ** (M - k)
if __name__ == "__main__":
probs = [pmf(k) for k in range(M + 1)]
# exact Fraction totals via reduce (no float rounding)
mean = reduce(operator.add, (k * p for k, p in enumerate(probs)))
print("m = {} true nulls, alpha = {:.2f} each".format(M, float(ALPHA)))
print("E[false champions] = {:.3f} (exact {})".format(
float(mean), mean))
print(" k P(V = k) P(V >= k)")
for k in range(0, 8):
tail = reduce(operator.add, probs[k:])
print(" {:d} {:.4f} {:.4f}".format(
k, float(probs[k]), float(tail)))
mode = max(range(M + 1), key=lambda k: (probs[k], -k))
print("most likely count = {}".format(mode))
print("P(no false champion) = 0.95^40 = {:.4f}".format(
float(probs[0])))
print("P(at least one) = {:.4f}".format(float(1 - probs[0])))Single source and citation
Fan, Y. (2026). Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once (Version 1.0.0), Chapter 1: Two False Champions among Forty Useless Rules. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-307 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026307,
author = {Fan, Yafu},
title = {Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 2 · Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction
- Chapter 3 · False Discovery Rate and Family-Wise Error Rate: Two Different Targets
- Chapter 4 · Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules
- Chapter 5 · Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction
- Chapter 6 · Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control
- Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities
- Chapter 8 · Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan