Multiple Comparisons and False-Discovery-Rate Control: the Benjamini-Hochberg Step-Up for Testing Dozens of Copy Rules at Once: chapter answers
Owners often test dozens of copy rules at once, judge each at p below 0.05, and trust that the champions so chosen are broadly reliable. This book overturns that belief with exact probabilities: when forty rules that all do nothing are tested together, two false champions appear in an average week, and at least one appears with probability 0.8715; in simulation the best false champion of each week claims a relative lift of 34.2 % on average while its true lift is zero. The project blueprint attributed the claim to Benjamini and Hochberg's paper of 1995; after checking, the book corrects the b…
DOI pending · monograph page goes live after publication · 5DL-B-307 · 中文版
- Chapter 1 · Two False Champions among Forty Useless Rules
The shape matters more. The chance of no false champion at all is only 0.1285, so out of a hundred such weeks only about thirteen give a clean leaderboard, and the chance of at least one is 0.8715. Exactly one and exact… - Chapter 2 · Four Rulers: Bonferroni, Holm, the BH Step-Up and the BY Correction
Rows five and six show where the extra two come from. R18 has p-value 0.001665, above the Bonferroni bar of 0.00125, so Bonferroni turns it down; but it sits fifth, and BH sets the fifth rung at 0.00625, which it clears… - Chapter 3 · False Discovery Rate and Family-Wise Error Rate: Two Different Targets
Without correction the share of weeks with a false champion is 0.8722, matching the exact value of 0.8715, with 1.9939 false champions per week on average. Bonferroni and Holm both come in at 0.0469; when everything is… - Chapter 4 · Power: Sweeping the Share of Real Effects, the Effect Size and the Number of Rules
The Bonferroni column barely moves, staying between 0.298 and 0.309, because its bar depends only on the number of rules and not on how many are real. Holm rises slightly, from 0.309 to 0.325: the more real rules, the m… - Chapter 5 · Correlated Tests: Shared Controls, Positive Regression Dependency and the BY Correction
Chapter 5 treats positive correlation from a shared control, two-sided tests and negative correlation, and constructs exactly a counterexample in which BH's false discovery rate reaches 0.0750, showing why the BY factor… - Chapter 6 · Sample Size: Visitors per Rule, and How to Split Traffic around a Shared Control
Chapter 6 converts correction into visitors per arm: 10,960 without correction, 26,476 under Bonferroni, and about 18,555 under BH when a fifth of rules are real. - Chapter 7 · Eastern and Western Sources: Three Men Make a Tiger, and Bonferroni's Inequalities
Chapter 7 examines "three men speaking make a tiger" from the Strategies of the Warring States and Bonferroni's inequalities of 1936: three independent reports push the posterior to 0.4219, while reports copied from one… - Chapter 8 · Closing the Loop: Multiple-Rule Correction for the Weekly Review Radar, and a Falsification Plan
Chapter 8 adds multiple-rule correction to the weekly review radar, where e-BH lowers the false discovery rate from 0.0896 to 0.0023, and presents the distilled FDR Step-Up Ladder and a pre-registered falsification plan…
Illustrative model calculation, not a live quote