Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites: chapter answers
What does it cost when an owner opens the dashboard every day and stops a test, launching the new page to everyone, as soon as it reads "significant"? How large an improvement can a site with two hundred visitors a day actually see with an A/B test? This book writes both questions as calculations that rely only on the Python standard library and can be reproduced character for character, and states which data would overturn its conclusions. Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized…
DOI pending · monograph page goes live after publication · 5DL-B-217 · 中文版
- Chapter 1 · The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant"
Chapter 1 computes exactly, by grid recursion, that looking once a day and stopping when significant gives a false-positive rate of 0.2197 after two weeks and 0.2802 after a month, correcting the founding claim of "a fi… - Chapter 2 · The Ceiling of a Small Site: Power, Minimum Detectable Effects and a Five-and-a-Half-Year Wait
Chapter 2 shows that at a 3% base a 5% relative lift needs 207,938 visitors per arm, 5.7 years at 200 visitors a day, and that a one-month test has power of 0.0609. - Chapter 3 · Where the Inflation Comes From: Random Walks, the Iterated Logarithm and Exaggerated Winners
Chapter 3 explains the inflation with random walks and the law of the iterated logarithm and finds that a peeking small site reports on average 18.3 times the true lift. - Chapter 4 · Wald's Sequential Probability Ratio Test: a Look at Every Single Conversion
Chapter 4 derives Wald's sequential probability ratio test, which at the specified effect needs on average 0.64 of the fixed sample. - Chapter 5 · Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions
Chapter 5 gives group-sequential bars: with five looks, 2.4133 for Pocock and a final 2.0402 for O'Brien-Fleming. - Chapter 6 · Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data
Chapter 6 shows that a mixture sequential test watched daily for a year keeps its false-alarm rate at 0.0280; that a Bayesian "chance to beat control" under a default prior is true only 0.2158 of the time in a world whe… - Chapter 7 · Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea
Chapter 7 examines the standard and the three tests of the Mozi chapter "Against Fate, Part One" and Fisher's lady tasting tea (all four cups right by pure guessing: 0.0143); written as a Bayes formula, the three tests… - Chapter 8 · Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book
Chapter 8 offers a stopping-rule dashboard for thirty sites, the use of the A/B Stopping-Rule Calculator and a pre-registered plan for refuting the book. Marked boxes separate published evidence, runnable Python calcula…
Illustrative model calculation, not a live quote