Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites: chapter answers

What does it cost when an owner opens the dashboard every day and stops a test, launching the new page to everyone, as soon as it reads "significant"? How large an improvement can a site with two hundred visitors a day actually see with an A/B test? This book writes both questions as calculations that rely only on the Python standard library and can be reproduced character for character, and states which data would overturn its conclusions. Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized…

DOI pending · monograph page goes live after publication · 5DL-B-217 · 中文版

Illustrative model calculation, not a live quote