What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea”?
Answer: Chapter 7 examines the standard and the three tests of the Mozi chapter "Against Fate, Part One" and Fisher's lady tasting tea (all four cups right by pure guessing: 0.0143); written as a Bayes formula, the three tests show that only 0.133 of the changes a peeking small site launches are real.
Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. printed values that match the answer: 0.133
| ship B when 'significant and ahead'; per 100 tests | ||||
| regime | pi | real wins | null ships | share real |
| big, planned | 0.10 | 8.0 | 2.2 | 0.780 |
| big, planned | 0.30 | 24.0 | 1.8 | 0.932 |
| big, peeking | 0.10 | 8.9 | 12.6 | 0.414 |
| big, peeking | 0.30 | 26.8 | 9.8 | 0.732 |
| small, planned | 0.10 | 0.5 | 2.2 | 0.175 |
| small, planned | 0.30 | 1.4 | 1.8 | 0.450 |
| small, peeking | 0.10 | 1.9 | 12.3 | 0.133 |
| small, peeking | 0.30 | 5.7 | 9.6 | 0.371 |
Show the full program
# Chapter 7. p72: Mozi's three tests written as one Bayes formula
# Model: "basis" (ben) = the prior share pi of tested ideas that truly
# help, read from the shop's own history; "verification" (yuan) = what
# the eyes and ears of the data say, the chance a real win is shipped
# (power, right sign) and a null one is shipped (false ship rate);
# "application" (yong) = what 100 tests actually deliver.
# Rates are copied from earlier programs: planned big site alpha/2 =
# 0.025 with power 0.80 (p21); peeking for 28 days 0.2802/2 = 0.1401
# (p11) with power 0.8925 (p61 naive, H1); small site planned 0.025
# with right-sign power 0.0477 (p23); small site peeking 0.2743/2
# (p12) with right-sign power 0.2855 * (1 - 0.339) (p32). pi values
# are illustrative.
REGIMES = (
("big, planned", 0.025, 0.80),
("big, peeking", 0.2802 / 2, 0.8925),
("small, planned", 0.025, 0.0477),
("small, peeking", 0.2743 / 2, 0.2855 * (1 - 0.339)),
)
def main():
print("ship B when 'significant and ahead'; per 100 tests")
print("regime pi real wins null ships share real")
for name, fp, pw in REGIMES:
for pi in (0.10, 0.30):
real = 100 * pi * pw
null = 100 * (1 - pi) * fp
print("{:14} {:.2f} {:9.1f} {:10.1f} {:10.3f}".format(
name, pi, real, null, real / (real + null)))
main()Single source and citation
Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 7: Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant"
- Chapter 2 · The Ceiling of a Small Site: Power, Minimum Detectable Effects and a Five-and-a-Half-Year Wait
- Chapter 3 · Where the Inflation Comes From: Random Walks, the Iterated Logarithm and Exaggerated Winners
- Chapter 4 · Wald's Sequential Probability Ratio Test: a Look at Every Single Conversion
- Chapter 5 · Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions
- Chapter 6 · Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data
- Chapter 8 · Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book