What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea”?

Answer: Chapter 7 examines the standard and the three tests of the Mozi chapter "Against Fate, Part One" and Fisher's lady tasting tea (all four cups right by pure guessing: 0.0143); written as a Bayes formula, the three tests show that only 0.133 of the changes a peeking small site launches are real.

Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. printed values that match the answer: 0.133

ship B when 'significant and ahead'; per 100 tests
regimepireal winsnull shipsshare real
big, planned0.108.02.20.780
big, planned0.3024.01.80.932
big, peeking0.108.912.60.414
big, peeking0.3026.89.80.732
small, planned0.100.52.20.175
small, planned0.301.41.80.450
small, peeking0.101.912.30.133
small, peeking0.305.79.60.371
Show the full program
# Chapter 7. p72: Mozi's three tests written as one Bayes formula
# Model: "basis" (ben) = the prior share pi of tested ideas that truly
# help, read from the shop's own history; "verification" (yuan) = what
# the eyes and ears of the data say, the chance a real win is shipped
# (power, right sign) and a null one is shipped (false ship rate);
# "application" (yong) = what 100 tests actually deliver.
# Rates are copied from earlier programs: planned big site alpha/2 =
# 0.025 with power 0.80 (p21); peeking for 28 days 0.2802/2 = 0.1401
# (p11) with power 0.8925 (p61 naive, H1); small site planned 0.025
# with right-sign power 0.0477 (p23); small site peeking 0.2743/2
# (p12) with right-sign power 0.2855 * (1 - 0.339) (p32). pi values
# are illustrative.
REGIMES = (
    ("big, planned", 0.025, 0.80),
    ("big, peeking", 0.2802 / 2, 0.8925),
    ("small, planned", 0.025, 0.0477),
    ("small, peeking", 0.2743 / 2, 0.2855 * (1 - 0.339)),
)


def main():
    print("ship B when 'significant and ahead'; per 100 tests")
    print("regime          pi    real wins  null ships  share real")
    for name, fp, pw in REGIMES:
        for pi in (0.10, 0.30):
            real = 100 * pi * pw
            null = 100 * (1 - pi) * fp
            print("{:14}  {:.2f}  {:9.1f}  {:10.1f}  {:10.3f}".format(
                name, pi, real, null, real / (real + null)))


main()

Single source and citation

Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 7: Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026217,
  author    = {Fan, Yafu},
  title     = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book