What does Peeking and Low Traffic in A/B Tests conclude, in numbers, about “Wald's Sequential Probability Ratio Test: a Look at Every Single Conversion”?
Answer: Chapter 4 derives Wald's sequential probability ratio test, which at the specified effect needs on average 0.64 of the fixed sample.
Scope of the numbers: Its sites, traffic, conversion rates and lifts are illustrative parameters; all calculations are generated by its programs; the book is not individualized business, legal or financial advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.64
| Wald SPRT on conversions, alpha 0.05, beta 0.20, base 0.03 | |
| log A | 2.7726 log B = -1.5581 |
| lift 10%: +0.04652 per B conversion, -0.04879 per A | |
| lift ASN|H0 conv ASN|H1 conv H1 visitors fixed 1-sided | |
| 5% | 4,509 6,409 208,417 327,585 |
| 10% | 1,182 1,681 53,360 83,827 |
| 20% | 323 461 13,961 21,919 |
| 50% | 66 95 2,525 3,965 |
| 10% lift: SPRT needs 0.64 of the fixed sample under H1 |
Show the full program
# Chapter 4. p41: Wald's SPRT on conversions only, boundaries and ASN
# Model: traffic is split 50/50. Given that a conversion happened, it
# came from arm B with probability q = pB / (pA + pB): q0 = 1/2 under H0,
# q1 = (1 + d) / (2 + d) under H1 with relative lift d. Each conversion
# adds log(q1/q0) (from B) or log((1-q1)/(1-q0)) (from A) to the log
# likelihood ratio; stop for H1 above log A, for H0 below log B, with
# Wald's A = (1 - beta) / alpha and B = beta / (1 - alpha). Expected
# conversions to a decision use Wald's approximation (Wald 1945). The
# fixed-sample comparison is p21's formula made one-sided at 0.05 (the
# SPRT here is one-sided too), visitors at base 0.03.
import math
from statistics import NormalDist
N01 = NormalDist()
ALPHA, BETA, BASE = 0.05, 0.20, 0.03
def n_fixed(p1, rel, alpha=2 * ALPHA, power=1 - BETA):
p2 = p1 * (1 + rel)
za, zb = N01.inv_cdf(1 - alpha / 2), N01.inv_cdf(power)
pb = (p1 + p2) / 2
num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(
p1 * (1 - p1) + p2 * (1 - p2))
return (num / (p2 - p1)) ** 2
def sprt(d):
q0, q1 = 0.5, (1 + d) / (2 + d)
up, dn = math.log(q1 / q0), math.log((1 - q1) / (1 - q0))
la, lb = math.log((1 - BETA) / ALPHA), math.log(BETA / (1 - ALPHA))
e0 = q0 * up + (1 - q0) * dn
e1 = q1 * up + (1 - q1) * dn
asn0 = ((1 - ALPHA) * lb + ALPHA * la) / e0
asn1 = (BETA * lb + (1 - BETA) * la) / e1
return up, dn, la, lb, asn0, asn1
def main():
print("Wald SPRT on conversions, alpha 0.05, beta 0.20, base 0.03")
up, dn, la, lb, a0, a1 = sprt(0.10)
print("log A = {:.4f} log B = {:.4f}".format(la, lb))
print("lift 10%: +{:.5f} per B conversion, {:.5f} per A".format(
up, dn))
print("lift ASN|H0 conv ASN|H1 conv H1 visitors fixed 1-sided")
for d in (0.05, 0.10, 0.20, 0.50):
up, dn, la, lb, a0, a1 = sprt(d)
conv_rate = BASE * (2 + d) / 2
vis = a1 / conv_rate
fx = 2 * n_fixed(BASE, d)
print("{:4.0%} {:11,.0f} {:11,.0f} {:11,.0f} {:14,.0f}".format(
d, a0, a1, vis, fx))
up, dn, la, lb, a0, a1 = sprt(0.10)
print("10% lift: SPRT needs {:.2f} of the fixed sample under H1".format(
a1 / (BASE * 2.1 / 2) / (2 * n_fixed(BASE, 0.10))))
main()Single source and citation
Fan, Y. (2026). Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites (Version 1.0.0), Chapter 4: Wald's Sequential Probability Ratio Test: a Look at Every Single Conversion. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-217 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026217,
author = {Fan, Yafu},
title = {Peeking-Inflated False Positives in A/B Tests, Sequential Probability Ratio Tests and Minimum Samples for Low-Traffic Sites},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Price of Peeking: an Exact Account of "Look Daily, Stop When Significant"
- Chapter 2 · The Ceiling of a Small Site: Power, Minimum Detectable Effects and a Five-and-a-Half-Year Wait
- Chapter 3 · Where the Inflation Comes From: Random Walks, the Iterated Logarithm and Exaggerated Winners
- Chapter 5 · Looking Once a Week: Group-Sequential Designs and Alpha-Spending Functions
- Chapter 6 · Always-Valid Inference, the Bayesian "Chance to Beat Control" and the Limits of Pre-Experiment Data
- Chapter 7 · Eastern and Western Sources Examined: Mozi's Three Tests and Fisher's Cup of Tea
- Chapter 8 · Closing the Loop: a Stopping-Rule Dashboard for Thirty Sites, a Minimum-Sample Calculator and a Plan to Refute This Book