What does Completion Hazards, Feedback Loops and Funnels conclude, in numbers, about “Evidence: A/B Tests, Survivorship and Early Forecasts”?
Answer: Chapter 6 shows that an A/B test of paid conversion needs 42,691 viewers per arm, that peeking declares 0.239 of null tests significant, that studying only top accounts recommends a shock hook with lower mean revenue (RM 1,250 against RM 1,647), and that forecasting a year from the first three days falls short by 0.602.
Scope of the numbers: Marked boxes separate evidence, runnable Python programs, classical sources and hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.239, 42691
outcome p1 p2 viewers per arm 5 s retention 0.600 0.630 4,126 completion 0.200 0.220 6,507 paid conversion 0.010 0.012 42,691 peeking daily for 20 days with no true difference: share of tests declared 'significant' 0.239 Reading: an opening that lifts early retention by three points can be tested in a few days of ordinary traffic; a change in paid conversion needs tens of thousands of viewers per arm. Checking every day and stopping at the first good number turns a 0.05 test into roughly a one in four false alarm.
Show the full program
# Chapter 6. p11: How many impressions does a hook A/B test need?
# Two openings of the same video are shown to random halves of the
# audience. Per arm, for a two-sided test at 0.05 with power 0.80:
# n = (1.96 + 0.8416)^2 (p1 (1 - p1) + p2 (1 - p2)) / (p2 - p1)^2.
# Outcomes: 5-second retention 0.60 vs 0.63; completion 0.20 vs 0.22;
# paid conversion 0.010 vs 0.012 per viewer. Then the cost of peeking:
# under no true difference (both arms at 0.20), a creator checks the
# running z statistic after each of 20 daily batches of 500 viewers
# per arm and stops at the first |z| > 1.96. 4,000 simulated tests.
import math
import random
Z = 1.96 + 0.8416
def n_arm(p1, p2):
return math.ceil(Z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) /
(p2 - p1) ** 2)
if __name__ == "__main__":
print(" outcome p1 p2 viewers per arm")
for name, p1, p2 in (("5 s retention", 0.60, 0.63),
("completion", 0.20, 0.22),
("paid conversion", 0.010, 0.012)):
print(" {:16s} {:6.3f} {:7.3f} {:15,d}".format(
name, p1, p2, n_arm(p1, p2)))
rng = random.Random(20)
stops = 0
for _ in range(2000):
a = b = n = 0
for day in range(20):
for _ in range(500):
a += rng.random() < 0.2
b += rng.random() < 0.2
n += 500
pa, pb = a / n, b / n
pool = (a + b) / (2 * n)
se = math.sqrt(pool * (1 - pool) * 2 / n)
if se > 0 and abs(pa - pb) / se > 1.96:
stops += 1
break
print("\n peeking daily for 20 days with no true difference:")
print(" share of tests declared 'significant' {:.3f}".format(
stops / 2000))
print("\nReading: an opening that lifts early retention by three")
print("points can be tested in a few days of ordinary traffic; a")
print("change in paid conversion needs tens of thousands of viewers")
print("per arm. Checking every day and stopping at the first good")
print("number turns a 0.05 test into roughly a one in four false")
print("alarm.")Single source and citation
Fan, Y. (2026). Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels (Version 1.0.0), Chapter 6: Evidence: A/B Tests, Survivorship and Early Forecasts. 5D Library. https://doi.org/10.5281/zenodo.24090213
DOI: 10.5281/zenodo.24090213 · Read the monograph · 5DL-B-081 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026081,
author = {Fan, Yafu},
title = {Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090213},
url = {https://doi.org/10.5281/zenodo.24090213}
}Other chapter answers in this book
- Chapter 1 · The Kinetics of Swiping: a Weibull Hazard
- Chapter 2 · Tiered Traffic Pools and the First Gate
- Chapter 3 · Feedback Loops, the Matthew Effect and Gameable Metrics
- Chapter 4 · Reverse Selection by Hook Breadth and the Timing of the Offer
- Chapter 5 · From Views to Cash: Lags, Heavy Tails and Linked Bans
- Chapter 7 · Gauging Desire, Opening Lines and the Casket Without the Pearl
- Chapter 8 · A Ten-Flag Audit and a Pre-Registered Experiment Randomised by Video