What does Completion Hazards, Feedback Loops and Funnels conclude, in numbers, about “Evidence: A/B Tests, Survivorship and Early Forecasts”?

Answer: Chapter 6 shows that an A/B test of paid conversion needs 42,691 viewers per arm, that peeking declares 0.239 of null tests significant, that studying only top accounts recommends a shock hook with lower mean revenue (RM 1,250 against RM 1,647), and that forecasting a year from the first three days falls short by 0.602.

Scope of the numbers: Marked boxes separate evidence, runnable Python programs, classical sources and hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.239, 42691

   outcome            p1      p2     viewers per arm
   5 s retention     0.600   0.630           4,126
   completion        0.200   0.220           6,507
   paid conversion   0.010   0.012          42,691
   peeking daily for 20 days with no true difference:
   share of tests declared 'significant' 0.239
Reading: an opening that lifts early retention by three
points can be tested in a few days of ordinary traffic; a
change in paid conversion needs tens of thousands of viewers
per arm. Checking every day and stopping at the first good
number turns a 0.05 test into roughly a one in four false
alarm.
Show the full program
# Chapter 6. p11: How many impressions does a hook A/B test need?
# Two openings of the same video are shown to random halves of the
# audience. Per arm, for a two-sided test at 0.05 with power 0.80:
#   n = (1.96 + 0.8416)^2 (p1 (1 - p1) + p2 (1 - p2)) / (p2 - p1)^2.
# Outcomes: 5-second retention 0.60 vs 0.63; completion 0.20 vs 0.22;
# paid conversion 0.010 vs 0.012 per viewer. Then the cost of peeking:
# under no true difference (both arms at 0.20), a creator checks the
# running z statistic after each of 20 daily batches of 500 viewers
# per arm and stops at the first |z| > 1.96. 4,000 simulated tests.
import math
import random

Z = 1.96 + 0.8416

def n_arm(p1, p2):
    return math.ceil(Z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) /
                     (p2 - p1) ** 2)

if __name__ == "__main__":
    print("   outcome            p1      p2     viewers per arm")
    for name, p1, p2 in (("5 s retention", 0.60, 0.63),
                         ("completion", 0.20, 0.22),
                         ("paid conversion", 0.010, 0.012)):
        print("   {:16s} {:6.3f} {:7.3f} {:15,d}".format(
            name, p1, p2, n_arm(p1, p2)))
    rng = random.Random(20)
    stops = 0
    for _ in range(2000):
        a = b = n = 0
        for day in range(20):
            for _ in range(500):
                a += rng.random() < 0.2
                b += rng.random() < 0.2
            n += 500
            pa, pb = a / n, b / n
            pool = (a + b) / (2 * n)
            se = math.sqrt(pool * (1 - pool) * 2 / n)
            if se > 0 and abs(pa - pb) / se > 1.96:
                stops += 1
                break
    print("\n   peeking daily for 20 days with no true difference:")
    print("   share of tests declared 'significant' {:.3f}".format(
        stops / 2000))
    print("\nReading: an opening that lifts early retention by three")
    print("points can be tested in a few days of ordinary traffic; a")
    print("change in paid conversion needs tens of thousands of viewers")
    print("per arm. Checking every day and stopping at the first good")
    print("number turns a 0.05 test into roughly a one in four false")
    print("alarm.")

Single source and citation

Fan, Y. (2026). Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels (Version 1.0.0), Chapter 6: Evidence: A/B Tests, Survivorship and Early Forecasts. 5D Library. https://doi.org/10.5281/zenodo.24090213

DOI: 10.5281/zenodo.24090213 · Read the monograph · 5DL-B-081 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026081,
  author    = {Fan, Yafu},
  title     = {Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090213},
  url       = {https://doi.org/10.5281/zenodo.24090213}
}

Other chapter answers in this book