5D Library › Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels · 中文

Chapter 6. Evidence: A/B Tests, Survivorship and Early Forecasts

How many viewers does a hook A/B test need?

Online controlled experiments are the standard way to test whether a change works (Kohavi et al. 2020). Two openings of the same video go to random halves of the audience, and a proportion is compared. The program computes the viewers per arm needed for three outcomes at two-sided 0.05 and power 0.80, and simulates the habit of “look every day and stop as soon as it is significant”.

Calculation

# Chapter 6. p11: How many impressions does a hook A/B test need?
# Two openings of the same video are shown to random halves of the
# audience. Per arm, for a two-sided test at 0.05 with power 0.80:
#   n = (1.96 + 0.8416)^2 (p1 (1 - p1) + p2 (1 - p2)) / (p2 - p1)^2.
# Outcomes: 5-second retention 0.60 vs 0.63; completion 0.20 vs 0.22;
# paid conversion 0.010 vs 0.012 per viewer. Then the cost of peeking:
# under no true difference (both arms at 0.20), a creator checks the
# running z statistic after each of 20 daily batches of 500 viewers
# per arm and stops at the first |z| > 1.96. 4,000 simulated tests.
import math
import random

Z = 1.96 + 0.8416

def n_arm(p1, p2):
    return math.ceil(Z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) /
                     (p2 - p1) ** 2)

if __name__ == "__main__":
    print("   outcome            p1      p2     viewers per arm")
    for name, p1, p2 in (("5 s retention", 0.60, 0.63),
                         ("completion", 0.20, 0.22),
                         ("paid conversion", 0.010, 0.012)):
        print("   {:16s} {:6.3f} {:7.3f} {:15,d}".format(
            name, p1, p2, n_arm(p1, p2)))
    rng = random.Random(20)
    stops = 0
    for _ in range(2000):
        a = b = n = 0
        for day in range(20):
            for _ in range(500):
                a += rng.random() < 0.2
                b += rng.random() < 0.2
            n += 500
            pa, pb = a / n, b / n
            pool = (a + b) / (2 * n)
            se = math.sqrt(pool * (1 - pool) * 2 / n)
            if se > 0 and abs(pa - pb) / se > 1.96:
                stops += 1
                break
    print("\n   peeking daily for 20 days with no true difference:")
    print("   share of tests declared 'significant' {:.3f}".format(
        stops / 2000))
    print("\nReading: an opening that lifts early retention by three")
    print("points can be tested in a few days of ordinary traffic; a")
    print("change in paid conversion needs tens of thousands of viewers")
    print("per arm. Checking every day and stopping at the first good")
    print("number turns a 0.05 test into roughly a one in four false")
    print("alarm.")
#    outcome            p1      p2     viewers per arm
#    5 s retention     0.600   0.630           4,126
#    completion        0.200   0.220           6,507
#    paid conversion   0.010   0.012          42,691
#
#    peeking daily for 20 days with no true difference:
#    share of tests declared 'significant' 0.239
#
# Reading: an opening that lifts early retention by three
# points can be tested in a few days of ordinary traffic; a
# change in paid conversion needs tens of thousands of viewers
# per arm. Checking every day and stopping at the first good
# number turns a 0.05 test into roughly a one in four false
# alarm.

Raising 5-second retention from 0.60 to 0.63 needs 4,126 viewers per arm, a few days of ordinary traffic; completion from 0.20 to 0.22 needs 6,507; paid conversion from 0.010 to 0.012 needs 42,691. When the two arms do not differ at all, looking once a day for 20 days and stopping at the first significant result declares 0.239 of tests “significant”, about one in four rather than the nominal 0.05. An analyst who decides when to stop and what to compare after seeing the data is walking one of the forking paths (Gelman and Loken 2014).

Learning only from the top accounts

“The viral accounts all used this hook” is the most common source of advice. Denrell showed that learning only from successes systematically overrates risky strategies, because the failures of a risky strategy are missing from the sample (Denrell 2003). The program takes 10,000 creators, half using a “safe” hook and half a “shock” hook; the latter has lower typical views, wider spread, and audiences that buy at half the rate.

Calculation

# Chapter 6. p12: Learning from the top accounts: survivorship bias.
# 10,000 creators, half using a "safe" hook style and half a "shock"
# style. Monthly views are lognormal: safe log-mean 8.0, log-sd 0.5;
# shock log-mean 7.8, log-sd 1.2 (lower typical result, wider spread).
# Paid conversion per view: 0.010 for safe, 0.005 for shock (shock
# audiences buy less). Revenue = views x conversion x RM 49. Observers
# see only views, and study the top 1 % of accounts by views.
import math
import random

if __name__ == "__main__":
    rng = random.Random(2003)
    pop = []
    for i in range(10000):
        style = "safe" if i % 2 == 0 else "shock"
        mu, sd, cv = (8.0, 0.5, 0.010) if style == "safe" else \
                     (7.8, 1.2, 0.005)
        v = math.exp(rng.gauss(mu, sd))
        pop.append((v, style, v * cv * 49))
    top = sorted(pop, reverse=True)[:100]
    share = sum(1 for p in top if p[1] == "shock") / len(top)
    print("   share of shock style among the top 1 % by views: {:.2f}"
          .format(share))
    for style in ("safe", "shock"):
        g = [p for p in pop if p[1] == style]
        rev = sorted(p[2] for p in g)
        print("   {:5s}: mean views {:7,.0f}  mean revenue RM {:6,.0f}"
              "  median RM {:5,.0f}".format(
                  style, sum(p[0] for p in g) / len(g),
                  sum(rev) / len(rev), rev[len(rev) // 2]))
    rs = sorted([p for p in pop if p[1] == "safe"], key=lambda p: p[2])
    rk = sorted([p for p in pop if p[1] == "shock"], key=lambda p: p[2])
    beat = sum(1 for a, b in zip(rs, rk) if b[2] > a[2]) / len(rs)
    print("   quantile by quantile, shock beats safe on revenue in {:.2f}"
          .format(beat))
    print("   of the distribution")
    print("\nReading: the top of the view rankings is dominated by the")
    print("high-variance style, so studying winners says 'use shock")
    print("hooks'. Yet the typical shock creator earns less, and earns")
    print("less at most points of the distribution. The losers of the")
    print("risky style are invisible because nobody studies them.")
#    share of shock style among the top 1 % by views: 1.00
#    safe : mean views   3,362  mean revenue RM  1,647  median RM 1,473
#    shock: mean views   5,101  mean revenue RM  1,250  median RM   584
#    quantile by quantile, shock beats safe on revenue in 0.10
#    of the distribution
#
# Reading: the top of the view rankings is dominated by the
# high-variance style, so studying winners says 'use shock
# hooks'. Yet the typical shock creator earns less, and earns
# less at most points of the distribution. The losers of the
# risky style are invisible because nobody studies them.

All (1.00) of the top 1% of accounts by views use the shock hook, so studying winners says “use shock hooks”. The shock hook does bring more views on average (5,101 against 3,362), but its mean revenue is RM 1,250, below the safe hook’s RM 1,647; median revenue is RM 584 against RM 1,473. Compared quantile by quantile, the shock hook wins on revenue in only 0.10 of the distribution. The failures of the risky style are invisible because nobody studies them.

View decay and early forecasts

The program lets daily views be the sum of a fast feed component and a slow search component, computes the distribution of a year’s views, and measures the error of forecasting the year from the first three days with a single exponential decay.

Calculation

# Chapter 6. p13: How fast do views decay, and what do day-3 numbers say?
# Daily views on day t (t = 1, 2, ...) are the sum of a fast feed
# component and a slow search component:
#   v(t) = 10,000 exp(-t / 1.2) + 150 exp(-t / 90).
# The program reports the half-lives, the share of first-year views
# that arrive in the first 3 days, the day by which half and 90 % of
# the year's views have arrived, and how wrong a forecast is that fits
# one exponential to days 1 to 3 and extrapolates to a year.
import math

def v(t):
    return 10000 * math.exp(-t / 1.2) + 150 * math.exp(-t / 90)

if __name__ == "__main__":
    year = [v(t) for t in range(1, 366)]
    tot = sum(year)
    print("   half-life of feed views   {:.2f} days".format(
        1.2 * math.log(2)))
    print("   half-life of search views {:.1f} days".format(
        90 * math.log(2)))
    print("   first-year views          {:,.0f}".format(tot))
    print("   share in the first 3 days {:.3f}".format(sum(year[:3]) / tot))
    cum, d50, d90 = 0.0, None, None
    for t, x in enumerate(year, 1):
        cum += x
        if d50 is None and cum >= 0.5 * tot:
            d50 = t
        if d90 is None and cum >= 0.9 * tot:
            d90 = t
    print("   half of the year's views by day {}, 90 % by day {}".format(
        d50, d90))
    xs = [1.0, 2.0, 3.0]
    ys = [math.log(v(t)) for t in (1, 2, 3)]
    mx, my = sum(xs) / 3, sum(ys) / 3
    slope = (sum((x - mx) * (y - my) for x, y in zip(xs, ys)) /
             sum((x - mx) ** 2 for x in xs))
    icpt = my - slope * mx
    fit = sum(math.exp(icpt + slope * t) for t in range(1, 366))
    print("   one-exponential forecast from days 1 to 3: {:,.0f}".format(
        fit))
    print("   forecast error {:.3f} of the true total".format(
        (fit - tot) / tot))
    print("\nReading: the feed burst is over within days, yet it brings")
    print("only about a third of the year's views; the slow search tail")
    print("brings the rest over months. A forecast fitted to the first")
    print("three days misses that tail and undercounts the year by more")
    print("than half.")
#    half-life of feed views   0.83 days
#    half-life of search views 62.4 days
#    first-year views          20,879
#    share in the first 3 days 0.359
#    half of the year's views by day 21, 90 % by day 158
#    one-exponential forecast from days 1 to 3: 8,314
#    forecast error -0.602 of the true total
#
# Reading: the feed burst is over within days, yet it brings
# only about a third of the year's views; the slow search tail
# brings the rest over months. A forecast fitted to the first
# three days misses that tail and undercounts the year by more
# than half.

The feed component has a half-life of 0.83 days, the search component 62.4 days. Of 20,879 views in the first year, the first three days bring only 0.359; half have arrived by day 21 and ninety percent by day 158. A single exponential fitted to the first three days forecasts 8,314 views for the year, too low by 0.602. The burst ends quickly, but the long tail brings most of the views.

Cumulative share of a year’s views against days since posting (log axis). Dotted lines mark one half and ninety percent.

Summary

Rare events such as paid conversion take tens of thousands of viewers to test; peeking and stopping early raises the false positive rate to about one in four. Studying only the top accounts systematically recommends risky strategies with lower expected value. Forecasting a year from the first three days badly underestimates the long tail.

References

Denrell, Jerker. 2003. “Vicarious Learning, Undersampling of Failure, and the Myths of Management.” Organization Science 14 (3): 227–43.
Gelman, Andrew, and Eric Loken. 2014. “The Statistical Crisis in Science.” American Scientist 102 (6): 460–65.
Kohavi, Ron, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to a/b Testing. Cambridge University Press.

Fan, Y. (2026). Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels: Weibull Swipe-Away Kinetics, Multi-Stage Pool Promotion Thresholds and Traffic-to-Cashflow Conversion Bounds [短视频完播率风险函数、推荐反馈环与自动化获客漏斗] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090213 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).