5D Library › 短视频完播率风险函数、推荐反馈环与自动化获客漏斗 · English
第六章 证据:A/B 测试、幸存者偏差与早期预测
钩子 A/B 测试需要多少观众
在线对照实验是检验一个改动是否有效的标准方法 (Kohavi et al. 2020)。把同一条视频的两个开头随机分给两半观众,比较某个比例指标。程序计算三种指标在双侧 0.05、检验力 0.80 下每组所需的观众数,并模拟「每天看一眼、一显著就停」的做法。
推导
# Chapter 6. p11: How many impressions does a hook A/B test need?
# Two openings of the same video are shown to random halves of the
# audience. Per arm, for a two-sided test at 0.05 with power 0.80:
# n = (1.96 + 0.8416)^2 (p1 (1 - p1) + p2 (1 - p2)) / (p2 - p1)^2.
# Outcomes: 5-second retention 0.60 vs 0.63; completion 0.20 vs 0.22;
# paid conversion 0.010 vs 0.012 per viewer. Then the cost of peeking:
# under no true difference (both arms at 0.20), a creator checks the
# running z statistic after each of 20 daily batches of 500 viewers
# per arm and stops at the first |z| > 1.96. 4,000 simulated tests.
import math
import random
Z = 1.96 + 0.8416
def n_arm(p1, p2):
return math.ceil(Z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) /
(p2 - p1) ** 2)
if __name__ == "__main__":
print(" outcome p1 p2 viewers per arm")
for name, p1, p2 in (("5 s retention", 0.60, 0.63),
("completion", 0.20, 0.22),
("paid conversion", 0.010, 0.012)):
print(" {:16s} {:6.3f} {:7.3f} {:15,d}".format(
name, p1, p2, n_arm(p1, p2)))
rng = random.Random(20)
stops = 0
for _ in range(2000):
a = b = n = 0
for day in range(20):
for _ in range(500):
a += rng.random() < 0.2
b += rng.random() < 0.2
n += 500
pa, pb = a / n, b / n
pool = (a + b) / (2 * n)
se = math.sqrt(pool * (1 - pool) * 2 / n)
if se > 0 and abs(pa - pb) / se > 1.96:
stops += 1
break
print("\n peeking daily for 20 days with no true difference:")
print(" share of tests declared 'significant' {:.3f}".format(
stops / 2000))
print("\nReading: an opening that lifts early retention by three")
print("points can be tested in a few days of ordinary traffic; a")
print("change in paid conversion needs tens of thousands of viewers")
print("per arm. Checking every day and stopping at the first good")
print("number turns a 0.05 test into roughly a one in four false")
print("alarm.")
# outcome p1 p2 viewers per arm
# 5 s retention 0.600 0.630 4,126
# completion 0.200 0.220 6,507
# paid conversion 0.010 0.012 42,691
#
# peeking daily for 20 days with no true difference:
# share of tests declared 'significant' 0.239
#
# Reading: an opening that lifts early retention by three
# points can be tested in a few days of ordinary traffic; a
# change in paid conversion needs tens of thousands of viewers
# per arm. Checking every day and stopping at the first good
# number turns a 0.05 test into roughly a one in four false
# alarm.5 秒留存从 0.60 提高到 0.63,每组需要 4,126 名观众,几天的普通流量就够;完播率从 0.20 到 0.22 需要 6,507 名;付费转化率从 0.010 到 0.012 则需要 42,691 名。在两组其实没有差别的情况下,连续 20 天每天看一次、一见显著就停,有 0.239 的测试会宣告「显著」,约为四分之一,而不是名义上的 0.05。分析者在看到数据之后才决定何时停止、比较什么,正是分岔路径问题的一种 (Gelman and Loken 2014)。
只学头部账号
「爆款账号都用了某种钩子」是最常见的经验来源。Denrell 指出,只从成功者身上学习,会系统地高估高风险策略,因为高风险策略的失败者不在样本里 (Denrell 2003)。程序设 10,000 名创作者,一半用「稳健」钩子,一半用「惊悚」钩子;后者的典型播放更低、波动更大,观众的付费转化率也只有一半。
推导
# Chapter 6. p12: Learning from the top accounts: survivorship bias.
# 10,000 creators, half using a "safe" hook style and half a "shock"
# style. Monthly views are lognormal: safe log-mean 8.0, log-sd 0.5;
# shock log-mean 7.8, log-sd 1.2 (lower typical result, wider spread).
# Paid conversion per view: 0.010 for safe, 0.005 for shock (shock
# audiences buy less). Revenue = views x conversion x RM 49. Observers
# see only views, and study the top 1 % of accounts by views.
import math
import random
if __name__ == "__main__":
rng = random.Random(2003)
pop = []
for i in range(10000):
style = "safe" if i % 2 == 0 else "shock"
mu, sd, cv = (8.0, 0.5, 0.010) if style == "safe" else \
(7.8, 1.2, 0.005)
v = math.exp(rng.gauss(mu, sd))
pop.append((v, style, v * cv * 49))
top = sorted(pop, reverse=True)[:100]
share = sum(1 for p in top if p[1] == "shock") / len(top)
print(" share of shock style among the top 1 % by views: {:.2f}"
.format(share))
for style in ("safe", "shock"):
g = [p for p in pop if p[1] == style]
rev = sorted(p[2] for p in g)
print(" {:5s}: mean views {:7,.0f} mean revenue RM {:6,.0f}"
" median RM {:5,.0f}".format(
style, sum(p[0] for p in g) / len(g),
sum(rev) / len(rev), rev[len(rev) // 2]))
rs = sorted([p for p in pop if p[1] == "safe"], key=lambda p: p[2])
rk = sorted([p for p in pop if p[1] == "shock"], key=lambda p: p[2])
beat = sum(1 for a, b in zip(rs, rk) if b[2] > a[2]) / len(rs)
print(" quantile by quantile, shock beats safe on revenue in {:.2f}"
.format(beat))
print(" of the distribution")
print("\nReading: the top of the view rankings is dominated by the")
print("high-variance style, so studying winners says 'use shock")
print("hooks'. Yet the typical shock creator earns less, and earns")
print("less at most points of the distribution. The losers of the")
print("risky style are invisible because nobody studies them.")
# share of shock style among the top 1 % by views: 1.00
# safe : mean views 3,362 mean revenue RM 1,647 median RM 1,473
# shock: mean views 5,101 mean revenue RM 1,250 median RM 584
# quantile by quantile, shock beats safe on revenue in 0.10
# of the distribution
#
# Reading: the top of the view rankings is dominated by the
# high-variance style, so studying winners says 'use shock
# hooks'. Yet the typical shock creator earns less, and earns
# less at most points of the distribution. The losers of the
# risky style are invisible because nobody studies them.按播放排名的前 1% 账号全部(1.00)使用惊悚钩子,研究赢家会得出「用惊悚钩子」的结论。惊悚钩子的平均播放(5,101)确实高于稳健钩子(3,362),但平均收入为 RM 1,250,低于稳健钩子的 RM 1,647;收入中位数分别为 RM 584 与 RM 1,473。逐分位比较,惊悚钩子只在 0.10 的分布上赢过稳健钩子。高风险策略的失败者看不见,因为没人研究他们。
播放衰减与早期预测
程序设每日播放由快速的信息流部分与缓慢的搜索部分组成,并计算一年的播放分布,以及只用前三天的数据拟合单一指数衰减来预测全年的误差。
推导
# Chapter 6. p13: How fast do views decay, and what do day-3 numbers say?
# Daily views on day t (t = 1, 2, ...) are the sum of a fast feed
# component and a slow search component:
# v(t) = 10,000 exp(-t / 1.2) + 150 exp(-t / 90).
# The program reports the half-lives, the share of first-year views
# that arrive in the first 3 days, the day by which half and 90 % of
# the year's views have arrived, and how wrong a forecast is that fits
# one exponential to days 1 to 3 and extrapolates to a year.
import math
def v(t):
return 10000 * math.exp(-t / 1.2) + 150 * math.exp(-t / 90)
if __name__ == "__main__":
year = [v(t) for t in range(1, 366)]
tot = sum(year)
print(" half-life of feed views {:.2f} days".format(
1.2 * math.log(2)))
print(" half-life of search views {:.1f} days".format(
90 * math.log(2)))
print(" first-year views {:,.0f}".format(tot))
print(" share in the first 3 days {:.3f}".format(sum(year[:3]) / tot))
cum, d50, d90 = 0.0, None, None
for t, x in enumerate(year, 1):
cum += x
if d50 is None and cum >= 0.5 * tot:
d50 = t
if d90 is None and cum >= 0.9 * tot:
d90 = t
print(" half of the year's views by day {}, 90 % by day {}".format(
d50, d90))
xs = [1.0, 2.0, 3.0]
ys = [math.log(v(t)) for t in (1, 2, 3)]
mx, my = sum(xs) / 3, sum(ys) / 3
slope = (sum((x - mx) * (y - my) for x, y in zip(xs, ys)) /
sum((x - mx) ** 2 for x in xs))
icpt = my - slope * mx
fit = sum(math.exp(icpt + slope * t) for t in range(1, 366))
print(" one-exponential forecast from days 1 to 3: {:,.0f}".format(
fit))
print(" forecast error {:.3f} of the true total".format(
(fit - tot) / tot))
print("\nReading: the feed burst is over within days, yet it brings")
print("only about a third of the year's views; the slow search tail")
print("brings the rest over months. A forecast fitted to the first")
print("three days misses that tail and undercounts the year by more")
print("than half.")
# half-life of feed views 0.83 days
# half-life of search views 62.4 days
# first-year views 20,879
# share in the first 3 days 0.359
# half of the year's views by day 21, 90 % by day 158
# one-exponential forecast from days 1 to 3: 8,314
# forecast error -0.602 of the true total
#
# Reading: the feed burst is over within days, yet it brings
# only about a third of the year's views; the slow search tail
# brings the rest over months. A forecast fitted to the first
# three days misses that tail and undercounts the year by more
# than half.信息流部分的半衰期为 0.83 天,搜索部分为 62.4 天。全年 20,879 次播放中,前三天只占 0.359;一半的播放在第 21 天才到齐,九成在第 158 天。用前三天拟合单一指数预测全年为 8,314 次,低估了 0.602。爆发很快结束,但长尾带来了大部分播放。
小结
付费转化这样的稀有事件需要数万观众才能检验;偷看式的提前停止会把假阳性率提高到约四分之一。只研究头部账号会系统地推荐高风险、低期望的策略。以前三天的播放预测全年会严重低估长尾。
参考文献
Fan, Y. (2026). Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels: Weibull Swipe-Away Kinetics, Multi-Stage Pool Promotion Thresholds and Traffic-to-Cashflow Conversion Bounds [短视频完播率风险函数、推荐反馈环与自动化获客漏斗] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090213 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).