What does Completion Hazards, Feedback Loops and Funnels conclude, in numbers, about “A Ten-Flag Audit and a Pre-Registered Experiment Randomised by Video”?
Answer: Chapter 8 gives a ten-flag audit and a pre-registered experiment randomised by video (88 videos per arm). Marked boxes separate evidence, runnable Python programs, classical sources and hypotheses.
Scope of the numbers: Marked boxes separate evidence, runnable Python programs, classical sources and hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 88
| viewers per arm if viewers were independent | 34,999 |
| m viewers/video icc design effect videos per arm | |
| 500 | 0.0005 1.25 88 |
| 500 | 0.0020 2.00 140 |
| 2,000 | 0.0005 2.00 35 |
| 2,000 | 0.0020 5.00 88 |
| 8,000 | 0.0005 5.00 22 |
| 8,000 | 0.0020 17.00 75 |
| null simulation, video-level t test: false alarms 0.0465 | |
| Reading: the experiment must count videos, not viewers: | |
| between 22 and 140 videos per arm here, depending on how | |
| much one video's audience differs from another's. Analysed | |
| at the video level, the test keeps its stated error rate. |
Show the full program
# Chapter 8. p16: A pre-registered, video-randomised hook experiment.
# Each new video is randomly given a narrow hook (breadth 0.3) or a
# broad one (breadth 1.0). Viewers of one video share its audience
# draw, so the unit of analysis is the video, not the viewer.
# Primary outcome: paid conversion per viewer, assumed 0.010 under the
# narrow hook and 0.008 under the broad one at equal reach. Two-sided
# 0.05, power 0.80. Viewers per arm ignoring clustering:
# n = (1.96 + 0.8416)^2 (p1 (1 - p1) + p2 (1 - p2)) / (p1 - p2)^2;
# with m viewers per video and intraclass correlation icc, multiply by
# the design effect 1 + (m - 1) icc and divide by m to get videos.
# A simulation of 2,000 null experiments (both arms at 0.010, icc
# 0.002, m = 2,000, 60 videos per arm) checks the type I error of a
# video-level t test.
import math
import random
Z = 1.96 + 0.8416
def viewers(p1, p2):
return Z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2) ** 2
if __name__ == "__main__":
n = viewers(0.010, 0.008)
print(" viewers per arm if viewers were independent: {:,.0f}".format(
math.ceil(n)))
print(" m viewers/video icc design effect videos per arm")
for m in (500, 2000, 8000):
for icc in (0.0005, 0.002):
de = 1 + (m - 1) * icc
print(" {:15,d} {:7.4f} {:15.2f} {:16d}".format(
m, icc, de, math.ceil(n * de / m)))
rng = random.Random(16)
sd_p = math.sqrt(0.002 * 0.010 * 0.990)
rej = 0
for _ in range(2000):
arms = []
for _ in range(2):
rates = []
for _ in range(60):
p = min(max(rng.gauss(0.010, sd_p), 0.0), 1.0)
k = rng.gauss(2000 * p, math.sqrt(2000 * p * (1 - p)))
rates.append(k / 2000)
arms.append(rates)
ma, mb = (sum(a) / 60 for a in arms)
va = sum((x - ma) ** 2 for x in arms[0]) / 59
vb = sum((x - mb) ** 2 for x in arms[1]) / 59
t = (ma - mb) / math.sqrt(va / 60 + vb / 60)
rej += abs(t) > 1.98
print(" null simulation, video-level t test: false alarms {:.4f}"
.format(rej / 2000))
print("\nReading: the experiment must count videos, not viewers:")
print("between 22 and 140 videos per arm here, depending on how")
print("much one video's audience differs from another's. Analysed")
print("at the video level, the test keeps its stated error rate.")Single source and citation
Fan, Y. (2026). Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels (Version 1.0.0), Chapter 8: A Ten-Flag Audit and a Pre-Registered Experiment Randomised by Video. 5D Library. https://doi.org/10.5281/zenodo.24090213
DOI: 10.5281/zenodo.24090213 · Read the monograph · 5DL-B-081 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026081,
author = {Fan, Yafu},
title = {Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090213},
url = {https://doi.org/10.5281/zenodo.24090213}
}Other chapter answers in this book
- Chapter 1 · The Kinetics of Swiping: a Weibull Hazard
- Chapter 2 · Tiered Traffic Pools and the First Gate
- Chapter 3 · Feedback Loops, the Matthew Effect and Gameable Metrics
- Chapter 4 · Reverse Selection by Hook Breadth and the Timing of the Offer
- Chapter 5 · From Views to Cash: Lags, Heavy Tails and Linked Bans
- Chapter 6 · Evidence: A/B Tests, Survivorship and Early Forecasts
- Chapter 7 · Gauging Desire, Opening Lines and the Casket Without the Pearl