What does Completion Hazards, Feedback Loops and Funnels conclude, in numbers, about “A Ten-Flag Audit and a Pre-Registered Experiment Randomised by Video”?

Answer: Chapter 8 gives a ten-flag audit and a pre-registered experiment randomised by video (88 videos per arm). Marked boxes separate evidence, runnable Python programs, classical sources and hypotheses.

Scope of the numbers: Marked boxes separate evidence, runnable Python programs, classical sources and hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 88

viewers per arm if viewers were independent34,999
m viewers/video icc design effect videos per arm
5000.0005 1.25 88
5000.0020 2.00 140
2,0000.0005 2.00 35
2,0000.0020 5.00 88
8,0000.0005 5.00 22
8,0000.0020 17.00 75
null simulation, video-level t test: false alarms 0.0465
Reading: the experiment must count videos, not viewers:
between 22 and 140 videos per arm here, depending on how
much one video's audience differs from another's. Analysed
at the video level, the test keeps its stated error rate.
Show the full program
# Chapter 8. p16: A pre-registered, video-randomised hook experiment.
# Each new video is randomly given a narrow hook (breadth 0.3) or a
# broad one (breadth 1.0). Viewers of one video share its audience
# draw, so the unit of analysis is the video, not the viewer.
# Primary outcome: paid conversion per viewer, assumed 0.010 under the
# narrow hook and 0.008 under the broad one at equal reach. Two-sided
# 0.05, power 0.80. Viewers per arm ignoring clustering:
#   n = (1.96 + 0.8416)^2 (p1 (1 - p1) + p2 (1 - p2)) / (p1 - p2)^2;
# with m viewers per video and intraclass correlation icc, multiply by
# the design effect 1 + (m - 1) icc and divide by m to get videos.
# A simulation of 2,000 null experiments (both arms at 0.010, icc
# 0.002, m = 2,000, 60 videos per arm) checks the type I error of a
# video-level t test.
import math
import random

Z = 1.96 + 0.8416

def viewers(p1, p2):
    return Z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2) ** 2

if __name__ == "__main__":
    n = viewers(0.010, 0.008)
    print("   viewers per arm if viewers were independent: {:,.0f}".format(
        math.ceil(n)))
    print("   m viewers/video   icc     design effect   videos per arm")
    for m in (500, 2000, 8000):
        for icc in (0.0005, 0.002):
            de = 1 + (m - 1) * icc
            print("   {:15,d} {:7.4f} {:15.2f} {:16d}".format(
                m, icc, de, math.ceil(n * de / m)))
    rng = random.Random(16)
    sd_p = math.sqrt(0.002 * 0.010 * 0.990)
    rej = 0
    for _ in range(2000):
        arms = []
        for _ in range(2):
            rates = []
            for _ in range(60):
                p = min(max(rng.gauss(0.010, sd_p), 0.0), 1.0)
                k = rng.gauss(2000 * p, math.sqrt(2000 * p * (1 - p)))
                rates.append(k / 2000)
            arms.append(rates)
        ma, mb = (sum(a) / 60 for a in arms)
        va = sum((x - ma) ** 2 for x in arms[0]) / 59
        vb = sum((x - mb) ** 2 for x in arms[1]) / 59
        t = (ma - mb) / math.sqrt(va / 60 + vb / 60)
        rej += abs(t) > 1.98
    print("   null simulation, video-level t test: false alarms {:.4f}"
          .format(rej / 2000))
    print("\nReading: the experiment must count videos, not viewers:")
    print("between 22 and 140 videos per arm here, depending on how")
    print("much one video's audience differs from another's. Analysed")
    print("at the video level, the test keeps its stated error rate.")

Single source and citation

Fan, Y. (2026). Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels (Version 1.0.0), Chapter 8: A Ten-Flag Audit and a Pre-Registered Experiment Randomised by Video. 5D Library. https://doi.org/10.5281/zenodo.24090213

DOI: 10.5281/zenodo.24090213 · Read the monograph · 5DL-B-081 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026081,
  author    = {Fan, Yafu},
  title     = {Short-Video Completion Hazards, Recommender Exploration Loops and Automated Acquisition Funnels},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090213},
  url       = {https://doi.org/10.5281/zenodo.24090213}
}

Other chapter answers in this book