What does Creative Fatigue and Orthogonal Remixing conclude, in numbers, about “The Platform Will Not Run Your Experiment: the Bias of Allocating by Prediction and Thompson-Sampling Races”?

Answer: Chapter 6 shows that under adaptive platform allocation the reported relative difference between two identical creatives misses the truth by 0.348 on average, and that Thompson sampling lowers regret from 317.8 clicks under an equal split to a median of 187.1, at the price of biased estimates of the differences.

Scope of the numbers: Parameters are illustrative; all calculations come from the book's programs. This is general business analysis, not individual advice, and promises no advertising result.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 187.1, 317.8

impressions per campaign 200,000
equal split regret 317.8 clicks
Thompson regret 5/50/95%: 118.8 187.1 301.3 clicks
Thompson: best creative most shown in 0.793 of campaigns
best creative impression share 5/50/95%: 0.070 0.435 0.665
Show the full program
# Chapter 6. p62: Thompson sampling versus an equal split, nine creatives
# Nine L9 creatives; true CTRs are illustrative (one at 0.0120, the rest
# between 0.0095 and 0.0110). 200 rounds of 1,000 impressions. Batched
# Thompson sampling (Thompson 1933): each round draw one Beta posterior
# sample per creative and give the whole batch to the largest draw.
# Regret = clicks lost against always showing the best creative.
import math
import random

rng = random.Random(40062)
TRUE = [0.0120, 0.0110, 0.0108, 0.0105, 0.0103, 0.0100, 0.0099, 0.0097,
        0.0095]
ROUNDS, BATCH, RUNS = 200, 1000, 300
BEST = max(TRUE)


def clicks(n, p):
    return max(0, round(rng.gauss(n * p, math.sqrt(n * p * (1 - p)))))


def thompson():
    a = [1] * 9
    b = [1] * 9
    regret = []
    for _ in range(ROUNDS):
        draws = [rng.betavariate(a[k], b[k]) for k in range(9)]
        k = draws.index(max(draws))
        c = clicks(BATCH, TRUE[k])
        a[k] += c
        b[k] += BATCH - c
        regret.append((BEST - TRUE[k]) * BATCH)
    pulls = [a[k] + b[k] - 2 for k in range(9)]
    return math.fsum(regret), pulls.index(max(pulls)) == 0, pulls[0]


def equal_split():
    per = ROUNDS * BATCH / 9
    return math.fsum((BEST - p) * per for p in TRUE)


res = [thompson() for _ in range(RUNS)]
reg = sorted(r[0] for r in res)
share = sorted(r[2] / (ROUNDS * BATCH) for r in res)
print("impressions per campaign {:,}".format(ROUNDS * BATCH))
print("equal split regret {:.1f} clicks".format(equal_split()))
print("Thompson regret 5/50/95%: {:.1f} {:.1f} {:.1f} clicks".format(
    reg[RUNS // 20], reg[RUNS // 2], reg[RUNS * 19 // 20]))
print("Thompson: best creative most shown in {:.3f} of campaigns".format(
    len([1 for r in res if r[1]]) / RUNS))
print("best creative impression share 5/50/95%: {:.3f} {:.3f} {:.3f}".format(
    share[RUNS // 20], share[RUNS // 2], share[RUNS * 19 // 20]))

Single source and citation

Fan, Y. (2026). Ad-Creative Fatigue Half-Lives, Orthogonal Storyboard Recombination and Automated Permutation Rendering (Version 1.0.0), Chapter 6: The Platform Will Not Run Your Experiment: the Bias of Allocating by Prediction and Thompson-Sampling Races. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-400 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026400,
  author    = {Fan, Yafu},
  title     = {Ad-Creative Fatigue Half-Lives, Orthogonal Storyboard Recombination and Automated Permutation Rendering},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book