What does The Nightly Funnel conclude, in numbers, about “Thompson Sampling and Regret: from Two Small Samples in 1933 to the Even Split”?

Answer: Chapter 2 compares the even split, explore-then-commit, UCB1 and Thompson sampling;

Scope of the numbers: Its parameters are illustrative; all calculations are generated by its 23 programs; it is general business analysis, not business, legal or financial advice, and it promises no rise in conversion.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

rulemean regretP10P90share to best
equal60.060.060.00.333
etc33.024.059.90.603
ucb153.650.157.60.387
ts30.113.054.10.615
visitors 10,000; best variant pays 0.052
Show the full program
# Chapter 2. p21: cumulative regret of four allocation rules.
# Three page variants with illustrative pay rates. Regret is expected
# conversions lost against always showing the best variant:
# sum over visitors of (p_best - p_shown). Rules: equal split for the
# whole horizon (classic A/B), explore-then-commit (equal split for the
# first 40 percent, then the leader), UCB1 (Auer et al. 2002) and
# Thompson sampling with Beta(1, 1) priors (Thompson 1933).
import math
import random

P = [0.040, 0.046, 0.052]
T = 10000
REPS = 100
rng = random.Random(40521)
BEST = max(P)


def run(rule):
    s = [0] * 3
    n = [0] * 3
    reg = []
    for t in range(T):
        if rule == "equal":
            a = t % 3
        elif rule == "etc":
            if t < 0.4 * T:
                a = t % 3
            else:
                a = max(range(3), key=lambda k: s[k] / n[k])
        elif rule == "ucb1":
            if t < 3:
                a = t
            else:
                a = max(range(3), key=lambda k: s[k] / n[k]
                        + math.sqrt(2 * math.log(t) / n[k]))
        else:
            a = max(range(3), key=lambda k: rng.betavariate(
                1 + s[k], 1 + n[k] - s[k]))
        y = 1 if rng.random() < P[a] else 0
        s[a] += y
        n[a] += 1
        reg.append(BEST - P[a])
    return math.fsum(reg), n[2] / T


print("rule      mean regret   P10    P90   share to best")
for rule in ["equal", "etc", "ucb1", "ts"]:
    regs, shares = [], []
    for _ in range(REPS):
        r, sh = run(rule)
        regs.append(r)
        shares.append(sh)
    regs.sort()
    print("{:<8} {:>10.1f} {:>7.1f} {:>6.1f}   {:.3f}".format(
        rule, math.fsum(regs) / REPS, regs[REPS // 10],
        regs[9 * REPS // 10], math.fsum(shares) / REPS))
print("visitors {:,}; best variant pays {:.3f}".format(T, BEST))

Single source and citation

Fan, Y. (2026). Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement (Version 1.0.0), Chapter 2: Thompson Sampling and Regret: from Two Small Samples in 1933 to the Even Split. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-405 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026405,
  author    = {Fan, Yafu},
  title     = {Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book