What does The Nightly Funnel conclude, in numbers, about “Thompson Sampling and Regret: from Two Small Samples in 1933 to the Even Split”?
Answer: Chapter 2 compares the even split, explore-then-commit, UCB1 and Thompson sampling;
Scope of the numbers: Its parameters are illustrative; all calculations are generated by its 23 programs; it is general business analysis, not business, legal or financial advice, and it promises no rise in conversion.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
| rule | mean regret | P10 | P90 | share to best |
| equal | 60.0 | 60.0 | 60.0 | 0.333 |
| etc | 33.0 | 24.0 | 59.9 | 0.603 |
| ucb1 | 53.6 | 50.1 | 57.6 | 0.387 |
| ts | 30.1 | 13.0 | 54.1 | 0.615 |
| visitors 10,000; best variant pays 0.052 |
Show the full program
# Chapter 2. p21: cumulative regret of four allocation rules.
# Three page variants with illustrative pay rates. Regret is expected
# conversions lost against always showing the best variant:
# sum over visitors of (p_best - p_shown). Rules: equal split for the
# whole horizon (classic A/B), explore-then-commit (equal split for the
# first 40 percent, then the leader), UCB1 (Auer et al. 2002) and
# Thompson sampling with Beta(1, 1) priors (Thompson 1933).
import math
import random
P = [0.040, 0.046, 0.052]
T = 10000
REPS = 100
rng = random.Random(40521)
BEST = max(P)
def run(rule):
s = [0] * 3
n = [0] * 3
reg = []
for t in range(T):
if rule == "equal":
a = t % 3
elif rule == "etc":
if t < 0.4 * T:
a = t % 3
else:
a = max(range(3), key=lambda k: s[k] / n[k])
elif rule == "ucb1":
if t < 3:
a = t
else:
a = max(range(3), key=lambda k: s[k] / n[k]
+ math.sqrt(2 * math.log(t) / n[k]))
else:
a = max(range(3), key=lambda k: rng.betavariate(
1 + s[k], 1 + n[k] - s[k]))
y = 1 if rng.random() < P[a] else 0
s[a] += y
n[a] += 1
reg.append(BEST - P[a])
return math.fsum(reg), n[2] / T
print("rule mean regret P10 P90 share to best")
for rule in ["equal", "etc", "ucb1", "ts"]:
regs, shares = [], []
for _ in range(REPS):
r, sh = run(rule)
regs.append(r)
shares.append(sh)
regs.sort()
print("{:<8} {:>10.1f} {:>7.1f} {:>6.1f} {:.3f}".format(
rule, math.fsum(regs) / REPS, regs[REPS // 10],
regs[9 * REPS // 10], math.fsum(shares) / REPS))
print("visitors {:,}; best variant pays {:.3f}".format(T, BEST))Single source and citation
Fan, Y. (2026). Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement (Version 1.0.0), Chapter 2: Thompson Sampling and Regret: from Two Small Samples in 1933 to the Even Split. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-405 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026405,
author = {Fan, Yafu},
title = {Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Funnel Check-up: Six-Step Beta Posteriors, Nightly Change Priorities and a Compliant Candidate Pool
- Chapter 3 · The Reward Function: Net Contribution per Visitor, Not the Conversion Rate
- Chapter 4 · Non-stationary Traffic: Weekday Effects, Discounted Memory and Sliding Windows
- Chapter 5 · Elimination Rules: Minimum Exposure, Posterior Win Probability and Time-uniform Confidence Sequences
- Chapter 6 · Trustworthy Data: Negative Bias in Adaptive Data, Sample Ratio Mismatch and Guardrail Rollback
- Chapter 7 · Classical Sources East and West: "At the Limit, Change" in the Great Commentary, the Sunzi on Weak and Strong Points, and Random Allocation in Peirce and Jastrow
- Chapter 8 · The Nightly Funnel Ledger: Field Tools and Checker Specifications, a Pre-registered Test and the Conditions of Refutation