What does Bayesian Search and Optimal Stopping conclude, in numbers, about “Practice and a Pre-Registered Trial”?
Answer: The sample size rests on the following assumptions: a smallest effect of interest of 4 percentage points (0.04); a standard deviation of each participant's share of 0.12 (the binomial sampling error over 20 games is about 0.107, plus differences between people; the model's standard deviations are 0.107 to 0.120); two-sided \(\alpha = 0.05\); and power of 90 %. Each arm needs 190 analysable participants; allowing for about 15 % excluded for not finishing or failing the attention checks, each arm recruits 224, 448 in all. With a standard deviation of 0.10 or 0.14, each arm would need 132 or 258; with a smallest effect of interest of 0.03 and standard deviation 0.12, 337. All these parameters are assumptions. Before the main trial, a small pilot checks only that the task software, the length of the explanations and the attention checks work properly; it is not used to change the sample size.
Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (17 programs, all checked), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.03, 0.04, 0.05, 0.10, 0.107, 0.12
| model: control mean share 0.321 (SD 0.120), trained 0.367 (SD 0.107) | |
| model difference +0.046; ceiling with the optimal cutoff | 0.376 |
| analysable participants per arm, power 90 %: | |
| SD of share diff 0.03 diff 0.04 diff 0.05 | |
| 0.10 | 234 132 85 |
| 0.12 | 337 190 122 |
| 0.14 | 458 258 165 |
| pre-registered plan: SD 0.12, smallest effect of interest 0.04, 15 % excluded | |
| power 90 % | 190 analysable and 224 recruited per arm, 448 in all |
Show the full program
# Chapter 8 trial. Part 1: a MODEL (simulated participants, not data). 20 games of the
# secretary problem, n = 40, ranks only; each person skips round(40 x) then takes the
# first relative best; control x ~ N(0.22, 0.10), trained x ~ N(0.33, 0.07), truncated
# to [0.02, 0.90]: assumptions, not estimates. Outcome: share of games best chosen.
# Part 2: sample size for a difference in mean shares, two-sided alpha 0.05.
import math, random
from statistics import NormalDist, mean, stdev
N, GAMES = 40, 20
def p_success(r): # skip r - 1 applicants, exact probability
return 1 / N if r == 1 else (r - 1) / N * sum(1 / (i - 1) for i in range(r, N + 1))
def participant(mu, sd, rng):
x = min(0.90, max(0.02, rng.gauss(mu, sd)))
p = p_success(round(N * x) + 1)
return sum(rng.random() < p for _ in range(GAMES)) / GAMES
def n_per_arm(sd, diff, power=0.90, alpha=0.05):
z = NormalDist().inv_cdf
return math.ceil(2 * (z(1 - alpha / 2) + z(power)) ** 2 * sd ** 2 / diff ** 2)
if __name__ == "__main__":
rng = random.Random(1997)
ctrl = [participant(0.22, 0.10, rng) for _ in range(20000)]
trn = [participant(0.33, 0.07, rng) for _ in range(20000)]
print(f"model: control mean share {mean(ctrl):.3f} (SD {stdev(ctrl):.3f}),"
f" trained {mean(trn):.3f} (SD {stdev(trn):.3f})")
print(f"model difference {mean(trn) - mean(ctrl):+.3f};"
f" ceiling with the optimal cutoff: {p_success(16):.3f}")
print("analysable participants per arm, power 90 %:")
print(" SD of share diff 0.03 diff 0.04 diff 0.05")
for sd in (0.10, 0.12, 0.14):
cols = " ".join(f"{n_per_arm(sd, d):6d}" for d in (0.03, 0.04, 0.05))
print(f" {sd:.2f} {cols}")
print("pre-registered plan: SD 0.12, smallest effect of interest 0.04, 15 % excluded")
n = n_per_arm(0.12, 0.04); rec = math.ceil(n / 0.85)
print(f" power 90 %: {n} analysable and {rec} recruited per arm, {2 * rec} in all")Single source and citation
Fan, Y. (2026). Bayesian Search in Unknown Landscapes, Optimal Stopping and Gittins Indices (Version 1.0.0), Chapter 8: Practice and a Pre-Registered Trial. 5D Library. https://doi.org/10.5281/zenodo.23810102
DOI: 10.5281/zenodo.23810102 · Read the monograph · 5DL-B-042 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026042,
author = {Fan, Yafu},
title = {Bayesian Search in Unknown Landscapes, Optimal Stopping and Gittins Indices},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23810102},
url = {https://doi.org/10.5281/zenodo.23810102}
}Other chapter answers in this book
- Chapter 1 · The Secretary Problem and 1/e: When to Stop If You See Only Ranks and Cannot Go Back
- Chapter 2 · Variants of the Secretary Problem and How People Stop
- Chapter 3 · Bayesian Search: Where to Look, and What Not Finding Means
- Chapter 4 · Multi-Armed Bandits: Gittins Indices, Thompson Sampling and the Regret Lower Bound
- Chapter 5 · Sunk Costs: Rational Persistence, Fallacious Persistence and Stop-Loss Boundaries Set in Advance
- Chapter 6 · Applications: Testing Lanes and Selecting People