What does Bayesian Search and Optimal Stopping conclude, in numbers, about “Practice and a Pre-Registered Trial”?

Answer: The sample size rests on the following assumptions: a smallest effect of interest of 4 percentage points (0.04); a standard deviation of each participant's share of 0.12 (the binomial sampling error over 20 games is about 0.107, plus differences between people; the model's standard deviations are 0.107 to 0.120); two-sided \(\alpha = 0.05\); and power of 90 %. Each arm needs 190 analysable participants; allowing for about 15 % excluded for not finishing or failing the attention checks, each arm recruits 224, 448 in all. With a standard deviation of 0.10 or 0.14, each arm would need 132 or 258; with a smallest effect of interest of 0.03 and standard deviation 0.12, 337. All these parameters are assumptions. Before the main trial, a small pilot checks only that the task software, the length of the explanations and the attention checks work properly; it is not used to change the sample size.

Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (17 programs, all checked), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.03, 0.04, 0.05, 0.10, 0.107, 0.12

model: control mean share 0.321 (SD 0.120), trained 0.367 (SD 0.107)
model difference +0.046; ceiling with the optimal cutoff0.376
analysable participants per arm, power 90 %:
SD of share diff 0.03 diff 0.04 diff 0.05
0.10234 132 85
0.12337 190 122
0.14458 258 165
pre-registered plan: SD 0.12, smallest effect of interest 0.04, 15 % excluded
power 90 %190 analysable and 224 recruited per arm, 448 in all
Show the full program
# Chapter 8 trial. Part 1: a MODEL (simulated participants, not data). 20 games of the
# secretary problem, n = 40, ranks only; each person skips round(40 x) then takes the
# first relative best; control x ~ N(0.22, 0.10), trained x ~ N(0.33, 0.07), truncated
# to [0.02, 0.90]: assumptions, not estimates. Outcome: share of games best chosen.
# Part 2: sample size for a difference in mean shares, two-sided alpha 0.05.
import math, random
from statistics import NormalDist, mean, stdev

N, GAMES = 40, 20

def p_success(r):                        # skip r - 1 applicants, exact probability
    return 1 / N if r == 1 else (r - 1) / N * sum(1 / (i - 1) for i in range(r, N + 1))

def participant(mu, sd, rng):
    x = min(0.90, max(0.02, rng.gauss(mu, sd)))
    p = p_success(round(N * x) + 1)
    return sum(rng.random() < p for _ in range(GAMES)) / GAMES

def n_per_arm(sd, diff, power=0.90, alpha=0.05):
    z = NormalDist().inv_cdf
    return math.ceil(2 * (z(1 - alpha / 2) + z(power)) ** 2 * sd ** 2 / diff ** 2)

if __name__ == "__main__":
    rng = random.Random(1997)
    ctrl = [participant(0.22, 0.10, rng) for _ in range(20000)]
    trn = [participant(0.33, 0.07, rng) for _ in range(20000)]
    print(f"model: control mean share {mean(ctrl):.3f} (SD {stdev(ctrl):.3f}),"
          f" trained {mean(trn):.3f} (SD {stdev(trn):.3f})")
    print(f"model difference {mean(trn) - mean(ctrl):+.3f};"
          f" ceiling with the optimal cutoff: {p_success(16):.3f}")
    print("analysable participants per arm, power 90 %:")
    print("   SD of share   diff 0.03   diff 0.04   diff 0.05")
    for sd in (0.10, 0.12, 0.14):
        cols = "      ".join(f"{n_per_arm(sd, d):6d}" for d in (0.03, 0.04, 0.05))
        print(f"      {sd:.2f}       {cols}")
    print("pre-registered plan: SD 0.12, smallest effect of interest 0.04, 15 % excluded")
    n = n_per_arm(0.12, 0.04); rec = math.ceil(n / 0.85)
    print(f"   power 90 %: {n} analysable and {rec} recruited per arm, {2 * rec} in all")

Single source and citation

Fan, Y. (2026). Bayesian Search in Unknown Landscapes, Optimal Stopping and Gittins Indices (Version 1.0.0), Chapter 8: Practice and a Pre-Registered Trial. 5D Library. https://doi.org/10.5281/zenodo.23810102

DOI: 10.5281/zenodo.23810102 · Read the monograph · 5DL-B-042 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026042,
  author    = {Fan, Yafu},
  title     = {Bayesian Search in Unknown Landscapes, Optimal Stopping and Gittins Indices},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23810102},
  url       = {https://doi.org/10.5281/zenodo.23810102}
}

Other chapter answers in this book