What does Resampling for Shop Experiments conclude, in numbers, about “Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule”?
Answer: Chapter 3 explains the add-one rule and the number of shuffles for Monte Carlo permutation: when the exact p is 0.05901, ninety-nine shuffles call it significant with probability 0.299.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.05901, 0.299
| Monte Carlo standard error of p at p = 0.05 | |||
| B = | 99 | se = 0.0219 | (95% band +- 0.0429) |
| B = | 999 | se = 0.0069 | (95% band +- 0.0135) |
| B = | 9999 | se = 0.0022 | (95% band +- 0.0043) |
| B = | 99999 | se = 0.0007 | (95% band +- 0.0014) |
| P(declare p <= 0.05) for exact p = 0.05901 and 0.04 | |||
| B = | 99 | 0.299 | 0.637 |
| B = | 999 | 0.100 | 0.935 |
| B = | 9999 | 0.000 | 1.000 |
| H0 true, 5 + 5 exponential days, 10000 datasets | |||
| B = 50 (b+1)/(B+1): 0.0350 b/B: 0.0529 p=0: 0.0160 | |||
| B = 99 (b+1)/(B+1): 0.0485 b/B: 0.0485 p=0: 0.0097 | |||
| exact B = | 50 | (b+1)/(B+1): 0.0372 | b/B: 0.0568 |
| exact B = | 99 | (b+1)/(B+1): 0.0480 | b/B: 0.0480 |
Show the full program
# Chapter 3. p32: how many shuffles, and why add one.
# Part 1: the Monte Carlo p-value is a binomial share, so its standard
# error is sqrt(p (1 - p) / B). Part 2: the chance that a Monte Carlo
# run declares p <= 0.05 when the exact p is 0.05901 (the p31 data) or
# 0.04, from the exact binomial distribution (log-gamma terms, fsum).
# Part 3: under H0 (5 + 5 exponential days) the rule
# (b + 1) / (B + 1) <= 0.05 rejects at most 5 percent of the time,
# while the naive b / B <= 0.05 can exceed it; 10000 seeded datasets.
# The exact rates follow because the observed split is uniform among
# the C(10, 5) = 252 splits: if it ranks m-th, b ~ Binomial(B, m/252).
import math
import random
def binom_cdf(k, n, p):
if k < 0:
return 0.0
if p >= 1.0:
return 1.0 if k >= n else 0.0
terms = []
for j in range(0, min(k, n) + 1):
lt = (math.lgamma(n + 1) - math.lgamma(j + 1)
- math.lgamma(n - j + 1)
+ j * math.log(p) + (n - j) * math.log(1 - p))
terms.append(math.exp(lt))
return min(1.0, math.fsum(terms))
def exact_rate(nperm, kmax, splits=252):
terms = [binom_cdf(kmax, nperm, m / splits)
for m in range(1, splits + 1)]
return math.fsum(terms) / splits
def reject_prob(nperm, p_true, alpha=0.05):
kmax = math.floor(alpha * (nperm + 1) + 1e-9) - 1
return binom_cdf(kmax, nperm, p_true)
if __name__ == "__main__":
print("Monte Carlo standard error of p at p = 0.05")
for nperm in (99, 999, 9999, 99999):
se = math.sqrt(0.05 * 0.95 / nperm)
print(" B = {:6d} se = {:.4f} (95% band +- {:.4f})".format(
nperm, se, 1.96 * se))
print("P(declare p <= 0.05) for exact p = 0.05901 and 0.04")
for nperm in (99, 999, 9999):
print(" B = {:5d} {:.3f} {:.3f}".format(
nperm, reject_prob(nperm, 0.05901), reject_prob(nperm, 0.04)))
rng = random.Random(20261117)
reps = 10000
print("H0 true, 5 + 5 exponential days, {} datasets".format(reps))
for nperm in (50, 99):
plus = naive = zero = 0
for _ in range(reps):
pool = [rng.expovariate(1.0) for _ in range(10)]
obs = math.fsum(pool[5:])
b = 0
for _ in range(nperm):
rng.shuffle(pool)
b += math.fsum(pool[5:]) >= obs
plus += (b + 1) / (nperm + 1) <= 0.05
naive += b / nperm <= 0.05
zero += b == 0
print(" B = {:3d} (b+1)/(B+1): {:.4f} b/B: {:.4f} p=0: {:.4f}"
.format(nperm, plus / reps, naive / reps, zero / reps))
for nperm in (50, 99):
k1 = math.floor(0.05 * (nperm + 1) + 1e-9) - 1
k2 = math.floor(0.05 * nperm + 1e-9)
print(" exact B = {:3d} (b+1)/(B+1): {:.4f} b/B: {:.4f}".format(
nperm, exact_rate(nperm, k1), exact_rate(nperm, k2)))Single source and citation
Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 3: Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026308,
author = {Fan, Yafu},
title = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples
- Chapter 2 · From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report
- Chapter 4 · Bootstrap Intervals: Percentile, Bias-Corrected and Accelerated, and Undercoverage in Small Samples
- Chapter 5 · When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too
- Chapter 6 · Designing a Shop Experiment: No Peeking, No Early Stopping, Pair First, Size by Simulation
- Chapter 7 · Eastern and Western Sources: Tasting One Slice of Meat in the Lüshi Chunqiu and Fisher's Lady Tasting Tea
- Chapter 8 · An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book