What does Resampling for Shop Experiments conclude, in numbers, about “Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule”?

Answer: Chapter 3 explains the add-one rule and the number of shuffles for Monte Carlo permutation: when the exact p is 0.05901, ninety-nine shuffles call it significant with probability 0.299.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.05901, 0.299

Monte Carlo standard error of p at p = 0.05
B =99se = 0.0219(95% band +- 0.0429)
B =999se = 0.0069(95% band +- 0.0135)
B =9999se = 0.0022(95% band +- 0.0043)
B =99999se = 0.0007(95% band +- 0.0014)
P(declare p <= 0.05) for exact p = 0.05901 and 0.04
B =990.2990.637
B =9990.1000.935
B =99990.0001.000
H0 true, 5 + 5 exponential days, 10000 datasets
B = 50 (b+1)/(B+1): 0.0350 b/B: 0.0529 p=0: 0.0160
B = 99 (b+1)/(B+1): 0.0485 b/B: 0.0485 p=0: 0.0097
exact B =50(b+1)/(B+1): 0.0372b/B: 0.0568
exact B =99(b+1)/(B+1): 0.0480b/B: 0.0480
Show the full program
# Chapter 3. p32: how many shuffles, and why add one.
# Part 1: the Monte Carlo p-value is a binomial share, so its standard
# error is sqrt(p (1 - p) / B). Part 2: the chance that a Monte Carlo
# run declares p <= 0.05 when the exact p is 0.05901 (the p31 data) or
# 0.04, from the exact binomial distribution (log-gamma terms, fsum).
# Part 3: under H0 (5 + 5 exponential days) the rule
# (b + 1) / (B + 1) <= 0.05 rejects at most 5 percent of the time,
# while the naive b / B <= 0.05 can exceed it; 10000 seeded datasets.
# The exact rates follow because the observed split is uniform among
# the C(10, 5) = 252 splits: if it ranks m-th, b ~ Binomial(B, m/252).
import math
import random


def binom_cdf(k, n, p):
    if k < 0:
        return 0.0
    if p >= 1.0:
        return 1.0 if k >= n else 0.0
    terms = []
    for j in range(0, min(k, n) + 1):
        lt = (math.lgamma(n + 1) - math.lgamma(j + 1)
              - math.lgamma(n - j + 1)
              + j * math.log(p) + (n - j) * math.log(1 - p))
        terms.append(math.exp(lt))
    return min(1.0, math.fsum(terms))


def exact_rate(nperm, kmax, splits=252):
    terms = [binom_cdf(kmax, nperm, m / splits)
             for m in range(1, splits + 1)]
    return math.fsum(terms) / splits


def reject_prob(nperm, p_true, alpha=0.05):
    kmax = math.floor(alpha * (nperm + 1) + 1e-9) - 1
    return binom_cdf(kmax, nperm, p_true)


if __name__ == "__main__":
    print("Monte Carlo standard error of p at p = 0.05")
    for nperm in (99, 999, 9999, 99999):
        se = math.sqrt(0.05 * 0.95 / nperm)
        print("  B = {:6d}  se = {:.4f}  (95% band +- {:.4f})".format(
            nperm, se, 1.96 * se))
    print("P(declare p <= 0.05) for exact p = 0.05901 and 0.04")
    for nperm in (99, 999, 9999):
        print("  B = {:5d}   {:.3f}   {:.3f}".format(
            nperm, reject_prob(nperm, 0.05901), reject_prob(nperm, 0.04)))
    rng = random.Random(20261117)
    reps = 10000
    print("H0 true, 5 + 5 exponential days, {} datasets".format(reps))
    for nperm in (50, 99):
        plus = naive = zero = 0
        for _ in range(reps):
            pool = [rng.expovariate(1.0) for _ in range(10)]
            obs = math.fsum(pool[5:])
            b = 0
            for _ in range(nperm):
                rng.shuffle(pool)
                b += math.fsum(pool[5:]) >= obs
            plus += (b + 1) / (nperm + 1) <= 0.05
            naive += b / nperm <= 0.05
            zero += b == 0
        print("  B = {:3d}  (b+1)/(B+1): {:.4f}  b/B: {:.4f}  p=0: {:.4f}"
              .format(nperm, plus / reps, naive / reps, zero / reps))
    for nperm in (50, 99):
        k1 = math.floor(0.05 * (nperm + 1) + 1e-9) - 1
        k2 = math.floor(0.05 * nperm + 1e-9)
        print("  exact B = {:3d}  (b+1)/(B+1): {:.4f}  b/B: {:.4f}".format(
            nperm, exact_rate(nperm, k1), exact_rate(nperm, k2)))

Single source and citation

Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 3: Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026308,
  author    = {Fan, Yafu},
  title     = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book