What does Resampling for Shop Experiments conclude, in numbers, about “When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too”?
Answer: Chapter 5 shows that permutation tests also lose control when exchangeability fails: unequal variances with unequal groups give a false alarm rate of 0.250, and a before-and-after comparison gives 0.209 even with no trend.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.250
| equal means, unequal spread, nominal 0.05, 1000 data sets | ||||||
| n1 | sd1 | n2 | sd2 | perm raw | perm Welch | Welch t |
| 5 | 3.0 | 20 | 1.0 | 0.250 | 0.085 | 0.053 |
| 10 | 3.0 | 10 | 1.0 | 0.069 | 0.069 | 0.052 |
| 5 | 1.0 | 20 | 3.0 | 0.001 | 0.032 | 0.041 |
Show the full program
# Chapter 5. p51: when the arms are not exchangeable, the level slips.
# Model: two arms with EQUAL means but unequal spread (normal data, so
# skew plays no part); H0 "same mean" is true but H0 "same
# distribution" is false, so the labels are not exchangeable.
# Tests at nominal two-sided 0.05: (1) permutation of the raw mean
# difference, (2) permutation of the Welch statistic (studentized),
# (3) Welch t with the t distribution of p11. Monte Carlo permutation
# with B = 99 shuffles and p = (b + 1) / (B + 1); 1000 seeded data sets
# per row, Monte Carlo standard error about 0.007. Illustrative.
import math
import random
def betacf(a, b, x):
tiny = 1e-300
qab, qap, qam = a + b, a + 1.0, a - 1.0
c = 1.0
d = 1.0 - qab * x / qap
d = 1.0 / (d if abs(d) > tiny else tiny)
h = d
for m in range(1, 300):
m2 = 2 * m
aa = m * (b - m) * x / ((qam + m2) * (a + m2))
d = 1.0 + aa * d
d = 1.0 / (d if abs(d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
h *= d * c
aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
d = 1.0 + aa * d
d = 1.0 / (d if abs(d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
de = d * c
h *= de
if abs(de - 1.0) < 1e-15:
break
return h
def betai(a, b, x):
if x <= 0.0:
return 0.0
if x >= 1.0:
return 1.0
lb = (math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
+ a * math.log(x) + b * math.log(1.0 - x))
bt = math.exp(lb)
if x < (a + 1.0) / (a + b + 2.0):
return bt * betacf(a, b, x) / a
return 1.0 - bt * betacf(b, a, 1.0 - x) / b
def t_two_tail(t, v):
return betai(v / 2.0, 0.5, v / (v + t * t))
def t_cdf(t, v):
half = 0.5 * t_two_tail(t, v)
return 1.0 - half if t > 0 else half
def t_crit(alpha, v):
lo, hi = 0.0, 1000.0
for _ in range(200):
mid = 0.5 * (lo + hi)
if t_two_tail(mid, v) > alpha:
lo = mid
else:
hi = mid
return 0.5 * (lo + hi)
def welch(x, y):
nx, ny = len(x), len(y)
mx, my = math.fsum(x) / nx, math.fsum(y) / ny
vx = math.fsum((v - mx) ** 2 for v in x) / (nx - 1)
vy = math.fsum((v - my) ** 2 for v in y) / (ny - 1)
se2 = vx / nx + vy / ny
df = se2 ** 2 / ((vx / nx) ** 2 / (nx - 1) + (vy / ny) ** 2 / (ny - 1))
return (mx - my) / math.sqrt(se2), df
def raw(x, y):
return math.fsum(x) / len(x) - math.fsum(y) / len(y)
def one_row(n1, s1, n2, s2, reps, nperm, rng):
rej = [0, 0, 0]
for _ in range(reps):
x = [rng.gauss(0.0, s1) for _ in range(n1)]
y = [rng.gauss(0.0, s2) for _ in range(n2)]
d0 = abs(raw(x, y))
w0, df = welch(x, y)
rej[2] += t_two_tail(w0, df) <= 0.05
pool = x + y
b_raw = b_st = 0
for _ in range(nperm):
rng.shuffle(pool)
xs, ys = pool[:n1], pool[n1:]
b_raw += abs(raw(xs, ys)) >= d0
b_st += abs(welch(xs, ys)[0]) >= abs(w0)
rej[0] += (b_raw + 1) / (nperm + 1) <= 0.05
rej[1] += (b_st + 1) / (nperm + 1) <= 0.05
return [r / reps for r in rej]
if __name__ == "__main__":
rng = random.Random(20261120)
print("equal means, unequal spread, nominal 0.05, 1000 data sets")
print(" n1 sd1 n2 sd2 perm raw perm Welch Welch t")
for n1, s1, n2, s2 in ((5, 3.0, 20, 1.0), (10, 3.0, 10, 1.0),
(5, 1.0, 20, 3.0)):
r = one_row(n1, s1, n2, s2, 1000, 99, rng)
print(" {:3d} {:.1f} {:3d} {:.1f}"
" {:.3f} {:.3f} {:.3f}"
.format(n1, s1, n2, s2, r[0], r[1], r[2]))Single source and citation
Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 5: When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026308,
author = {Fan, Yafu},
title = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples
- Chapter 2 · From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report
- Chapter 3 · Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule
- Chapter 4 · Bootstrap Intervals: Percentile, Bias-Corrected and Accelerated, and Undercoverage in Small Samples
- Chapter 6 · Designing a Shop Experiment: No Peeking, No Early Stopping, Pair First, Size by Simulation
- Chapter 7 · Eastern and Western Sources: Tasting One Slice of Meat in the Lüshi Chunqiu and Fisher's Lady Tasting Tea
- Chapter 8 · An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book