What does Resampling for Shop Experiments conclude, in numbers, about “When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too”?

Answer: Chapter 5 shows that permutation tests also lose control when exchangeability fails: unequal variances with unequal groups give a false alarm rate of 0.250, and a before-and-after comparison gives 0.209 even with no trend.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.250

equal means, unequal spread, nominal 0.05, 1000 data sets
n1sd1n2sd2perm rawperm WelchWelch t
53.0201.00.2500.0850.053
103.0101.00.0690.0690.052
51.0203.00.0010.0320.041
Show the full program
# Chapter 5. p51: when the arms are not exchangeable, the level slips.
# Model: two arms with EQUAL means but unequal spread (normal data, so
# skew plays no part); H0 "same mean" is true but H0 "same
# distribution" is false, so the labels are not exchangeable.
# Tests at nominal two-sided 0.05: (1) permutation of the raw mean
# difference, (2) permutation of the Welch statistic (studentized),
# (3) Welch t with the t distribution of p11. Monte Carlo permutation
# with B = 99 shuffles and p = (b + 1) / (B + 1); 1000 seeded data sets
# per row, Monte Carlo standard error about 0.007. Illustrative.
import math
import random

def betacf(a, b, x):
    tiny = 1e-300
    qab, qap, qam = a + b, a + 1.0, a - 1.0
    c = 1.0
    d = 1.0 - qab * x / qap
    d = 1.0 / (d if abs(d) > tiny else tiny)
    h = d
    for m in range(1, 300):
        m2 = 2 * m
        aa = m * (b - m) * x / ((qam + m2) * (a + m2))
        d = 1.0 + aa * d
        d = 1.0 / (d if abs(d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        h *= d * c
        aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
        d = 1.0 + aa * d
        d = 1.0 / (d if abs(d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        de = d * c
        h *= de
        if abs(de - 1.0) < 1e-15:
            break
    return h


def betai(a, b, x):
    if x <= 0.0:
        return 0.0
    if x >= 1.0:
        return 1.0
    lb = (math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
          + a * math.log(x) + b * math.log(1.0 - x))
    bt = math.exp(lb)
    if x < (a + 1.0) / (a + b + 2.0):
        return bt * betacf(a, b, x) / a
    return 1.0 - bt * betacf(b, a, 1.0 - x) / b


def t_two_tail(t, v):
    return betai(v / 2.0, 0.5, v / (v + t * t))


def t_cdf(t, v):
    half = 0.5 * t_two_tail(t, v)
    return 1.0 - half if t > 0 else half


def t_crit(alpha, v):
    lo, hi = 0.0, 1000.0
    for _ in range(200):
        mid = 0.5 * (lo + hi)
        if t_two_tail(mid, v) > alpha:
            lo = mid
        else:
            hi = mid
    return 0.5 * (lo + hi)


def welch(x, y):
    nx, ny = len(x), len(y)
    mx, my = math.fsum(x) / nx, math.fsum(y) / ny
    vx = math.fsum((v - mx) ** 2 for v in x) / (nx - 1)
    vy = math.fsum((v - my) ** 2 for v in y) / (ny - 1)
    se2 = vx / nx + vy / ny
    df = se2 ** 2 / ((vx / nx) ** 2 / (nx - 1) + (vy / ny) ** 2 / (ny - 1))
    return (mx - my) / math.sqrt(se2), df


def raw(x, y):
    return math.fsum(x) / len(x) - math.fsum(y) / len(y)


def one_row(n1, s1, n2, s2, reps, nperm, rng):
    rej = [0, 0, 0]
    for _ in range(reps):
        x = [rng.gauss(0.0, s1) for _ in range(n1)]
        y = [rng.gauss(0.0, s2) for _ in range(n2)]
        d0 = abs(raw(x, y))
        w0, df = welch(x, y)
        rej[2] += t_two_tail(w0, df) <= 0.05
        pool = x + y
        b_raw = b_st = 0
        for _ in range(nperm):
            rng.shuffle(pool)
            xs, ys = pool[:n1], pool[n1:]
            b_raw += abs(raw(xs, ys)) >= d0
            b_st += abs(welch(xs, ys)[0]) >= abs(w0)
        rej[0] += (b_raw + 1) / (nperm + 1) <= 0.05
        rej[1] += (b_st + 1) / (nperm + 1) <= 0.05
    return [r / reps for r in rej]


if __name__ == "__main__":
    rng = random.Random(20261120)
    print("equal means, unequal spread, nominal 0.05, 1000 data sets")
    print("   n1  sd1   n2  sd2   perm raw  perm Welch  Welch t")
    for n1, s1, n2, s2 in ((5, 3.0, 20, 1.0), (10, 3.0, 10, 1.0),
                           (5, 1.0, 20, 3.0)):
        r = one_row(n1, s1, n2, s2, 1000, 99, rng)
        print("  {:3d}  {:.1f}  {:3d}  {:.1f}"
              "     {:.3f}      {:.3f}     {:.3f}"
              .format(n1, s1, n2, s2, r[0], r[1], r[2]))

Single source and citation

Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 5: When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026308,
  author    = {Fan, Yafu},
  title     = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book