What does Resampling for Shop Experiments conclude, in numbers, about “From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report”?

Answer: Chapter 2 moves from the lady's chance of 1/70 to an exact permutation test on twelve trading days, with a two-sided p of 0.0043 where the pooled t test gives 0.1094.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0043, 0.1094

means A 812.50 B 1206.33
mean diff obs393.83 one-sided 1/462 = 0.0022
relabelings 924 two-sided 1/2310.0043
median diff obs191.50 one-sided 1/154 = 0.0065
relabelings 924 two-sided 1/770.0130
pooled t t1.757 df = 10.00 two-sided p = 0.1094
Welch t t1.757 df = 5.12 two-sided p = 0.1378
Welch t without the 2310 day: t3.919 p = 0.0054
Show the full program
# Chapter 2. p22: an exact permutation test on twelve shop days.
# Model: 12 trading days, 6 randomly assigned to the old shelf layout
# (A) and 6 to the new one (B); daily gross takings in yuan are
# illustrative. Under H0 (layout makes no difference to any day) the
# labels are exchangeable, so all C(12, 6) = 924 relabelings are
# equally likely. The p-value is the share of relabelings whose
# statistic is at least as extreme as the observed one, exact as a
# fraction. Compared with pooled and Welch t (t distribution of p11).
import itertools
import math
import statistics
from fractions import Fraction

def betacf(a, b, x):
    tiny = 1e-300
    qab, qap, qam = a + b, a + 1.0, a - 1.0
    c = 1.0
    d = 1.0 - qab * x / qap
    d = 1.0 / (d if abs(d) > tiny else tiny)
    h = d
    for m in range(1, 300):
        m2 = 2 * m
        aa = m * (b - m) * x / ((qam + m2) * (a + m2))
        d = 1.0 + aa * d
        d = 1.0 / (d if abs(d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        h *= d * c
        aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
        d = 1.0 + aa * d
        d = 1.0 / (d if abs(d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        de = d * c
        h *= de
        if abs(de - 1.0) < 1e-15:
            break
    return h


def betai(a, b, x):
    if x <= 0.0:
        return 0.0
    if x >= 1.0:
        return 1.0
    lb = (math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
          + a * math.log(x) + b * math.log(1.0 - x))
    bt = math.exp(lb)
    if x < (a + 1.0) / (a + b + 2.0):
        return bt * betacf(a, b, x) / a
    return 1.0 - bt * betacf(b, a, 1.0 - x) / b


def t_two_tail(t, v):
    return betai(v / 2.0, 0.5, v / (v + t * t))


def t_cdf(t, v):
    half = 0.5 * t_two_tail(t, v)
    return 1.0 - half if t > 0 else half


def t_crit(alpha, v):
    lo, hi = 0.0, 1000.0
    for _ in range(200):
        mid = 0.5 * (lo + hi)
        if t_two_tail(mid, v) > alpha:
            lo = mid
        else:
            hi = mid
    return 0.5 * (lo + hi)

A = [812, 765, 903, 744, 790, 861]
B = [958, 1012, 880, 1105, 973, 2310]


def mean(v):
    return math.fsum(v) / len(v)


def perm_p(x, y, stat):
    pool = x + y
    n = len(pool)
    obs = stat(y, x)
    ge = two = 0
    total = 0
    for idx in itertools.combinations(range(n), len(y)):
        s = set(idx)
        yy = [pool[i] for i in idx]
        xx = [pool[i] for i in range(n) if i not in s]
        d = stat(yy, xx)
        total += 1
        ge += d >= obs - 1e-9
        two += abs(d) >= abs(obs) - 1e-9
    return obs, Fraction(ge, total), Fraction(two, total), total


def welch(x, y):
    nx, ny = len(x), len(y)
    vx, vy = statistics.variance(x), statistics.variance(y)
    se2 = vx / nx + vy / ny
    t = (mean(y) - mean(x)) / math.sqrt(se2)
    df = se2 ** 2 / ((vx / nx) ** 2 / (nx - 1) + (vy / ny) ** 2 / (ny - 1))
    return t, df


def pooled(x, y):
    nx, ny = len(x), len(y)
    ss = math.fsum([(nx - 1) * statistics.variance(x),
                    (ny - 1) * statistics.variance(y)])
    sp2 = ss / (nx + ny - 2)
    t = (mean(y) - mean(x)) / math.sqrt(sp2 * (1 / nx + 1 / ny))
    return t, nx + ny - 2


if __name__ == "__main__":
    md = lambda u, v: mean(u) - mean(v)
    me = lambda u, v: statistics.median(u) - statistics.median(v)
    print("means A {:.2f}  B {:.2f}".format(mean(A), mean(B)))
    for name, f in (("mean diff", md), ("median diff", me)):
        obs, one, two, tot = perm_p(A, B, f)
        print("{:11s} obs {:8.2f}  one-sided {} = {:.4f}".format(
            name, obs, one, float(one)))
        print("{:11s} relabelings {}  two-sided {} = {:.4f}".format(
            "", tot, two, float(two)))
    for name, f in (("pooled t", pooled), ("Welch t", welch)):
        t, df = f(A, B)
        print("{:8s} t = {:.3f}  df = {:.2f}  two-sided p = {:.4f}".format(
            name, t, df, t_two_tail(t, df)))
    t, df = welch(A, B[:5])
    print("Welch t without the 2310 day: t = {:.3f}  p = {:.4f}".format(
        t, t_two_tail(t, df)))

Single source and citation

Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 2: From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026308,
  author    = {Fan, Yafu},
  title     = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book