What does Resampling for Shop Experiments conclude, in numbers, about “From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report”?
Answer: Chapter 2 moves from the lady's chance of 1/70 to an exact permutation test on twelve trading days, with a two-sided p of 0.0043 where the pooled t test gives 0.1094.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0043, 0.1094
| means A 812.50 B 1206.33 | |
| mean diff obs | 393.83 one-sided 1/462 = 0.0022 |
| relabelings 924 two-sided 1/231 | 0.0043 |
| median diff obs | 191.50 one-sided 1/154 = 0.0065 |
| relabelings 924 two-sided 1/77 | 0.0130 |
| pooled t t | 1.757 df = 10.00 two-sided p = 0.1094 |
| Welch t t | 1.757 df = 5.12 two-sided p = 0.1378 |
| Welch t without the 2310 day: t | 3.919 p = 0.0054 |
Show the full program
# Chapter 2. p22: an exact permutation test on twelve shop days.
# Model: 12 trading days, 6 randomly assigned to the old shelf layout
# (A) and 6 to the new one (B); daily gross takings in yuan are
# illustrative. Under H0 (layout makes no difference to any day) the
# labels are exchangeable, so all C(12, 6) = 924 relabelings are
# equally likely. The p-value is the share of relabelings whose
# statistic is at least as extreme as the observed one, exact as a
# fraction. Compared with pooled and Welch t (t distribution of p11).
import itertools
import math
import statistics
from fractions import Fraction
def betacf(a, b, x):
tiny = 1e-300
qab, qap, qam = a + b, a + 1.0, a - 1.0
c = 1.0
d = 1.0 - qab * x / qap
d = 1.0 / (d if abs(d) > tiny else tiny)
h = d
for m in range(1, 300):
m2 = 2 * m
aa = m * (b - m) * x / ((qam + m2) * (a + m2))
d = 1.0 + aa * d
d = 1.0 / (d if abs(d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
h *= d * c
aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
d = 1.0 + aa * d
d = 1.0 / (d if abs(d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
de = d * c
h *= de
if abs(de - 1.0) < 1e-15:
break
return h
def betai(a, b, x):
if x <= 0.0:
return 0.0
if x >= 1.0:
return 1.0
lb = (math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
+ a * math.log(x) + b * math.log(1.0 - x))
bt = math.exp(lb)
if x < (a + 1.0) / (a + b + 2.0):
return bt * betacf(a, b, x) / a
return 1.0 - bt * betacf(b, a, 1.0 - x) / b
def t_two_tail(t, v):
return betai(v / 2.0, 0.5, v / (v + t * t))
def t_cdf(t, v):
half = 0.5 * t_two_tail(t, v)
return 1.0 - half if t > 0 else half
def t_crit(alpha, v):
lo, hi = 0.0, 1000.0
for _ in range(200):
mid = 0.5 * (lo + hi)
if t_two_tail(mid, v) > alpha:
lo = mid
else:
hi = mid
return 0.5 * (lo + hi)
A = [812, 765, 903, 744, 790, 861]
B = [958, 1012, 880, 1105, 973, 2310]
def mean(v):
return math.fsum(v) / len(v)
def perm_p(x, y, stat):
pool = x + y
n = len(pool)
obs = stat(y, x)
ge = two = 0
total = 0
for idx in itertools.combinations(range(n), len(y)):
s = set(idx)
yy = [pool[i] for i in idx]
xx = [pool[i] for i in range(n) if i not in s]
d = stat(yy, xx)
total += 1
ge += d >= obs - 1e-9
two += abs(d) >= abs(obs) - 1e-9
return obs, Fraction(ge, total), Fraction(two, total), total
def welch(x, y):
nx, ny = len(x), len(y)
vx, vy = statistics.variance(x), statistics.variance(y)
se2 = vx / nx + vy / ny
t = (mean(y) - mean(x)) / math.sqrt(se2)
df = se2 ** 2 / ((vx / nx) ** 2 / (nx - 1) + (vy / ny) ** 2 / (ny - 1))
return t, df
def pooled(x, y):
nx, ny = len(x), len(y)
ss = math.fsum([(nx - 1) * statistics.variance(x),
(ny - 1) * statistics.variance(y)])
sp2 = ss / (nx + ny - 2)
t = (mean(y) - mean(x)) / math.sqrt(sp2 * (1 / nx + 1 / ny))
return t, nx + ny - 2
if __name__ == "__main__":
md = lambda u, v: mean(u) - mean(v)
me = lambda u, v: statistics.median(u) - statistics.median(v)
print("means A {:.2f} B {:.2f}".format(mean(A), mean(B)))
for name, f in (("mean diff", md), ("median diff", me)):
obs, one, two, tot = perm_p(A, B, f)
print("{:11s} obs {:8.2f} one-sided {} = {:.4f}".format(
name, obs, one, float(one)))
print("{:11s} relabelings {} two-sided {} = {:.4f}".format(
"", tot, two, float(two)))
for name, f in (("pooled t", pooled), ("Welch t", welch)):
t, df = f(A, B)
print("{:8s} t = {:.3f} df = {:.2f} two-sided p = {:.4f}".format(
name, t, df, t_two_tail(t, df)))
t, df = welch(A, B[:5])
print("Welch t without the 2310 day: t = {:.3f} p = {:.4f}".format(
t, t_two_tail(t, df)))Single source and citation
Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 2: From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026308,
author = {Fan, Yafu},
title = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples
- Chapter 3 · Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule
- Chapter 4 · Bootstrap Intervals: Percentile, Bias-Corrected and Accelerated, and Undercoverage in Small Samples
- Chapter 5 · When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too
- Chapter 6 · Designing a Shop Experiment: No Peeking, No Early Stopping, Pair First, Size by Simulation
- Chapter 7 · Eastern and Western Sources: Tasting One Slice of Meat in the Lüshi Chunqiu and Fisher's Lady Tasting Tea
- Chapter 8 · An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book