What does Resampling for Shop Experiments conclude, in numbers, about “Designing a Shop Experiment: No Peeking, No Early Stopping, Pair First, Size by Simulation”?
Answer: Chapter 6 treats no peeking, no early stopping and pairing: looking every day raises the false alarm rate to 0.1714, and an exact sign-flip test on weekly pairs gives p of 0.0234 against 0.8008 unpaired.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0234, 0.8008
| weekly differences: 55 30 -12 70 25 48 5 38 | |
| mean difference 32.38 | |
| paired sign-flip exact: 6 of 256 | p = 0.0234 |
| unpaired exact: 10306 of 12870 | p = 0.8008 |
| paired t = 3.432 | p = 0.0109 |
| sd between weeks 251.0 | sd of differences 26.7 |
Show the full program
# Chapter 6. p63: pair what varies, then flip signs exactly.
# Model: two near-identical counters in one shop for 8 weeks; each
# week a coin decides which counter shows the new display. Weekly
# takings (yuan, illustrative) swing with holidays, so weeks differ far
# more than counters do. Paired analysis: under H0 the sign of each
# weekly difference is a fair coin, so all 2^8 = 256 sign patterns are
# equally likely (exact sign-flip test). Unpaired analysis of the same
# 16 numbers enumerates all C(16, 8) = 12870 splits. Paired t (p11).
import itertools
import math
def betacf(a, b, x):
tiny = 1e-300
qab, qap, qam = a + b, a + 1.0, a - 1.0
c = 1.0
d = 1.0 - qab * x / qap
d = 1.0 / (d if abs(d) > tiny else tiny)
h = d
for m in range(1, 300):
m2 = 2 * m
aa = m * (b - m) * x / ((qam + m2) * (a + m2))
d = 1.0 + aa * d
d = 1.0 / (d if abs(d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
h *= d * c
aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
d = 1.0 + aa * d
d = 1.0 / (d if abs(d) > tiny else tiny)
c = 1.0 + aa / c
c = c if abs(c) > tiny else tiny
de = d * c
h *= de
if abs(de - 1.0) < 1e-15:
break
return h
def betai(a, b, x):
if x <= 0.0:
return 0.0
if x >= 1.0:
return 1.0
lb = (math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
+ a * math.log(x) + b * math.log(1.0 - x))
bt = math.exp(lb)
if x < (a + 1.0) / (a + b + 2.0):
return bt * betacf(a, b, x) / a
return 1.0 - bt * betacf(b, a, 1.0 - x) / b
def t_two_tail(t, v):
return betai(v / 2.0, 0.5, v / (v + t * t))
def t_cdf(t, v):
half = 0.5 * t_two_tail(t, v)
return 1.0 - half if t > 0 else half
def t_crit(alpha, v):
lo, hi = 0.0, 1000.0
for _ in range(200):
mid = 0.5 * (lo + hi)
if t_two_tail(mid, v) > alpha:
lo = mid
else:
hi = mid
return 0.5 * (lo + hi)
OLD = [820, 1150, 900, 1480, 760, 1010, 1290, 880]
NEW = [875, 1180, 888, 1550, 785, 1058, 1295, 918]
if __name__ == "__main__":
d = [b - a for a, b in zip(OLD, NEW)]
n = len(d)
obs = math.fsum(d)
ge = 0
for signs in itertools.product((1, -1), repeat=n):
ge += abs(math.fsum(s * v for s, v in zip(signs, d))) >= abs(obs)
print("weekly differences: {}".format(" ".join(str(v) for v in d)))
print("mean difference {:.2f}".format(obs / n))
print("paired sign-flip exact: {} of {} p = {:.4f}".format(
ge, 2 ** n, ge / 2 ** n))
pool = OLD + NEW
tot = math.fsum(pool)
d0 = abs(math.fsum(NEW) - math.fsum(OLD))
ge2 = cnt = 0
for idx in itertools.combinations(range(16), 8):
s = math.fsum(pool[i] for i in idx)
cnt += 1
ge2 += abs(2 * s - tot) >= d0 - 1e-9
print("unpaired exact: {} of {} p = {:.4f}".format(ge2, cnt, ge2 / cnt))
m = obs / n
sd = math.sqrt(math.fsum((v - m) ** 2 for v in d) / (n - 1))
t = m / (sd / math.sqrt(n))
print("paired t = {:.3f} p = {:.4f}".format(t, t_two_tail(t, n - 1)))
sdw = math.sqrt(math.fsum((v - math.fsum(OLD) / n) ** 2 for v in OLD)
/ (n - 1))
print("sd between weeks {:.1f} sd of differences {:.1f}".format(sdw, sd))Single source and citation
Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 6: Designing a Shop Experiment: No Peeking, No Early Stopping, Pair First, Size by Simulation. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026308,
author = {Fan, Yafu},
title = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples
- Chapter 2 · From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report
- Chapter 3 · Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule
- Chapter 4 · Bootstrap Intervals: Percentile, Bias-Corrected and Accelerated, and Undercoverage in Small Samples
- Chapter 5 · When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too
- Chapter 7 · Eastern and Western Sources: Tasting One Slice of Meat in the Lüshi Chunqiu and Fisher's Lady Tasting Tea
- Chapter 8 · An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book