What does Resampling for Shop Experiments conclude, in numbers, about “An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book”?
Answer: Chapter 8 gives the analysis template for small shop experiments, the distilled Permutation Tester and a pre-registered refutation plan. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 20
new-layout days: 1 2 5 6 8 10 14 raw mean old 1013.1 new 1092.6 diff 79.4 lift 0.078 secondary, raw: exact p 1698 of 3432 = 0.4948 primary, centred: diff 79.4 exact p 20 of 3432 = 0.0058 95% percentile interval [40.8, 117.6] 95% BCa interval [38.4, 115.2] pre-registered decision at 0.05: adopt new layout
Show the full program
# Chapter 8. p81: the shop trial analysis template, end to end.
# Pre-registered before day 1: 14 trading days; 7 drawn by lot (seed
# 20261126) for the new layout; outcome = daily takings in yuan.
# Primary statistic: difference in mean (new minus old) of takings
# centred on their own block mean, blocks fixed in advance (weekend =
# days 6, 7, 13, 14; weekdays = the rest). Centring does not depend on
# the assignment, so the permutation test stays exact. Two-sided exact
# test over all C(14, 7) = 3432 assignments at 0.05; no interim looks.
# Secondary: the raw difference. Always report a 95 percent two-sample
# bootstrap interval of the primary statistic (resampling within each
# arm, B = 4000, percentile and simplified BCa, leave-one-out
# jackknife). Takings are illustrative.
import itertools
import math
import random
from statistics import NormalDist
TAKINGS = [985, 1012, 905, 872, 1040, 1395, 1268,
905, 958, 951, 921, 866, 1302, 1360]
WEEKEND = (5, 6, 12, 13)
N01 = NormalDist()
def mean(v):
return math.fsum(v) / len(v)
def quantile(sv, q):
pos = q * (len(sv) - 1)
i = int(pos)
if i >= len(sv) - 1:
return sv[-1]
return sv[i] + (pos - i) * (sv[i + 1] - sv[i])
def centred(y):
wk = [y[i] for i in WEEKEND]
wd = [y[i] for i in range(len(y)) if i not in WEEKEND]
mk, md = mean(wk), mean(wd)
return [y[i] - (mk if i in WEEKEND else md) for i in range(len(y))]
def exact_p(y, new):
k, tot = len(new), math.fsum(y)
s0 = math.fsum(y[i] for i in new)
d0 = abs(s0 / k - (tot - s0) / (len(y) - k))
ge = cnt = 0
for idx in itertools.combinations(range(len(y)), k):
s = math.fsum(y[i] for i in idx)
cnt += 1
ge += abs(s / k - (tot - s) / (len(y) - k)) >= d0 - 1e-9
return ge, cnt
def boot_diff(a, b, nboot, rng):
out = [mean(rng.choices(b, k=len(b))) - mean(rng.choices(a, k=len(a)))
for _ in range(nboot)]
return sorted(out)
def bca_diff(a, b, bd, alpha=0.05):
th = mean(b) - mean(a)
below = len([v for v in bd if v < th])
z0 = N01.inv_cdf(below / len(bd))
jack = ([mean(b) - mean(a[:i] + a[i + 1:]) for i in range(len(a))]
+ [mean(b[:i] + b[i + 1:]) - mean(a) for i in range(len(b))])
jm = mean(jack)
d = [jm - v for v in jack]
acc = math.fsum(v ** 3 for v in d) / (
6.0 * math.fsum(v ** 2 for v in d) ** 1.5)
ends = []
for q in (alpha / 2, 1 - alpha / 2):
zq = N01.inv_cdf(q)
ends.append(quantile(bd, N01.cdf(z0 + (z0 + zq)
/ (1 - acc * (z0 + zq)))))
return ends
if __name__ == "__main__":
rng = random.Random(20261126)
new = sorted(d - 1 for d in rng.sample(range(1, 15), 7))
old = [i for i in range(14) if i not in new]
ra = [TAKINGS[i] for i in old]
rb = [TAKINGS[i] for i in new]
print("new-layout days: {}".format(" ".join(str(i + 1) for i in new)))
print("raw mean old {:.1f} new {:.1f} diff {:.1f} lift {:.3f}".format(
mean(ra), mean(rb), mean(rb) - mean(ra), mean(rb) / mean(ra) - 1))
ge0, cnt = exact_p(TAKINGS, new)
print("secondary, raw: exact p {} of {} = {:.4f}".format(
ge0, cnt, ge0 / cnt))
adj = centred(TAKINGS)
a = [adj[i] for i in old]
b = [adj[i] for i in new]
ge, cnt = exact_p(adj, new)
print("primary, centred: diff {:.1f} exact p {} of {} = {:.4f}".format(
mean(b) - mean(a), ge, cnt, ge / cnt))
bd = boot_diff(a, b, 4000, random.Random(20261127))
print("95% percentile interval [{:.1f}, {:.1f}]".format(
quantile(bd, 0.025), quantile(bd, 0.975)))
lo, hi = bca_diff(a, b, bd)
print("95% BCa interval [{:.1f}, {:.1f}]".format(lo, hi))
verdict = "adopt new layout" if ge / cnt <= 0.05 else "keep old layout"
print("pre-registered decision at 0.05: {}".format(verdict))Single source and citation
Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 8: An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026308,
author = {Fan, Yafu},
title = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples
- Chapter 2 · From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report
- Chapter 3 · Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule
- Chapter 4 · Bootstrap Intervals: Percentile, Bias-Corrected and Accelerated, and Undercoverage in Small Samples
- Chapter 5 · When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too
- Chapter 6 · Designing a Shop Experiment: No Peeking, No Early Stopping, Pair First, Size by Simulation
- Chapter 7 · Eastern and Western Sources: Tasting One Slice of Meat in the Lüshi Chunqiu and Fisher's Lady Tasting Tea