What does Resampling for Shop Experiments conclude, in numbers, about “An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book”?

Answer: Chapter 8 gives the analysis template for small shop experiments, the distilled Permutation Tester and a pre-registered refutation plan. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 20

new-layout days: 1 2 5 6 8 10 14
raw mean old 1013.1  new 1092.6  diff 79.4  lift 0.078
secondary, raw: exact p 1698 of 3432 = 0.4948
primary, centred: diff 79.4  exact p 20 of 3432 = 0.0058
95% percentile interval [40.8, 117.6]
95% BCa interval        [38.4, 115.2]
pre-registered decision at 0.05: adopt new layout
Show the full program
# Chapter 8. p81: the shop trial analysis template, end to end.
# Pre-registered before day 1: 14 trading days; 7 drawn by lot (seed
# 20261126) for the new layout; outcome = daily takings in yuan.
# Primary statistic: difference in mean (new minus old) of takings
# centred on their own block mean, blocks fixed in advance (weekend =
# days 6, 7, 13, 14; weekdays = the rest). Centring does not depend on
# the assignment, so the permutation test stays exact. Two-sided exact
# test over all C(14, 7) = 3432 assignments at 0.05; no interim looks.
# Secondary: the raw difference. Always report a 95 percent two-sample
# bootstrap interval of the primary statistic (resampling within each
# arm, B = 4000, percentile and simplified BCa, leave-one-out
# jackknife). Takings are illustrative.
import itertools
import math
import random
from statistics import NormalDist

TAKINGS = [985, 1012, 905, 872, 1040, 1395, 1268,
           905, 958, 951, 921, 866, 1302, 1360]
WEEKEND = (5, 6, 12, 13)
N01 = NormalDist()


def mean(v):
    return math.fsum(v) / len(v)


def quantile(sv, q):
    pos = q * (len(sv) - 1)
    i = int(pos)
    if i >= len(sv) - 1:
        return sv[-1]
    return sv[i] + (pos - i) * (sv[i + 1] - sv[i])


def centred(y):
    wk = [y[i] for i in WEEKEND]
    wd = [y[i] for i in range(len(y)) if i not in WEEKEND]
    mk, md = mean(wk), mean(wd)
    return [y[i] - (mk if i in WEEKEND else md) for i in range(len(y))]


def exact_p(y, new):
    k, tot = len(new), math.fsum(y)
    s0 = math.fsum(y[i] for i in new)
    d0 = abs(s0 / k - (tot - s0) / (len(y) - k))
    ge = cnt = 0
    for idx in itertools.combinations(range(len(y)), k):
        s = math.fsum(y[i] for i in idx)
        cnt += 1
        ge += abs(s / k - (tot - s) / (len(y) - k)) >= d0 - 1e-9
    return ge, cnt


def boot_diff(a, b, nboot, rng):
    out = [mean(rng.choices(b, k=len(b))) - mean(rng.choices(a, k=len(a)))
           for _ in range(nboot)]
    return sorted(out)


def bca_diff(a, b, bd, alpha=0.05):
    th = mean(b) - mean(a)
    below = len([v for v in bd if v < th])
    z0 = N01.inv_cdf(below / len(bd))
    jack = ([mean(b) - mean(a[:i] + a[i + 1:]) for i in range(len(a))]
            + [mean(b[:i] + b[i + 1:]) - mean(a) for i in range(len(b))])
    jm = mean(jack)
    d = [jm - v for v in jack]
    acc = math.fsum(v ** 3 for v in d) / (
        6.0 * math.fsum(v ** 2 for v in d) ** 1.5)
    ends = []
    for q in (alpha / 2, 1 - alpha / 2):
        zq = N01.inv_cdf(q)
        ends.append(quantile(bd, N01.cdf(z0 + (z0 + zq)
                                         / (1 - acc * (z0 + zq)))))
    return ends


if __name__ == "__main__":
    rng = random.Random(20261126)
    new = sorted(d - 1 for d in rng.sample(range(1, 15), 7))
    old = [i for i in range(14) if i not in new]
    ra = [TAKINGS[i] for i in old]
    rb = [TAKINGS[i] for i in new]
    print("new-layout days: {}".format(" ".join(str(i + 1) for i in new)))
    print("raw mean old {:.1f}  new {:.1f}  diff {:.1f}  lift {:.3f}".format(
        mean(ra), mean(rb), mean(rb) - mean(ra), mean(rb) / mean(ra) - 1))
    ge0, cnt = exact_p(TAKINGS, new)
    print("secondary, raw: exact p {} of {} = {:.4f}".format(
        ge0, cnt, ge0 / cnt))
    adj = centred(TAKINGS)
    a = [adj[i] for i in old]
    b = [adj[i] for i in new]
    ge, cnt = exact_p(adj, new)
    print("primary, centred: diff {:.1f}  exact p {} of {} = {:.4f}".format(
        mean(b) - mean(a), ge, cnt, ge / cnt))
    bd = boot_diff(a, b, 4000, random.Random(20261127))
    print("95% percentile interval [{:.1f}, {:.1f}]".format(
        quantile(bd, 0.025), quantile(bd, 0.975)))
    lo, hi = bca_diff(a, b, bd)
    print("95% BCa interval        [{:.1f}, {:.1f}]".format(lo, hi))
    verdict = "adopt new layout" if ge / cnt <= 0.05 else "keep old layout"
    print("pre-registered decision at 0.05: {}".format(verdict))

Single source and citation

Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 8: An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026308,
  author    = {Fan, Yafu},
  title     = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book