The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions: chapter answers
Shops run experiments on few and skewed numbers, yet owners habitually click the t test and read the p-value. This book uses programs to overturn the default trust that the t test stays reliable on small samples, while computing its robust side just as carefully. After checking the project blueprint, the book corrects it: the distortion of the t test concentrates in the one-sample case, one-sided tests and unequal groups, and with equal groups, the same shape and two-sided tests the test is conservative; Fisher (1935) and Efron (1979) gave the randomization test and the bootstrap respectively…
DOI pending · monograph page goes live after publication · 5DL-B-308 · 中文版
- Chapter 1 · Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples
Chapter 1 finds actual two-sided false alarm rates of 0.1013 for a one-sample t test on ten exponential observations and 0.1840 on five lognormal ones, falling to conservative levels such as 0.0289 for equal groups of t… - Chapter 2 · From Eight Teacups to Twelve Trading Days: Why the Exact Permutation Test Does Not Over-Report
Chapter 2 moves from the lady's chance of 1/70 to an exact permutation test on twelve trading days, with a two-sided p of 0.0043 where the pooled t test gives 0.1094. - Chapter 3 · Splits Too Many to List: Seeded Monte Carlo Permutation and the Add-One Rule
Chapter 3 explains the add-one rule and the number of shuffles for Monte Carlo permutation: when the exact p is 0.05901, ninety-nine shuffles call it significant with probability 0.299. - Chapter 4 · Bootstrap Intervals: Percentile, Bias-Corrected and Accelerated, and Undercoverage in Small Samples
Chapter 4 gives percentile and simplified BCa bootstrap intervals; with five exponential observations a nominal ninety-five per cent percentile interval covers 0.798 of the time. - Chapter 5 · When Exchangeability Fails: Unequal Variances, Before-and-After Comparisons and Correlated Days Break the Permutation Test Too
Chapter 5 shows that permutation tests also lose control when exchangeability fails: unequal variances with unequal groups give a false alarm rate of 0.250, and a before-and-after comparison gives 0.209 even with no tre… - Chapter 6 · Designing a Shop Experiment: No Peeking, No Early Stopping, Pair First, Size by Simulation
Chapter 6 treats no peeking, no early stopping and pairing: looking every day raises the false alarm rate to 0.1714, and an exact sign-flip test on weekly pairs gives p of 0.0234 against 0.8008 unpaired. - Chapter 7 · Eastern and Western Sources: Tasting One Slice of Meat in the Lüshi Chunqiu and Fisher's Lady Tasting Tea
With d equal to zero, the simulated rejection rates of the four designs are 0.0521, 0.0151, 0.0408 and 0.0358, matching the exact values 0.0500, 0.0143, 0.0400 and 0.0352 in the first row, which shows the simulation is… - Chapter 8 · An Analysis Template for Small Shop Experiments, the Permutation Tester and a Plan to Refute the Book
Chapter 8 gives the analysis template for small shop experiments, the distilled Permutation Tester and a pre-registered refutation plan. Marked boxes separate published evidence, runnable Python calculations (20 program…
Illustrative model calculation, not a live quote