What does Resampling for Shop Experiments conclude, in numbers, about “Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples”?

Answer: Chapter 1 finds actual two-sided false alarm rates of 0.1013 for a one-sample t test on ten exponential observations and 0.1840 on five lognormal ones, falling to conservative levels such as 0.0289 for equal groups of the same shape.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.1013, 0.1840

one-sample t, nominal two-sided 0.050 (0.025 per tail)
populationnlow tailhigh tailtwo-sided
exp(1)50.11450.00400.1185
exp(1)100.09740.00400.1013
exp(1)200.07740.00600.0834
exp(1)400.06040.00860.0689
logn(0,1)50.18250.00150.1840
logn(0,1)100.16120.00110.1623
logn(0,1)200.13360.00170.1353
logn(0,1)400.10920.00240.1116
Show the full program
# Chapter 1. p12: actual type I error of the one-sample t test, skewed.
# Model: n draws from a skewed population whose true mean is known;
# test H0: mean = true mean with the two-sided t test at nominal 0.05
# (each tail nominal 0.025). Populations: exponential(1), mean 1, and
# lognormal(0, 1), mean exp(0.5). Critical values come from the
# hand-written t distribution of p11 (continued fraction, bisection).
# 20000 seeded replicates per cell; Monte Carlo standard error of a
# rate near 0.05 is about 0.0015. Sample sizes are illustrative.
import math
import random

def betacf(a, b, x):
    tiny = 1e-300
    qab, qap, qam = a + b, a + 1.0, a - 1.0
    c = 1.0
    d = 1.0 - qab * x / qap
    d = 1.0 / (d if abs(d) > tiny else tiny)
    h = d
    for m in range(1, 300):
        m2 = 2 * m
        aa = m * (b - m) * x / ((qam + m2) * (a + m2))
        d = 1.0 + aa * d
        d = 1.0 / (d if abs(d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        h *= d * c
        aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2))
        d = 1.0 + aa * d
        d = 1.0 / (d if abs(d) > tiny else tiny)
        c = 1.0 + aa / c
        c = c if abs(c) > tiny else tiny
        de = d * c
        h *= de
        if abs(de - 1.0) < 1e-15:
            break
    return h


def betai(a, b, x):
    if x <= 0.0:
        return 0.0
    if x >= 1.0:
        return 1.0
    lb = (math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
          + a * math.log(x) + b * math.log(1.0 - x))
    bt = math.exp(lb)
    if x < (a + 1.0) / (a + b + 2.0):
        return bt * betacf(a, b, x) / a
    return 1.0 - bt * betacf(b, a, 1.0 - x) / b


def t_two_tail(t, v):
    return betai(v / 2.0, 0.5, v / (v + t * t))


def t_cdf(t, v):
    half = 0.5 * t_two_tail(t, v)
    return 1.0 - half if t > 0 else half


def t_crit(alpha, v):
    lo, hi = 0.0, 1000.0
    for _ in range(200):
        mid = 0.5 * (lo + hi)
        if t_two_tail(mid, v) > alpha:
            lo = mid
        else:
            hi = mid
    return 0.5 * (lo + hi)


def t_stat(x, mu):
    n = len(x)
    m = math.fsum(x) / n
    s2 = math.fsum((v - m) ** 2 for v in x) / (n - 1)
    return (m - mu) / math.sqrt(s2 / n)


def run(draw, mu, n, reps, rng, crit):
    lo = hi = 0
    for _ in range(reps):
        t = t_stat([draw(rng) for _ in range(n)], mu)
        lo += t < -crit
        hi += t > crit
    return lo / reps, hi / reps


if __name__ == "__main__":
    rng = random.Random(20261113)
    reps = 20000
    pops = (("exp(1)", lambda r: r.expovariate(1.0), 1.0),
            ("logn(0,1)", lambda r: r.lognormvariate(0.0, 1.0),
             math.exp(0.5)))
    print("one-sample t, nominal two-sided 0.050 (0.025 per tail)")
    print("  population   n   low tail  high tail  two-sided")
    for name, draw, mu in pops:
        for n in (5, 10, 20, 40):
            crit = t_crit(0.05, n - 1)
            lo, hi = run(draw, mu, n, reps, rng, crit)
            print("  {:10s} {:3d}    {:.4f}     {:.4f}     {:.4f}".format(
                name, n, lo, hi, lo + hi))

Single source and citation

Fan, Y. (2026). The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions (Version 1.0.0), Chapter 1: Nominal Five Per Cent, Actual False Alarms: the Type I Error of the t Test on Small Skewed Samples. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-308 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026308,
  author    = {Fan, Yafu},
  title     = {The Bootstrap and Permutation Tests: Resampling Inference for Small Shop Experiments without Normality Assumptions},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book