What does Synthetic Control for Site Redesigns conclude, in numbers, about “The Placebo Permutation Test: the Post/Pre Ratio, Ranks and the Floor of the p-Value”?

Answer: Chapter 4 gives the placebo permutation test: in the illustrative panel the redesigned site's post/pre ratio ranks second, p is 0.0588, short of five per cent.

Scope of the numbers: On an illustrative panel of thirty-four sites with a true lift of twelve orders a week, before-after reports 37.77, mostly because of the season; weighting thirty-three untouched donor sites with simplex-constrained weights into a synthetic site gives 8.83. Over two hundred panels the root mean squared error is 35.61 for before-after and 5.42 for synthetic control.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0588

ranksitepre RMSPEpost RMSPEpost/pre
1s105.609.781.75
2s005.909.801.66
3s175.499.081.65
4s335.589.191.65
5s307.6912.141.58
6s076.6310.171.53
7s228.3912.491.49
8s257.2810.361.42
redesigned site s00: rank 2 of 34, p = 2/34 = 0.0588
smallest attainable p with 33 donors: 1/34 = 0.0294
ADH 2010 floor: 1/39 = 0.026; 19-state subset 1/20 = 0.050
Show the full program
# Chapter 4. p41: in-space placebo test on the post/pre RMSPE ratio.
# Panel as in p11 (illustrative, true lift +12). The redesigned site
# and each of the 33 untreated sites are fitted in turn (a placebo uses
# the other 32 untreated sites as donors; the redesigned site is never
# a donor). Statistic: RMSPE of the 12 post weeks / RMSPE of the 52 pre
# weeks. p-value = rank of the redesigned site among the 34 ratios,
# divided by 34. Also the published floor of this test in Abadie,
# Diamond & Hainmueller (2010): 38 donor states, p = 1/39; and with the
# 19 states whose pre MSPE was at most twice California's, 1/20.
import math
import random
from operator import mul

# 示意面板:每站周订单 = 基准 + 季节载荷 * 季节 + 大盘载荷 * 大盘 + 噪声
T0, POST, NS = 52, 12, 34


def panel(rep=0, n=NS, t0=T0, post=POST, tau=12.0, start=10):
    rng = random.Random(20261113 + 7919 * rep)
    weeks = t0 + post
    mk = [0.0]
    for _ in range(weeks - 1):
        mk.append(mk[-1] + rng.gauss(0.05, 0.25))
    ss = [math.sin(2.0 * math.pi * (start + t) / 52.0)
          for t in range(weeks)]
    ys, cf, pars = [], [], []
    for i in range(n):
        if i == 0:
            mu, ra, rb, sd = 240.0, 0.20, 1.30, 7.0
        else:
            mu = rng.uniform(120.0, 360.0)
            ra = rng.uniform(0.04, 0.22)
            rb = rng.uniform(0.5, 1.7)
            sd = rng.uniform(5.0, 9.0)
        a, b = mu * ra, mu * rb / 20.0
        y = [mu + a * ss[t] + b * mk[t] + rng.gauss(0.0, sd)
             for t in range(weeks)]
        cf.append(y[:])
        if i == 0:
            for t in range(t0, weeks):
                y[t] = y[t] + tau
        ys.append(y)
        pars.append((mu, a, b, sd))
    return ys, cf, pars, ss, mk


def dot(a, b):
    return math.fsum(map(mul, a, b))


def mean(xs):
    return math.fsum(xs) / len(xs)


def rmspe(xs):
    return math.sqrt(math.fsum([x * x for x in xs]) / len(xs))


def gram(ys, t0):
    n = len(ys)
    pre = [y[:t0] for y in ys]
    g = [[0.0] * n for _ in range(n)]
    for i in range(n):
        for k in range(i, n):
            v = dot(pre[i], pre[k]) / t0
            g[i][k] = v
            g[k][i] = v
    return g


def solve(a, b):
    m = len(b)
    r = [row[:] + [b[i]] for i, row in enumerate(a)]
    for c in range(m):
        p = max(range(c, m), key=lambda i: abs(r[i][c]))
        r[c], r[p] = r[p], r[c]
        for i in range(c + 1, m):
            f = r[i][c] / r[c][c]
            if f:
                r[i] = [x - f * y for x, y in zip(r[i], r[c])]
    x = [0.0] * m
    for i in range(m - 1, -1, -1):
        s = math.fsum([r[i][k] * x[k] for k in range(i + 1, m)])
        x[i] = (r[i][m] - s) / r[i][i]
    return x


def eq_qp(g, c, sup):
    m = len(sup)
    a = [[2.0 * g[i][k] for k in sup] + [1.0] for i in sup]
    a.append([1.0] * m + [0.0])
    b = [2.0 * c[i] for i in sup] + [1.0]
    return solve(a, b)[:m]


def simplex_ls(g, c, tol=1e-10):
    # 精确有效集法:min w'Gw - 2c'w,w >= 0,权重和为 1
    j = len(c)
    k0 = min(range(j), key=lambda k: g[k][k] - 2.0 * c[k])
    w = [0.0] * j
    w[k0] = 1.0
    sup = [k0]
    for _ in range(4 * j + 20):
        gr = [2.0 * (math.fsum([g[i][k] * w[k] for k in sup]) - c[i])
              for i in range(j)]
        nu = -math.fsum([gr[k] for k in sup]) / len(sup)
        out = [k for k in range(j) if k not in sup]
        if not out:
            break
        kb = min(out, key=lambda k: gr[k] + nu)
        if gr[kb] + nu >= -tol:
            break
        sup = sorted(sup + [kb])
        while True:
            z = eq_qp(g, c, sup)
            if min(z) > 0.0:
                for k, v in zip(sup, z):
                    w[k] = v
                break
            step = min(w[k] / (w[k] - v) for k, v in zip(sup, z)
                       if v <= 0.0)
            for k, v in zip(sup, z):
                w[k] = w[k] + step * (v - w[k])
            sup = [k for k in sup if w[k] > 1e-12]
            for k in range(j):
                if k not in sup:
                    w[k] = 0.0
    return w


def synth(ys, g, tr, donors):
    gs = [[g[i][k] for k in donors] for i in donors]
    cs = [g[i][tr] for i in donors]
    w = simplex_ls(gs, cs)
    weeks = len(ys[tr])
    gap = [ys[tr][t] - dot(w, [ys[d][t] for d in donors])
           for t in range(weeks)]
    return w, gap


def ratio(gap, t0=T0):
    return rmspe(gap[t0:]) / rmspe(gap[:t0])


def placebo(ys, g, tr, pool, t0=T0):
    # 逐个捐赠站当处理站;处理站本身不进入安慰剂的捐赠池
    out = {}
    for j in pool:
        dn = [k for k in pool if k != j]
        out[j] = synth(ys, g, j, dn)[1]
    return out


if __name__ == "__main__":
    ys, cf, pars, ss, mk = panel(0)
    pool = list(range(1, NS))
    g = gram(ys, T0)
    gaps = placebo(ys, g, 0, pool)
    gaps[0] = synth(ys, g, 0, pool)[1]
    rows = sorted(((ratio(gp), j, rmspe(gp[:T0]), rmspe(gp[T0:]))
                   for j, gp in gaps.items()), reverse=True)
    print("rank  site  pre RMSPE  post RMSPE  post/pre")
    for k, (rt, j, pr, po) in enumerate(rows[:8]):
        print("{:4d}  s{:02d}   {:8.2f}   {:9.2f}   {:7.2f}".format(
            k + 1, j, pr, po, rt))
    rk = [r[1] for r in rows].index(0) + 1
    print("redesigned site s00: rank {} of {}, p = {}/{} = {:.4f}".format(
        rk, len(rows), rk, len(rows), rk / len(rows)))
    print("smallest attainable p with 33 donors: 1/34 = {:.4f}".format(
        1 / 34))
    print("ADH 2010 floor: 1/39 = {:.3f}; 19-state subset 1/20 = {:.3f}"
          .format(1 / 39, 1 / 20))

Single source and citation

Fan, Y. (2026). Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests (Version 1.0.0), Chapter 4: The Placebo Permutation Test: the Post/Pre Ratio, Ranks and the Floor of the p-Value. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-304 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026304,
  author    = {Fan, Yafu},
  title     = {Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book