What does Synthetic Control for Site Redesigns conclude, in numbers, about “The Placebo Permutation Test: the Post/Pre Ratio, Ranks and the Floor of the p-Value”?
Answer: Chapter 4 gives the placebo permutation test: in the illustrative panel the redesigned site's post/pre ratio ranks second, p is 0.0588, short of five per cent.
Scope of the numbers: On an illustrative panel of thirty-four sites with a true lift of twelve orders a week, before-after reports 37.77, mostly because of the season; weighting thirty-three untouched donor sites with simplex-constrained weights into a synthetic site gives 8.83. Over two hundred panels the root mean squared error is 35.61 for before-after and 5.42 for synthetic control.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0588
| rank | site | pre RMSPE | post RMSPE | post/pre |
|---|---|---|---|---|
| 1 | s10 | 5.60 | 9.78 | 1.75 |
| 2 | s00 | 5.90 | 9.80 | 1.66 |
| 3 | s17 | 5.49 | 9.08 | 1.65 |
| 4 | s33 | 5.58 | 9.19 | 1.65 |
| 5 | s30 | 7.69 | 12.14 | 1.58 |
| 6 | s07 | 6.63 | 10.17 | 1.53 |
| 7 | s22 | 8.39 | 12.49 | 1.49 |
| 8 | s25 | 7.28 | 10.36 | 1.42 |
| redesigned site s00: rank 2 of 34, p = 2/34 = 0.0588 | ||||
| smallest attainable p with 33 donors: 1/34 = 0.0294 | ||||
| ADH 2010 floor: 1/39 = 0.026; 19-state subset 1/20 = 0.050 |
Show the full program
# Chapter 4. p41: in-space placebo test on the post/pre RMSPE ratio.
# Panel as in p11 (illustrative, true lift +12). The redesigned site
# and each of the 33 untreated sites are fitted in turn (a placebo uses
# the other 32 untreated sites as donors; the redesigned site is never
# a donor). Statistic: RMSPE of the 12 post weeks / RMSPE of the 52 pre
# weeks. p-value = rank of the redesigned site among the 34 ratios,
# divided by 34. Also the published floor of this test in Abadie,
# Diamond & Hainmueller (2010): 38 donor states, p = 1/39; and with the
# 19 states whose pre MSPE was at most twice California's, 1/20.
import math
import random
from operator import mul
# 示意面板:每站周订单 = 基准 + 季节载荷 * 季节 + 大盘载荷 * 大盘 + 噪声
T0, POST, NS = 52, 12, 34
def panel(rep=0, n=NS, t0=T0, post=POST, tau=12.0, start=10):
rng = random.Random(20261113 + 7919 * rep)
weeks = t0 + post
mk = [0.0]
for _ in range(weeks - 1):
mk.append(mk[-1] + rng.gauss(0.05, 0.25))
ss = [math.sin(2.0 * math.pi * (start + t) / 52.0)
for t in range(weeks)]
ys, cf, pars = [], [], []
for i in range(n):
if i == 0:
mu, ra, rb, sd = 240.0, 0.20, 1.30, 7.0
else:
mu = rng.uniform(120.0, 360.0)
ra = rng.uniform(0.04, 0.22)
rb = rng.uniform(0.5, 1.7)
sd = rng.uniform(5.0, 9.0)
a, b = mu * ra, mu * rb / 20.0
y = [mu + a * ss[t] + b * mk[t] + rng.gauss(0.0, sd)
for t in range(weeks)]
cf.append(y[:])
if i == 0:
for t in range(t0, weeks):
y[t] = y[t] + tau
ys.append(y)
pars.append((mu, a, b, sd))
return ys, cf, pars, ss, mk
def dot(a, b):
return math.fsum(map(mul, a, b))
def mean(xs):
return math.fsum(xs) / len(xs)
def rmspe(xs):
return math.sqrt(math.fsum([x * x for x in xs]) / len(xs))
def gram(ys, t0):
n = len(ys)
pre = [y[:t0] for y in ys]
g = [[0.0] * n for _ in range(n)]
for i in range(n):
for k in range(i, n):
v = dot(pre[i], pre[k]) / t0
g[i][k] = v
g[k][i] = v
return g
def solve(a, b):
m = len(b)
r = [row[:] + [b[i]] for i, row in enumerate(a)]
for c in range(m):
p = max(range(c, m), key=lambda i: abs(r[i][c]))
r[c], r[p] = r[p], r[c]
for i in range(c + 1, m):
f = r[i][c] / r[c][c]
if f:
r[i] = [x - f * y for x, y in zip(r[i], r[c])]
x = [0.0] * m
for i in range(m - 1, -1, -1):
s = math.fsum([r[i][k] * x[k] for k in range(i + 1, m)])
x[i] = (r[i][m] - s) / r[i][i]
return x
def eq_qp(g, c, sup):
m = len(sup)
a = [[2.0 * g[i][k] for k in sup] + [1.0] for i in sup]
a.append([1.0] * m + [0.0])
b = [2.0 * c[i] for i in sup] + [1.0]
return solve(a, b)[:m]
def simplex_ls(g, c, tol=1e-10):
# 精确有效集法:min w'Gw - 2c'w,w >= 0,权重和为 1
j = len(c)
k0 = min(range(j), key=lambda k: g[k][k] - 2.0 * c[k])
w = [0.0] * j
w[k0] = 1.0
sup = [k0]
for _ in range(4 * j + 20):
gr = [2.0 * (math.fsum([g[i][k] * w[k] for k in sup]) - c[i])
for i in range(j)]
nu = -math.fsum([gr[k] for k in sup]) / len(sup)
out = [k for k in range(j) if k not in sup]
if not out:
break
kb = min(out, key=lambda k: gr[k] + nu)
if gr[kb] + nu >= -tol:
break
sup = sorted(sup + [kb])
while True:
z = eq_qp(g, c, sup)
if min(z) > 0.0:
for k, v in zip(sup, z):
w[k] = v
break
step = min(w[k] / (w[k] - v) for k, v in zip(sup, z)
if v <= 0.0)
for k, v in zip(sup, z):
w[k] = w[k] + step * (v - w[k])
sup = [k for k in sup if w[k] > 1e-12]
for k in range(j):
if k not in sup:
w[k] = 0.0
return w
def synth(ys, g, tr, donors):
gs = [[g[i][k] for k in donors] for i in donors]
cs = [g[i][tr] for i in donors]
w = simplex_ls(gs, cs)
weeks = len(ys[tr])
gap = [ys[tr][t] - dot(w, [ys[d][t] for d in donors])
for t in range(weeks)]
return w, gap
def ratio(gap, t0=T0):
return rmspe(gap[t0:]) / rmspe(gap[:t0])
def placebo(ys, g, tr, pool, t0=T0):
# 逐个捐赠站当处理站;处理站本身不进入安慰剂的捐赠池
out = {}
for j in pool:
dn = [k for k in pool if k != j]
out[j] = synth(ys, g, j, dn)[1]
return out
if __name__ == "__main__":
ys, cf, pars, ss, mk = panel(0)
pool = list(range(1, NS))
g = gram(ys, T0)
gaps = placebo(ys, g, 0, pool)
gaps[0] = synth(ys, g, 0, pool)[1]
rows = sorted(((ratio(gp), j, rmspe(gp[:T0]), rmspe(gp[T0:]))
for j, gp in gaps.items()), reverse=True)
print("rank site pre RMSPE post RMSPE post/pre")
for k, (rt, j, pr, po) in enumerate(rows[:8]):
print("{:4d} s{:02d} {:8.2f} {:9.2f} {:7.2f}".format(
k + 1, j, pr, po, rt))
rk = [r[1] for r in rows].index(0) + 1
print("redesigned site s00: rank {} of {}, p = {}/{} = {:.4f}".format(
rk, len(rows), rk, len(rows), rk / len(rows)))
print("smallest attainable p with 33 donors: 1/34 = {:.4f}".format(
1 / 34))
print("ADH 2010 floor: 1/39 = {:.3f}; 19-state subset 1/20 = {:.3f}"
.format(1 / 39, 1 / 20))Single source and citation
Fan, Y. (2026). Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests (Version 1.0.0), Chapter 4: The Placebo Permutation Test: the Post/Pre Ratio, Ranks and the Floor of the p-Value. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-304 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026304,
author = {Fan, Yafu},
title = {Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Three Confounders in a Before-After Comparison: How Season, Market and Noise Pose as a Redesign Effect
- Chapter 2 · Simplex-Constrained Least Squares: How Donor Weights Are Found and Why They May Not Be Negative
- Chapter 3 · Pre-Period Fit Is the Entry Ticket: Root Mean Squared Prediction Error, the Convex Hull and Hold-Out Checks
- Chapter 5 · The Donor Pool: No Spillover, Enough Donors and Interpolation Bias
- Chapter 6 · Power and the Length of the Evaluation: How Many Weeks to See Five Per Cent
- Chapter 7 · Eastern and Western Sources: "Take by Kind, Give by Kind" in the Mozi and Mill's Method of Difference
- Chapter 8 · Closing the Loop: Evaluating Single-Site Redesigns across Thirty-Four Sites, with a Pre-Registered Protocol