What does Synthetic Control for Site Redesigns conclude, in numbers, about “Closing the Loop: Evaluating Single-Site Redesigns across Thirty-Four Sites, with a Pre-Registered Protocol”?
Answer: Chapter 8 gives a thirty-four-site evaluation sheet, the distilled Synthetic Control Evaluator and a pre-registered protocol: for a lift of five per cent of volume the single-site test has power of about 0.780 at thirteen weeks; several sites can be pooled by the sum of their ranks, and five sites that each rank third give a combined p of 0.000066. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Scope of the numbers: On an illustrative panel of thirty-four sites with a true lift of twelve orders a week, before-after reports 37.77, mostly because of the season; weighting thirty-three untouched donor sites with simplex-constrained weights into a synthetic site gives 8.83. Over two hundred panels the root mean squared error is 35.61 for before-after and 5.42 for synthetic control.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.780
| power to reach p = 1/34, 150 panels | ||||||
| lift | 4wk | 8wk | 13wk | 17wk | 26wk | shortest n for 0.80 |
| 6 | 0.213 | 0.247 | 0.253 | 0.267 | 0.233 | none |
| 9 | 0.293 | 0.433 | 0.533 | 0.500 | 0.500 | none |
| 12 | 0.520 | 0.627 | 0.780 | 0.793 | 0.753 | none |
| 15 | 0.727 | 0.873 | 0.927 | 0.940 | 0.867 | 8 |
Show the full program
# Chapter 8. p82: pre-registered post length for a target power.
# Panel as in p11 (illustrative) with 26 post weeks and true lift 0;
# 150 panels. As in p61 the placebo ratios do not depend on the lift,
# so for each post length n the power to reach p = 1/34 is computed
# for lifts of 6, 9, 12 and 15 orders/week. The pre-registered rule:
# choose the shortest n (from 4, 8, 13, 17, 26) with power >= 0.80;
# "none" if no listed n reaches it.
import math
import random
from operator import mul
# 示意面板:每站周订单 = 基准 + 季节载荷 * 季节 + 大盘载荷 * 大盘 + 噪声
T0, POST, NS = 52, 12, 34
def panel(rep=0, n=NS, t0=T0, post=POST, tau=12.0, start=10):
rng = random.Random(20261113 + 7919 * rep)
weeks = t0 + post
mk = [0.0]
for _ in range(weeks - 1):
mk.append(mk[-1] + rng.gauss(0.05, 0.25))
ss = [math.sin(2.0 * math.pi * (start + t) / 52.0)
for t in range(weeks)]
ys, cf, pars = [], [], []
for i in range(n):
if i == 0:
mu, ra, rb, sd = 240.0, 0.20, 1.30, 7.0
else:
mu = rng.uniform(120.0, 360.0)
ra = rng.uniform(0.04, 0.22)
rb = rng.uniform(0.5, 1.7)
sd = rng.uniform(5.0, 9.0)
a, b = mu * ra, mu * rb / 20.0
y = [mu + a * ss[t] + b * mk[t] + rng.gauss(0.0, sd)
for t in range(weeks)]
cf.append(y[:])
if i == 0:
for t in range(t0, weeks):
y[t] = y[t] + tau
ys.append(y)
pars.append((mu, a, b, sd))
return ys, cf, pars, ss, mk
def dot(a, b):
return math.fsum(map(mul, a, b))
def mean(xs):
return math.fsum(xs) / len(xs)
def rmspe(xs):
return math.sqrt(math.fsum([x * x for x in xs]) / len(xs))
def gram(ys, t0):
n = len(ys)
pre = [y[:t0] for y in ys]
g = [[0.0] * n for _ in range(n)]
for i in range(n):
for k in range(i, n):
v = dot(pre[i], pre[k]) / t0
g[i][k] = v
g[k][i] = v
return g
def solve(a, b):
m = len(b)
r = [row[:] + [b[i]] for i, row in enumerate(a)]
for c in range(m):
p = max(range(c, m), key=lambda i: abs(r[i][c]))
r[c], r[p] = r[p], r[c]
for i in range(c + 1, m):
f = r[i][c] / r[c][c]
if f:
r[i] = [x - f * y for x, y in zip(r[i], r[c])]
x = [0.0] * m
for i in range(m - 1, -1, -1):
s = math.fsum([r[i][k] * x[k] for k in range(i + 1, m)])
x[i] = (r[i][m] - s) / r[i][i]
return x
def eq_qp(g, c, sup):
m = len(sup)
a = [[2.0 * g[i][k] for k in sup] + [1.0] for i in sup]
a.append([1.0] * m + [0.0])
b = [2.0 * c[i] for i in sup] + [1.0]
return solve(a, b)[:m]
def simplex_ls(g, c, tol=1e-10):
# 精确有效集法:min w'Gw - 2c'w,w >= 0,权重和为 1
j = len(c)
k0 = min(range(j), key=lambda k: g[k][k] - 2.0 * c[k])
w = [0.0] * j
w[k0] = 1.0
sup = [k0]
for _ in range(4 * j + 20):
gr = [2.0 * (math.fsum([g[i][k] * w[k] for k in sup]) - c[i])
for i in range(j)]
nu = -math.fsum([gr[k] for k in sup]) / len(sup)
out = [k for k in range(j) if k not in sup]
if not out:
break
kb = min(out, key=lambda k: gr[k] + nu)
if gr[kb] + nu >= -tol:
break
sup = sorted(sup + [kb])
while True:
z = eq_qp(g, c, sup)
if min(z) > 0.0:
for k, v in zip(sup, z):
w[k] = v
break
step = min(w[k] / (w[k] - v) for k, v in zip(sup, z)
if v <= 0.0)
for k, v in zip(sup, z):
w[k] = w[k] + step * (v - w[k])
sup = [k for k in sup if w[k] > 1e-12]
for k in range(j):
if k not in sup:
w[k] = 0.0
return w
def synth(ys, g, tr, donors):
gs = [[g[i][k] for k in donors] for i in donors]
cs = [g[i][tr] for i in donors]
w = simplex_ls(gs, cs)
weeks = len(ys[tr])
gap = [ys[tr][t] - dot(w, [ys[d][t] for d in donors])
for t in range(weeks)]
return w, gap
def ratio(gap, t0=T0):
return rmspe(gap[t0:]) / rmspe(gap[:t0])
def placebo(ys, g, tr, pool, t0=T0):
# 逐个捐赠站当处理站;处理站本身不进入安慰剂的捐赠池
out = {}
for j in pool:
dn = [k for k in pool if k != j]
out[j] = synth(ys, g, j, dn)[1]
return out
if __name__ == "__main__":
pool = list(range(1, NS))
lifts = (6.0, 9.0, 12.0, 15.0)
wins = (4, 8, 13, 17, 26)
hit = {(d, n): 0 for d in lifts for n in wins}
reps = 150
for r in range(reps):
ys, cf, pars, ss, mk = panel(r, post=26, tau=0.0)
g = gram(ys, T0)
gaps = placebo(ys, g, 0, pool)
g0 = synth(ys, g, 0, pool)[1]
pre0 = rmspe(g0[:T0])
for n in wins:
top = max(rmspe(gp[T0:T0 + n]) / rmspe(gp[:T0])
for gp in gaps.values())
for d in lifts:
st = rmspe([x + d for x in g0[T0:T0 + n]]) / pre0
if st > top:
hit[(d, n)] = hit[(d, n)] + 1
print("power to reach p = 1/34, {} panels".format(reps))
print("lift " + " ".join("{:2d}wk ".format(n) for n in wins)
+ " shortest n for 0.80")
for d in lifts:
pw = [hit[(d, n)] / reps for n in wins]
ok = [n for n, p in zip(wins, pw) if p >= 0.80]
print("{:4.0f} ".format(d) + " ".join(
"{:.3f}".format(p) for p in pw) + " {}".format(
ok[0] if ok else "none"))Single source and citation
Fan, Y. (2026). Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests (Version 1.0.0), Chapter 8: Closing the Loop: Evaluating Single-Site Redesigns across Thirty-Four Sites, with a Pre-Registered Protocol. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-304 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026304,
author = {Fan, Yafu},
title = {Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Three Confounders in a Before-After Comparison: How Season, Market and Noise Pose as a Redesign Effect
- Chapter 2 · Simplex-Constrained Least Squares: How Donor Weights Are Found and Why They May Not Be Negative
- Chapter 3 · Pre-Period Fit Is the Entry Ticket: Root Mean Squared Prediction Error, the Convex Hull and Hold-Out Checks
- Chapter 4 · The Placebo Permutation Test: the Post/Pre Ratio, Ranks and the Floor of the p-Value
- Chapter 5 · The Donor Pool: No Spillover, Enough Donors and Interpolation Bias
- Chapter 6 · Power and the Length of the Evaluation: How Many Weeks to See Five Per Cent
- Chapter 7 · Eastern and Western Sources: "Take by Kind, Give by Kind" in the Mozi and Mill's Method of Difference