What does Synthetic Control for Site Redesigns conclude, in numbers, about “Pre-Period Fit Is the Entry Ticket: Root Mean Squared Prediction Error, the Convex Hull and Hold-Out Checks”?
Answer: On panel zero, synthetic control has a fitting error of 6.39 and a hold-out error of 4.00. Free regression fits to 3.11 but holds out at 15.51. The averages over two hundred panels are clearer: synthetic control fits at 6.57 and holds out at 9.29, a ratio of 1.41; free regression fits at 2.96 and holds out at 19.72, a ratio of 6.66. The plain donor mean fits at 18.09 and holds out at 19.55; the shifted mean fits at 13.63 and holds out at 15.23; the best single donor fits at 13.82 and holds out at 16.39. The last three have ratios close to one because they learn almost nothing during fitting and so cannot overfit, but their absolute errors are all much larger than those of synthetic control.
Scope of the numbers: On an illustrative panel of thirty-four sites with a true lift of twelve orders a week, before-after reports 37.77, mostly because of the season; weighting thirty-three untouched donor sites with simplex-constrained weights into a synthetic site gives 8.83. Over two hundred panels the root mean squared error is 35.61 for before-after and 5.42 for synthetic control.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 1.41, 13.63, 13.82, 15.23, 15.51, 16.39
| panel 0 method fit wk 0-39 check wk 40-51 | |
| simplex | 6.39 4.00 |
| free OLS | 3.11 15.51 |
| donor mean | 11.52 13.72 |
| mean+shift | 11.50 14.24 |
| best single | 13.63 19.30 |
| 200 panels, means: | |
| simplex | 6.57 9.29 ratio 1.41 |
| free OLS | 2.96 19.72 ratio 6.66 |
| donor mean | 18.09 19.55 ratio 1.08 |
| mean+shift | 13.63 15.23 ratio 1.12 |
| best single | 13.82 16.39 ratio 1.19 |
Show the full program
# Chapter 3. p33: hold out the last 12 pre weeks before believing a fit.
# Panel as in p11 (illustrative). Weights are fitted on weeks 0-39
# only and judged on weeks 40-51, still before the redesign. Methods:
# synthetic control (simplex), free OLS, the plain donor mean, the
# donor mean shifted to the site's level (a DiD-style comparison), and
# the single best donor. One panel, then means over 200 panels.
import math
import random
from operator import mul
# 示意面板:每站周订单 = 基准 + 季节载荷 * 季节 + 大盘载荷 * 大盘 + 噪声
T0, POST, NS = 52, 12, 34
def panel(rep=0, n=NS, t0=T0, post=POST, tau=12.0, start=10):
rng = random.Random(20261113 + 7919 * rep)
weeks = t0 + post
mk = [0.0]
for _ in range(weeks - 1):
mk.append(mk[-1] + rng.gauss(0.05, 0.25))
ss = [math.sin(2.0 * math.pi * (start + t) / 52.0)
for t in range(weeks)]
ys, cf, pars = [], [], []
for i in range(n):
if i == 0:
mu, ra, rb, sd = 240.0, 0.20, 1.30, 7.0
else:
mu = rng.uniform(120.0, 360.0)
ra = rng.uniform(0.04, 0.22)
rb = rng.uniform(0.5, 1.7)
sd = rng.uniform(5.0, 9.0)
a, b = mu * ra, mu * rb / 20.0
y = [mu + a * ss[t] + b * mk[t] + rng.gauss(0.0, sd)
for t in range(weeks)]
cf.append(y[:])
if i == 0:
for t in range(t0, weeks):
y[t] = y[t] + tau
ys.append(y)
pars.append((mu, a, b, sd))
return ys, cf, pars, ss, mk
def dot(a, b):
return math.fsum(map(mul, a, b))
def mean(xs):
return math.fsum(xs) / len(xs)
def rmspe(xs):
return math.sqrt(math.fsum([x * x for x in xs]) / len(xs))
def gram(ys, t0):
n = len(ys)
pre = [y[:t0] for y in ys]
g = [[0.0] * n for _ in range(n)]
for i in range(n):
for k in range(i, n):
v = dot(pre[i], pre[k]) / t0
g[i][k] = v
g[k][i] = v
return g
def solve(a, b):
m = len(b)
r = [row[:] + [b[i]] for i, row in enumerate(a)]
for c in range(m):
p = max(range(c, m), key=lambda i: abs(r[i][c]))
r[c], r[p] = r[p], r[c]
for i in range(c + 1, m):
f = r[i][c] / r[c][c]
if f:
r[i] = [x - f * y for x, y in zip(r[i], r[c])]
x = [0.0] * m
for i in range(m - 1, -1, -1):
s = math.fsum([r[i][k] * x[k] for k in range(i + 1, m)])
x[i] = (r[i][m] - s) / r[i][i]
return x
def eq_qp(g, c, sup):
m = len(sup)
a = [[2.0 * g[i][k] for k in sup] + [1.0] for i in sup]
a.append([1.0] * m + [0.0])
b = [2.0 * c[i] for i in sup] + [1.0]
return solve(a, b)[:m]
def simplex_ls(g, c, tol=1e-10):
# 精确有效集法:min w'Gw - 2c'w,w >= 0,权重和为 1
j = len(c)
k0 = min(range(j), key=lambda k: g[k][k] - 2.0 * c[k])
w = [0.0] * j
w[k0] = 1.0
sup = [k0]
for _ in range(4 * j + 20):
gr = [2.0 * (math.fsum([g[i][k] * w[k] for k in sup]) - c[i])
for i in range(j)]
nu = -math.fsum([gr[k] for k in sup]) / len(sup)
out = [k for k in range(j) if k not in sup]
if not out:
break
kb = min(out, key=lambda k: gr[k] + nu)
if gr[kb] + nu >= -tol:
break
sup = sorted(sup + [kb])
while True:
z = eq_qp(g, c, sup)
if min(z) > 0.0:
for k, v in zip(sup, z):
w[k] = v
break
step = min(w[k] / (w[k] - v) for k, v in zip(sup, z)
if v <= 0.0)
for k, v in zip(sup, z):
w[k] = w[k] + step * (v - w[k])
sup = [k for k in sup if w[k] > 1e-12]
for k in range(j):
if k not in sup:
w[k] = 0.0
return w
def synth(ys, g, tr, donors):
gs = [[g[i][k] for k in donors] for i in donors]
cs = [g[i][tr] for i in donors]
w = simplex_ls(gs, cs)
weeks = len(ys[tr])
gap = [ys[tr][t] - dot(w, [ys[d][t] for d in donors])
for t in range(weeks)]
return w, gap
def ratio(gap, t0=T0):
return rmspe(gap[t0:]) / rmspe(gap[:t0])
def placebo(ys, g, tr, pool, t0=T0):
# 逐个捐赠站当处理站;处理站本身不进入安慰剂的捐赠池
out = {}
for j in pool:
dn = [k for k in pool if k != j]
out[j] = synth(ys, g, j, dn)[1]
return out
def predict(ys, method, donors, t_fit):
g = gram(ys, t_fit)
gs = [[g[i][k] for k in donors] for i in donors]
cs = [g[i][0] for i in donors]
weeks = len(ys[0])
if method == "simplex":
w = simplex_ls(gs, cs)
elif method == "free OLS":
w = solve(gs, cs)
elif method == "best single":
k = min(range(len(donors)),
key=lambda k: gs[k][k] - 2.0 * cs[k])
w = [0.0] * len(donors)
w[k] = 1.0
else:
w = [1.0 / len(donors)] * len(donors)
yh = [dot(w, [ys[d][t] for d in donors]) for t in range(weeks)]
if method == "mean+shift":
sh = mean([ys[0][t] - yh[t] for t in range(t_fit)])
yh = [v + sh for v in yh]
return yh
if __name__ == "__main__":
donors = list(range(1, NS))
meths = ("simplex", "free OLS", "donor mean", "mean+shift",
"best single")
acc = {m: ([], []) for m in meths}
for r in range(200):
ys, cf, pars, ss, mk = panel(r)
for m in meths:
yh = predict(ys, m, donors, 40)
fit = rmspe([ys[0][t] - yh[t] for t in range(40)])
val = rmspe([ys[0][t] - yh[t] for t in range(40, T0)])
acc[m][0].append(fit)
acc[m][1].append(val)
if r == 0 and m == meths[0]:
print("panel 0 method fit wk 0-39 check wk 40-51")
if r == 0:
print(" {:<12s} {:10.2f} {:13.2f}".format(
m, fit, val))
print("200 panels, means:")
for m in meths:
f, v = acc[m]
print(" {:<12s} {:10.2f} {:13.2f} ratio {:.2f}".format(
m, mean(f), mean(v), mean(v) / mean(f)))Single source and citation
Fan, Y. (2026). Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests (Version 1.0.0), Chapter 3: Pre-Period Fit Is the Entry Ticket: Root Mean Squared Prediction Error, the Convex Hull and Hold-Out Checks. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-304 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026304,
author = {Fan, Yafu},
title = {Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Three Confounders in a Before-After Comparison: How Season, Market and Noise Pose as a Redesign Effect
- Chapter 2 · Simplex-Constrained Least Squares: How Donor Weights Are Found and Why They May Not Be Negative
- Chapter 4 · The Placebo Permutation Test: the Post/Pre Ratio, Ranks and the Floor of the p-Value
- Chapter 5 · The Donor Pool: No Spillover, Enough Donors and Interpolation Bias
- Chapter 6 · Power and the Length of the Evaluation: How Many Weeks to See Five Per Cent
- Chapter 7 · Eastern and Western Sources: "Take by Kind, Give by Kind" in the Mozi and Mill's Method of Difference
- Chapter 8 · Closing the Loop: Evaluating Single-Site Redesigns across Thirty-Four Sites, with a Pre-Registered Protocol