What does Synthetic Control for Site Redesigns conclude, in numbers, about “Power and the Length of the Evaluation: How Many Weeks to See Five Per Cent”?

Answer: Start with the first row, zero lift. Requiring first place, the false alarm rate is 0.053 at four weeks, 0.040 at eight and 0.020 at twelve, near or below the nominal level, in line with Chapter 4. With a lift of 6 orders a week, two and a half per cent of size, power is only 0.240 to 0.267 and barely grows with more weeks. With 12 orders it is 0.633 at four weeks, 0.733 at eight and 0.773 at twelve; allowing the top two, it reaches 0.867 at twelve weeks. With 18 orders, power is already 0.933 at four weeks. With 24 orders it is 1.000 at twelve weeks.

Scope of the numbers: On an illustrative panel of thirty-four sites with a true lift of twelve orders a week, before-after reports 37.77, mostly because of the season; weighting thirty-three untouched donor sites with simplex-constrained weights into a synthetic site gives 8.83. Over two hundred panels the root mean squared error is 35.61 for before-after and 5.42 for synthetic control.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.020, 0.040, 0.053, 0.240, 0.267, 0.633

power over 150 panels; rows = lift, cols = post weeks
liftp<=0.0294: 4wk8wk12wkp<=0.0588: 4wk8wk12wk
00.0530.0400.0200.0870.0670.027
60.2400.2530.2670.3330.4130.427
120.6330.7330.7730.7670.8330.867
180.9330.9400.9470.9800.9870.993
240.9730.9931.0001.0001.0001.000
Show the full program
# Chapter 6. p61: power of the placebo test by lift and post length.
# Panel as in p11 (illustrative) with true lift 0; 150 panels. The
# synthetic weights depend only on pre weeks, so a lift d added after
# launch shifts the redesigned site's post gap by exactly d while the
# 33 placebo ratios stay as they are. Rejection: the redesigned site's
# post/pre ratio is the largest of 34 (p = 1/34 = 0.0294) or among the
# two largest (p = 2/34 = 0.0588). Post windows: first 4, 8 or 12
# post weeks; lifts 0 to 24 orders/week (5 per cent of 240 is 12).
import math
import random
from operator import mul

# 示意面板:每站周订单 = 基准 + 季节载荷 * 季节 + 大盘载荷 * 大盘 + 噪声
T0, POST, NS = 52, 12, 34


def panel(rep=0, n=NS, t0=T0, post=POST, tau=12.0, start=10):
    rng = random.Random(20261113 + 7919 * rep)
    weeks = t0 + post
    mk = [0.0]
    for _ in range(weeks - 1):
        mk.append(mk[-1] + rng.gauss(0.05, 0.25))
    ss = [math.sin(2.0 * math.pi * (start + t) / 52.0)
          for t in range(weeks)]
    ys, cf, pars = [], [], []
    for i in range(n):
        if i == 0:
            mu, ra, rb, sd = 240.0, 0.20, 1.30, 7.0
        else:
            mu = rng.uniform(120.0, 360.0)
            ra = rng.uniform(0.04, 0.22)
            rb = rng.uniform(0.5, 1.7)
            sd = rng.uniform(5.0, 9.0)
        a, b = mu * ra, mu * rb / 20.0
        y = [mu + a * ss[t] + b * mk[t] + rng.gauss(0.0, sd)
             for t in range(weeks)]
        cf.append(y[:])
        if i == 0:
            for t in range(t0, weeks):
                y[t] = y[t] + tau
        ys.append(y)
        pars.append((mu, a, b, sd))
    return ys, cf, pars, ss, mk


def dot(a, b):
    return math.fsum(map(mul, a, b))


def mean(xs):
    return math.fsum(xs) / len(xs)


def rmspe(xs):
    return math.sqrt(math.fsum([x * x for x in xs]) / len(xs))


def gram(ys, t0):
    n = len(ys)
    pre = [y[:t0] for y in ys]
    g = [[0.0] * n for _ in range(n)]
    for i in range(n):
        for k in range(i, n):
            v = dot(pre[i], pre[k]) / t0
            g[i][k] = v
            g[k][i] = v
    return g


def solve(a, b):
    m = len(b)
    r = [row[:] + [b[i]] for i, row in enumerate(a)]
    for c in range(m):
        p = max(range(c, m), key=lambda i: abs(r[i][c]))
        r[c], r[p] = r[p], r[c]
        for i in range(c + 1, m):
            f = r[i][c] / r[c][c]
            if f:
                r[i] = [x - f * y for x, y in zip(r[i], r[c])]
    x = [0.0] * m
    for i in range(m - 1, -1, -1):
        s = math.fsum([r[i][k] * x[k] for k in range(i + 1, m)])
        x[i] = (r[i][m] - s) / r[i][i]
    return x


def eq_qp(g, c, sup):
    m = len(sup)
    a = [[2.0 * g[i][k] for k in sup] + [1.0] for i in sup]
    a.append([1.0] * m + [0.0])
    b = [2.0 * c[i] for i in sup] + [1.0]
    return solve(a, b)[:m]


def simplex_ls(g, c, tol=1e-10):
    # 精确有效集法:min w'Gw - 2c'w,w >= 0,权重和为 1
    j = len(c)
    k0 = min(range(j), key=lambda k: g[k][k] - 2.0 * c[k])
    w = [0.0] * j
    w[k0] = 1.0
    sup = [k0]
    for _ in range(4 * j + 20):
        gr = [2.0 * (math.fsum([g[i][k] * w[k] for k in sup]) - c[i])
              for i in range(j)]
        nu = -math.fsum([gr[k] for k in sup]) / len(sup)
        out = [k for k in range(j) if k not in sup]
        if not out:
            break
        kb = min(out, key=lambda k: gr[k] + nu)
        if gr[kb] + nu >= -tol:
            break
        sup = sorted(sup + [kb])
        while True:
            z = eq_qp(g, c, sup)
            if min(z) > 0.0:
                for k, v in zip(sup, z):
                    w[k] = v
                break
            step = min(w[k] / (w[k] - v) for k, v in zip(sup, z)
                       if v <= 0.0)
            for k, v in zip(sup, z):
                w[k] = w[k] + step * (v - w[k])
            sup = [k for k in sup if w[k] > 1e-12]
            for k in range(j):
                if k not in sup:
                    w[k] = 0.0
    return w


def synth(ys, g, tr, donors):
    gs = [[g[i][k] for k in donors] for i in donors]
    cs = [g[i][tr] for i in donors]
    w = simplex_ls(gs, cs)
    weeks = len(ys[tr])
    gap = [ys[tr][t] - dot(w, [ys[d][t] for d in donors])
           for t in range(weeks)]
    return w, gap


def ratio(gap, t0=T0):
    return rmspe(gap[t0:]) / rmspe(gap[:t0])


def placebo(ys, g, tr, pool, t0=T0):
    # 逐个捐赠站当处理站;处理站本身不进入安慰剂的捐赠池
    out = {}
    for j in pool:
        dn = [k for k in pool if k != j]
        out[j] = synth(ys, g, j, dn)[1]
    return out


if __name__ == "__main__":
    pool = list(range(1, NS))
    lifts = (0.0, 6.0, 12.0, 18.0, 24.0)
    wins = (4, 8, 12)
    hit1 = {(d, n): 0 for d in lifts for n in wins}
    hit2 = {(d, n): 0 for d in lifts for n in wins}
    reps = 150
    for r in range(reps):
        ys, cf, pars, ss, mk = panel(r, tau=0.0)
        g = gram(ys, T0)
        gaps = placebo(ys, g, 0, pool)
        g0 = synth(ys, g, 0, pool)[1]
        pre0 = rmspe(g0[:T0])
        for n in wins:
            pr = sorted((rmspe(gp[T0:T0 + n]) / rmspe(gp[:T0])
                         for gp in gaps.values()), reverse=True)
            for d in lifts:
                st = rmspe([x + d for x in g0[T0:T0 + n]]) / pre0
                if st > pr[0]:
                    hit1[(d, n)] = hit1[(d, n)] + 1
                if st > pr[1]:
                    hit2[(d, n)] = hit2[(d, n)] + 1
    print("power over {} panels; rows = lift, cols = post weeks".format(
        reps))
    print("lift   p<=0.0294: 4wk   8wk  12wk   p<=0.0588: 4wk   8wk  12wk")
    for d in lifts:
        a = "  ".join("{:.3f}".format(hit1[(d, n)] / reps) for n in wins)
        b = "  ".join("{:.3f}".format(hit2[(d, n)] / reps) for n in wins)
        print("{:4.0f}            {}             {}".format(d, a, b))

Single source and citation

Fan, Y. (2026). Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests (Version 1.0.0), Chapter 6: Power and the Length of the Evaluation: How Many Weeks to See Five Per Cent. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-304 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026304,
  author    = {Fan, Yafu},
  title     = {Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book