Synthetic Control and the Causal Effect of a Single-Site Redesign: Rebuilding the Counterfactual from Weighted Donor Sites and Placebo Permutation Tests: chapter answers
People who run several sites commonly judge a single-site redesign by taking a few weeks before and after the change and subtracting. This book tests that habit with reproducible calculation. On an illustrative panel of thirty-four sites with a true lift of twelve orders a week, before-after reports 37.77, mostly because of the season; weighting thirty-three untouched donor sites with simplex-constrained weights into a synthetic site gives 8.83. Over two hundred panels the root mean squared error is 35.61 for before-after and 5.42 for synthetic control. The method is the synthetic control of…
DOI pending · monograph page goes live after publication · 5DL-B-304 · 中文版
- Chapter 1 · Three Confounders in a Before-After Comparison: How Season, Market and Noise Pose as a Redesign Effect
Before-after has a mean estimate of 45.64, a bias of 33.64 and a root mean squared error of 35.61. In 0.980 of the panels its error exceeds 6 orders, and in 0.970 of them it reports more than twice the true lift. At thi… - Chapter 2 · Simplex-Constrained Least Squares: How Donor Weights Are Found and Why They May Not Be Negative
Chapter 2 finds simplex-constrained least-squares weights by projected gradient and by exact enumeration, and shows that with only thirty-six pre weeks the error of free regression rises to 18.22 while simplex weights s… - Chapter 3 · Pre-Period Fit Is the Entry Ticket: Root Mean Squared Prediction Error, the Convex Hull and Hold-Out Checks
On panel zero, synthetic control has a fitting error of 6.39 and a hold-out error of 4.00. Free regression fits to 3.11 but holds out at 15.51. The averages over two hundred panels are clearer: synthetic control fits at… - Chapter 4 · The Placebo Permutation Test: the Post/Pre Ratio, Ranks and the Floor of the p-Value
Chapter 4 gives the placebo permutation test: in the illustrative panel the redesigned site's post/pre ratio ranks second, p is 0.0588, short of five per cent. - Chapter 5 · The Donor Pool: No Spillover, Enough Donors and Interpolation Bias
With three donors the mean pre-period error is 19.95, 0.745 of the panels fail the fit, and the lift estimate has a root mean squared error of 31.81, hardly better than before-after; the floor of the p-value is 0.2500,… - Chapter 6 · Power and the Length of the Evaluation: How Many Weeks to See Five Per Cent
Start with the first row, zero lift. Requiring first place, the false alarm rate is 0.053 at four weeks, 0.040 at eight and 0.020 at twelve, near or below the nominal level, in line with Chapter 4. With a lift of 6 orde… - Chapter 7 · Eastern and Western Sources: "Take by Kind, Give by Kind" in the Mozi and Mill's Method of Difference
Literal twins number only 0.325 per panel on average, and only 0.260 of the panels have at least one. The kind of instance Mill asks for simply does not exist in about three shops in four with thirty-four sites. Before-… - Chapter 8 · Closing the Loop: Evaluating Single-Site Redesigns across Thirty-Four Sites, with a Pre-Registered Protocol
Chapter 8 gives a thirty-four-site evaluation sheet, the distilled Synthetic Control Evaluator and a pre-registered protocol: for a lift of five per cent of volume the single-site test has power of about 0.780 at thirte…
Illustrative model calculation, not a live quote