What does Whitelisting, Attribution and Dual-Track Fees conclude, in numbers, about “Incrementality Experiments: Holdouts, Geographic Tests and the Shapley Split”?
Answer: The number of people a holdout needs grows with the inverse square of ad spend per user. At a coefficient of variation of 10 and RM 0.02 per user, merely telling whether incremental ROAS reaches the break-even line needs 1,061,621 people in each group. A ghost-ads design removes the factor of 11.1 that exposure dilution brings, but needs platform support. A geographic experiment across Malaysia's sixteen regions has a power of only 0.734 at a budget share of 0.03, rising to 0.999 when the share doubles to 0.06; one experiment estimates 2.390 with an exact randomisation p-value of 0.0005. The Shapley values of a factorial experiment divide an increment of RM 288,000 into 124,500, 102,000 and 61,500, exactly the whole; but A's return with the other channels running is 3.600, above its Shapley 3.112, so dividing money and spending it need two different rods.
Scope of the numbers: Chapter 1 takes apart an illustrative order priced at RM 59.90: the goods margin is 0.6837, but after platform fees, returns and refused cash-on-delivery parcels the contribution rate is 0.3467. The book is not financial, legal or tax advice and promises no income or return.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0005, 0.03, 0.06, 0.734, 0.999, 2.390
spend share 0.03, true iROAS 0.000: mean -0.010, sd 1.252, one-sided power 0.050 spend share 0.03, true iROAS 2.884: mean +2.825, sd 1.265, one-sided power 0.734 spend share 0.06, true iROAS 0.000: mean +0.016, sd 0.625, one-sided power 0.050 spend share 0.06, true iROAS 2.884: mean +2.889, sd 0.655, one-sided power 0.999 one experiment at share 0.06: estimate 2.390; exact randomisation p over 12,870 assignments 0.0005
Key formula
Show the full program
# Chapter 5. p52: a geo experiment over Malaysia's 16 states and federal territories (synthetic).
# Weekly GMV of region i is BASE * w_i * season_t * (1 + trend_i * t) * noise. Eight regions get
# whitelisting spend of a share k of their baseline GMV for 4 weeks after 4 pre-weeks. Estimator:
# incremental GMV = sum over treated of (post - pre * control growth), divided by treated spend.
# Region weights are rough illustrative shares, not census figures.
import itertools
import math
import random
rng = random.Random(16)
W = [0.20, 0.12, 0.10, 0.08, 0.08, 0.07, 0.06, 0.06, 0.06, 0.05, 0.04, 0.04, 0.03, 0.006, 0.004, 0.004]
BASE, PRE, POST = 400000.0, 4, 4
SEASON = [1.00, 0.97, 1.05, 1.10, 1.02, 0.99, 1.08, 1.12]
def world(r, treated, k):
pre, post, spend = [], [], []
for i, w in enumerate(W):
tr = rng.gauss(0.0, 0.01)
g = [BASE * w * SEASON[t] * (1.0 + tr * t) * math.exp(rng.gauss(0.0, 0.06)) for t in range(8)]
s = k * BASE * w * POST if i in treated else 0.0
pre.append(math.fsum(g[:PRE]))
post.append(math.fsum(g[PRE:]) + r * s)
spend.append(s)
return pre, post, spend
def estimate(pre, post, treated, k):
ctrl = [i for i in range(16) if i not in treated]
growth = math.fsum(post[i] for i in ctrl) / math.fsum(pre[i] for i in ctrl)
inc = math.fsum(post[i] - pre[i] * growth for i in treated)
return inc / math.fsum(k * BASE * W[i] * POST for i in treated)
for k in (0.03, 0.06):
for r in (0.0, 2.884):
est = []
for _ in range(1000):
t = set(rng.sample(range(16), 8))
pre, post, _ = world(r, t, k)
est.append(estimate(pre, post, t, k))
mean = math.fsum(est) / len(est)
sd = math.sqrt(math.fsum((x - mean) ** 2 for x in est) / (len(est) - 1))
if r == 0.0:
crit = sorted(est)[949]
power = sum(1 for x in est if x > crit) / len(est)
print("spend share {:.2f}, true iROAS {:.3f}: mean {:+.3f}, sd {:.3f}, one-sided power {:.3f}".format(
k, r, mean, sd, power))
t = set(rng.sample(range(16), 8))
pre, post, _ = world(2.884, t, 0.06)
obs = estimate(pre, post, t, 0.06)
null = [estimate(pre, post, set(c), 0.06) for c in itertools.combinations(range(16), 8)]
p = sum(1 for x in null if x >= obs) / len(null)
print("one experiment at share 0.06: estimate {:.3f}; exact randomisation p over {:,} assignments {:.4f}"
.format(
obs, len(null), p))Single source and citation
Fan, Y. (2026). Southeast Asian Social-Commerce Whitelist Amplification, Cross-Channel Attribution and Dual-Track Commission Growth (Version 1.0.0), Chapter 5: Incrementality Experiments: Holdouts, Geographic Tests and the Shapley Split. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-351 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026351,
author = {Fan, Yafu},
title = {Southeast Asian Social-Commerce Whitelist Amplification, Cross-Channel Attribution and Dual-Track Commission Growth},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Account of One Order: Platform Fees, Returns and Cash on Delivery
- Chapter 2 · The Break-Even Point of Advertising: Contribution Rate, Sensitivity and Four Routes
- Chapter 3 · Whitelist Amplification: Paid-to-Organic Spillover, Mega-Sale Confounding and Marginal Return
- Chapter 4 · Attribution Stacking: Double Counting On and Off the Platform, Deduplication Rules and the Sum of ROAS
- Chapter 6 · The Dual-Track Contract: Fixed Licensing Fee, Tiered Commission and the Arithmetic of Principal and Agent
- Chapter 7 · The Classics Examined: Bai Gui, the Fair Merchant, the Commenda and Adam Smith
- Chapter 8 · The Conversion Tool: Break-Even, Attribution Dedup and Dual-Track Contract Calculator, with a Pre-Registered Test