What does Whitelisting, Attribution and Dual-Track Fees conclude, in numbers, about “Incrementality Experiments: Holdouts, Geographic Tests and the Shapley Split”?

Answer: The number of people a holdout needs grows with the inverse square of ad spend per user. At a coefficient of variation of 10 and RM 0.02 per user, merely telling whether incremental ROAS reaches the break-even line needs 1,061,621 people in each group. A ghost-ads design removes the factor of 11.1 that exposure dilution brings, but needs platform support. A geographic experiment across Malaysia's sixteen regions has a power of only 0.734 at a budget share of 0.03, rising to 0.999 when the share doubles to 0.06; one experiment estimates 2.390 with an exact randomisation p-value of 0.0005. The Shapley values of a factorial experiment divide an increment of RM 288,000 into 124,500, 102,000 and 61,500, exactly the whole; but A's return with the other channels running is 3.600, above its Shapley 3.112, so dividing money and spending it need two different rods.

Scope of the numbers: Chapter 1 takes apart an illustrative order priced at RM 59.90: the goods margin is 0.6837, but after platform fees, returns and refused cash-on-delivery parcels the contribution rate is 0.3467. The book is not financial, legal or tax advice and promises no income or return.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0005, 0.03, 0.06, 0.734, 0.999, 2.390

spend share 0.03, true iROAS 0.000: mean -0.010, sd 1.252, one-sided power 0.050
spend share 0.03, true iROAS 2.884: mean +2.825, sd 1.265, one-sided power 0.734
spend share 0.06, true iROAS 0.000: mean +0.016, sd 0.625, one-sided power 0.050
spend share 0.06, true iROAS 2.884: mean +2.889, sd 0.655, one-sided power 0.999
one experiment at share 0.06: estimate 2.390; exact randomisation p over 12,870 assignments 0.0005

Key formula

\[n = \frac{2\,(z_{\alpha}+z_{\beta})^2\,(\text{CV}\,\mu)^2}{(R\,c)^2},\]
Show the full program
# Chapter 5. p52: a geo experiment over Malaysia's 16 states and federal territories (synthetic).
# Weekly GMV of region i is BASE * w_i * season_t * (1 + trend_i * t) * noise.  Eight regions get
# whitelisting spend of a share k of their baseline GMV for 4 weeks after 4 pre-weeks.  Estimator:
# incremental GMV = sum over treated of (post - pre * control growth), divided by treated spend.
# Region weights are rough illustrative shares, not census figures.
import itertools
import math
import random

rng = random.Random(16)
W = [0.20, 0.12, 0.10, 0.08, 0.08, 0.07, 0.06, 0.06, 0.06, 0.05, 0.04, 0.04, 0.03, 0.006, 0.004, 0.004]
BASE, PRE, POST = 400000.0, 4, 4
SEASON = [1.00, 0.97, 1.05, 1.10, 1.02, 0.99, 1.08, 1.12]


def world(r, treated, k):
    pre, post, spend = [], [], []
    for i, w in enumerate(W):
        tr = rng.gauss(0.0, 0.01)
        g = [BASE * w * SEASON[t] * (1.0 + tr * t) * math.exp(rng.gauss(0.0, 0.06)) for t in range(8)]
        s = k * BASE * w * POST if i in treated else 0.0
        pre.append(math.fsum(g[:PRE]))
        post.append(math.fsum(g[PRE:]) + r * s)
        spend.append(s)
    return pre, post, spend


def estimate(pre, post, treated, k):
    ctrl = [i for i in range(16) if i not in treated]
    growth = math.fsum(post[i] for i in ctrl) / math.fsum(pre[i] for i in ctrl)
    inc = math.fsum(post[i] - pre[i] * growth for i in treated)
    return inc / math.fsum(k * BASE * W[i] * POST for i in treated)


for k in (0.03, 0.06):
    for r in (0.0, 2.884):
        est = []
        for _ in range(1000):
            t = set(rng.sample(range(16), 8))
            pre, post, _ = world(r, t, k)
            est.append(estimate(pre, post, t, k))
        mean = math.fsum(est) / len(est)
        sd = math.sqrt(math.fsum((x - mean) ** 2 for x in est) / (len(est) - 1))
        if r == 0.0:
            crit = sorted(est)[949]
        power = sum(1 for x in est if x > crit) / len(est)
        print("spend share {:.2f}, true iROAS {:.3f}: mean {:+.3f}, sd {:.3f}, one-sided power {:.3f}".format(
            k, r, mean, sd, power))
t = set(rng.sample(range(16), 8))
pre, post, _ = world(2.884, t, 0.06)
obs = estimate(pre, post, t, 0.06)
null = [estimate(pre, post, set(c), 0.06) for c in itertools.combinations(range(16), 8)]
p = sum(1 for x in null if x >= obs) / len(null)
print("one experiment at share 0.06: estimate {:.3f}; exact randomisation p over {:,} assignments {:.4f}"
      .format(
    obs, len(null), p))

Single source and citation

Fan, Y. (2026). Southeast Asian Social-Commerce Whitelist Amplification, Cross-Channel Attribution and Dual-Track Commission Growth (Version 1.0.0), Chapter 5: Incrementality Experiments: Holdouts, Geographic Tests and the Shapley Split. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-351 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026351,
  author    = {Fan, Yafu},
  title     = {Southeast Asian Social-Commerce Whitelist Amplification, Cross-Channel Attribution and Dual-Track Commission Growth},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book