5D Library › Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels · 中文

Chapter 8 Practice and a Pre-Registered Test

Note

The checklist and study protocol in this chapter are for research and self-checking, not marketing, investment or legal advice. The checklist cannot tell you how much to spend on which channel; it can only help you write your assumptions down before you decide. The users, paths, tests, customers and spend in the book are fictional or simulated; any real advertising can lose money.

Part 1 A four-step checklist

Chapters 1–6 can be condensed into a four-step checklist for reviewing a quarter’s marketing reports. Its thresholds (for example, “flag when the last-click share falls outside the confidence interval of the holdout estimate”) are examples of rules the user should write down in advance, not recommended values.

  1. Attribution and incrementality. What rule produced the channel credits in the report? For the main channels at least, is there an incrementality estimate from a randomised holdout or ghost-ad experiment? Does the attributed share fall within the experiment’s confidence interval (Chapters 1 and 2)?
  2. Stopping rules. Were the sample size and stopping rule of each A/B test fixed in advance? If the results were looked at along the way, how many times, and was a group-sequential or always-valid method used? For bandit-allocated experiments, are reported effects propensity-weighted (Chapters 3 and 4)?
  3. Customer value. Was lifetime value computed by multiplying first-year retention, or with a model that allows for heterogeneity? Is average order value correlated with purchase frequency (Chapter 5)?
  4. Budget at the margin. Is each channel’s budget set by its average return (total revenue / total spend) or its marginal return (how much an extra thousand brings in)? A channel with a high average return may already have a marginal return below cost (Chapter 6).

The program applies the checklist to a fictional quarter: 400,000 users from the fictional world of Chapter 1, with a holdout for every channel drawn as an independent random 10 % of users (ghost-ad style: the ad is suppressed among those who would have been shown it); the look records of three A/B tests; a lifetime value computed by multiplying first-year retention; and the average and marginal returns of three channels on a fictional concave response curve.

Calculation

# The four-step check applied to a FICTIONAL quarter. 400,000 users from the world of
# p01; for each channel, an independent random 10 % of users are held out (the ad is
# suppressed, ghost-ad style, so the holdout is defined among would-be-exposed users).
# Step 1: last-click credit against the holdout estimate of incremental conversions.
# Step 2: a test log with the number of looks at the data, against p04's error rates.
# Step 3: a lifetime claim built by multiplying first-year retention (against p09).
# Step 4: average against marginal return per channel (concave response, fictional).
import math, random

# --- helpers repeated from earlier programs, so that this one runs on its own ---
CH = "SFDE"
P_HIGH = 0.20
EXPO = {"S": (0.60, 0.10), "F": (0.35, 0.30), "D": (0.50, 0.50), "E": (0.60, 0.15)}
BASE = (0.08, 0.005)                                   # (high intent, low intent)
LIFT = {"S": 0.010, "F": 0.030, "D": 0.008, "E": 0.012}
WHEN = {"S": (0.6, 1.0), "F": (0.0, 0.7), "D": (0.0, 0.5), "E": (0.2, 1.0)}

def truth(n, sel=1.0):
    """Expected incremental conversions per channel (exact, from the model)."""
    out = {}
    for c in CH:
        p_hi, p_lo = EXPO[c]
        expo = P_HIGH * (p_lo + sel * (p_hi - p_lo)) + (1 - P_HIGH) * p_lo
        out[c] = n * expo * LIFT[c]
    return out

def stay_probs(bounds, h=0.1, drift=0.0):
    """P(|S_j| < bounds[j-1] for all j <= k), k = 1..K, S a Gaussian random walk with
    N(drift, 1) steps: Simpson's rule on a grid spanning each continuation interval."""
    phi = lambda x: math.exp(-x * x / 2) / math.sqrt(2 * math.pi)
    def grid(b):
        n = 2 * max(1, math.ceil(b / h)); d = 2 * b / n
        w = [(1 if i in (0, n) else 4 if i % 2 else 2) * d / 3 for i in range(n + 1)]
        return [-b + i * d for i in range(n + 1)], w
    xs, w = grid(bounds[0]); f = [phi(x - drift) for x in xs]
    out = [sum(a * b for a, b in zip(w, f))]
    for b in bounds[1:]:
        ys, v = grid(b)
        g = [sum(wi * fi * phi(y - x - drift) for x, wi, fi in zip(xs, w, f)
                 if abs(y - x - drift) < 8.5)
             for y in ys]
        xs, w, f = ys, v, g
        out.append(sum(a * b for a, b in zip(w, f)))
    return out

def sbg_surv(t, al, be):
    return math.exp(math.lgamma(be + t) - math.lgamma(be) + math.lgamma(al + be)
                    - math.lgamma(al + be + t))
# --- end of repeated helpers ---

def quarter(rng, n):
    rows = []
    for _ in range(n):
        hi = rng.random() < P_HIGH
        elig = [c for c in CH if rng.random() < EXPO[c][0 if hi else 1]]
        held = {c for c in CH if rng.random() < 0.10}
        seen = sorted((rng.uniform(*WHEN[c]), c) for c in elig if c not in held)
        path = [c for _, c in seen]
        conv = rng.random() < BASE[0 if hi else 1] + sum(LIFT[c] for c in path)
        rows.append((set(elig), held, path, conv))
    return rows

if __name__ == "__main__":
    n = 400_000; rows = quarter(random.Random(48), n)
    names = {"S": "Search", "F": "Social", "D": "Display", "E": "Email"}
    print("1 attribution vs incrementality (flag if last-click credit is outside the")
    print("  95 % interval of the holdout estimate); conversions in the quarter")
    tr = truth(n)
    for c in CH:
        e = [r for r in rows if c in r[0]]
        t = [r for r in e if c not in r[1]]; h = [r for r in e if c in r[1]]
        pt, ph = sum(r[3] for r in t) / len(t), sum(r[3] for r in h) / len(h)
        se = math.sqrt(pt * (1 - pt) / len(t) + ph * (1 - ph) / len(h))
        inc, lo, hi = (pt - ph) * len(t), (pt - ph - 1.96 * se) * len(t), (
            pt - ph + 1.96 * se) * len(t)
        lc = sum(1 for r in rows if r[3] and r[2] and r[2][-1] == c)
        flag = "FLAG" if not lo <= lc <= hi else "ok"
        print(f"   {names[c]:8s} last click {lc:5,d}; holdout {inc:5,.0f} ({lo:5,.0f}"
              f" to {hi:5,.0f}); truth {tr[c] * 0.9:5,.0f} -> {flag}")
    print("2 test log (flag if the stopping rule was not fixed in advance)")
    z = 1.959964; st = stay_probs([z * math.sqrt(k) for k in range(1, 31)])
    for name, looks, fixed in (("landing page B", 1, True), ("subject line", 14, False),
                               ("price badge", 30, False)):
        a = 1 - st[looks - 1]
        yn = "yes" if fixed else "no "
        print(f"   {name:15s} looks {looks:2d}, rule fixed {yn}:"
              f" actual false-positive rate {a:.3f} -> {'ok' if fixed else 'FLAG'}")
    print("3 lifetime claim (flag if a heterogeneous-cohort model differs by > 20 %)")
    al, be, r1, d = 0.70407, 1.18204, 0.631, 0.10           # sBG fit of p09
    ls = sum(sbg_surv(t, al, be) / (1 + d) ** t for t in range(400))
    ln = sum((r1 / (1 + d)) ** t for t in range(400))
    print(f"   naive {ln:.2f} years, sBG {ls:.2f} years ({ls / ln - 1:+.0%})"
          f" -> {'FLAG' if abs(ls / ln - 1) > 0.2 else 'ok'}")
    print("4 budget: average and marginal return per extra 1,000 (flag if marginal < 1)")
    for c, spend, k in (("S", 40_000, 9.0), ("F", 60_000, 5.0), ("E", 60_000, 3.0),
                        ("D", 30_000, 1.2)):
        rev = lambda a: k * 20_000 * math.log(1 + a / 20_000)       # fictional response
        avg = rev(spend) / spend; marg = (rev(spend + 1000) - rev(spend)) / 1000
        print(f"   {names[c]:8s} spend {spend:6,d}: average {avg:4.2f},"
              f" marginal {marg:4.2f}"
              f" -> {'FLAG' if marg < 1 else 'ok'}")
# 1 attribution vs incrementality (flag if last-click credit is outside the
#   95 % interval of the holdout estimate); conversions in the quarter
#    Search   last click 4,889; holdout   766 (  344 to 1,187); truth   720 -> FLAG
#    Social   last click 2,956; holdout 3,111 (2,734 to 3,488); truth 3,348 -> ok
#    Display  last click 2,231; holdout 1,332 (  855 to 1,809); truth 1,440 -> FLAG
#    Email    last click 3,488; holdout 1,384 (  958 to 1,810); truth 1,037 -> FLAG
# 2 test log (flag if the stopping rule was not fixed in advance)
#    landing page B  looks  1, rule fixed yes: actual false-positive rate 0.050 -> ok
#    subject line    looks 14, rule fixed no : actual false-positive rate 0.220 -> FLAG
#    price badge     looks 30, rule fixed no : actual false-positive rate 0.280 -> FLAG
# 3 lifetime claim (flag if a heterogeneous-cohort model differs by > 20 %)
#    naive 2.35 years, sBG 3.69 years (+58%) -> FLAG
# 4 budget: average and marginal return per extra 1,000 (flag if marginal < 1)
#    Search   spend 40,000: average 4.94, marginal 2.98 -> ok
#    Social   spend 60,000: average 2.31, marginal 1.24 -> ok
#    Email    spend 60,000: average 1.39, marginal 0.75 -> FLAG
#    Display  spend 30,000: average 0.73, marginal 0.48 -> FLAG

Step 1 flags three channels: search’s last-click credit is 4,889 conversions while the holdout estimates an incrementality of only 766 (95 % interval 344 to 1,187), with a true value of 720; email 3,488 against 1,384; display 2,231 against 1,332. Social’s last-click credit of 2,956 falls within the experimental interval (2,734 to 3,488) and is not flagged; this does not show last click to be “accurate” for social, only that it happens to be close. Step 2 flags two tests: one looked at 14 times and one 30 times, neither with a stopping rule fixed in advance, with actual type I errors of 22.0 % and 28.0 %. Step 3 flags: multiplication gives an expected lifetime of 2.35 years, the sBG 3.69 years, a 58 % difference. Step 4 flags two channels: display’s average return is already below 1 (0.73) and its marginal return lower still (0.48); email’s average return is 1.39, so each unit spent seems to bring back 1.39, but its marginal return is only 0.75, so spending more loses money, exactly the case that looking only at average returns misses. Search’s average return is 4.94 and its marginal return of 2.98 is still above 1, so on this fictional curve there is still room to spend more. The lesson of the example is that the four steps each look at a different side: attribution reports, test conclusions, customer values and budget margins can each be wrong, and not necessarily in the same direction.

Part 2 A pre-registered simulation benchmark

Question and design choices

On a synthetic multichannel data generator with known true increments, built and frozen by an independent third party, is the error of observational Shapley attribution against the true incremental shares smaller than that of last click, by at least the smallest effect of interest?

The author’s framework proposed two directions: (a) comparing attribution methods on synthetic data whose truth is known; (b) randomised holdout experiments with real advertisers. The book chooses (a), with three remarks. First, why not (b): a real advertiser’s holdout tests the incrementality of a particular campaign, not the attribution method itself; comparing methods requires knowing every channel’s true incremental share, which in real data can only be approximated by a separate holdout for each channel, and Chapter 1 showed how noisy a single holdout often is (Lewis and Rao 2015); in addition, real advertisers’ data would inevitably touch the author’s business, adding to the conflict of interest. Second, (a) tests only how the methods behave in a known world: the generator determines the result, and the conclusions cannot be carried over directly to any real advertising market; see “What this study cannot show” below. Third, it is run by an independent third party: the generator’s structure, parameter distributions and random seeds are designed and sealed by researchers unconnected with the author, who submits only the code of the two attribution methods.

Protocol

Element Specification
Generator A synthetic data generator designed by the third party, its structure and parameter distributions made public at registration: each “world” has 3–5 channels, and the dependence of exposure on latent purchase intent (targeting), each channel’s causal lift, its typical position on paths and the presence of a two-channel synergy are drawn from the registered distributions; 20,000 users per world
True increments For each world, the interventional characteristic function v*(S)v^*(S) (expected conversions with only the channels in SS switched on) is computed exactly from the generator’s causal structure; the true shares are the normalised Shapley values of v*v^*, which in additive worlds equal each channel’s incrementality
Two methods X: observational Shapley (as in program p02 of Chapter 2, v(S)v(S) the conversion rate of users whose exposure set is exactly SS); Y: last click. Both see only paths and conversions, not intent or v*v^*
Blinding Method code registered by hash; the third party encrypts and seals the file of true shares, which the people running and analysing the study do not touch before results are locked; the author has no access to seeds or generator parameters
Primary outcome For each world D=MAE(Y)−MAE(X)D = \mathrm{MAE}(Y) – \mathrm{MAE}(X) (difference of the mean absolute errors of channel shares, in percentage points), averaged over worlds; 95 % confidence interval by a bootstrap over worlds (10,000 resamples)
Pre-specified qualifier Whether X’s own mean absolute error exceeds 5 percentage points; if so, the conclusion carries the rider “closer than last click, but still far from the incremental truth”
Secondary outcomes Sensitivity analysis: DD recomputed with a different definition of the truth, the normalised single-channel removal increments v*(N)−v*(N\{c})v^*(N) – v^*(N \setminus \{c\}); mean absolute error of the Markov removal effect; errors by targeting strength and by presence of synergy; ratio of conversions credited to ads to true total incrementality; all descriptive or exploratory

There is a single primary outcome, “the difference in mean absolute errors”, because the core claim of the author’s framework is “Shapley beats last click”, and this outcome corresponds to it directly. The Markov method is a secondary outcome because Chapter 2 showed that its performance in synergy worlds depends on the generator’s structure; making it a co-primary outcome with Shapley would need a multiplicity correction and would turn the question from “is it better than last click” into “which allocation formula is best”. The qualifier guards against a conclusion “Shapley is significantly better” being misread as “Shapley is good enough”: Chapter 2 showed that with targeting it remains far from incrementality.

Guarding against data snooping

There is a single pre-fixed implementation of each method and a single primary outcome, and the author does no tuning on the generator; the generator structure, parameter distributions, method code and analysis code are registered publicly before any run, with hashes of the code registered alongside the protocol (Nosek et al. 2018); the third party carries out all runs and analyses; secondary and exploratory results are clearly labelled and cannot replace the primary conclusion. Book 33 discussed the statistical principles behind these practices.

Sample size

The sample size needs an assumption about the standard deviation of DD. The book sets it with a pilot generator: a toy generator written by the author, similar in structure to the table above, and not the generator the study will use.

Calculation

# Sample size for the Chapter 8 pre-registered simulation benchmark. Unit: a synthetic
# world drawn from a generator frozen by an independent third party. In each world the
# TRUE credit of a channel is the Shapley value of the interventional game v*(S) =
# expected conversions if exactly the channels in S were switched on (known exactly
# from the generator; for additive worlds it equals each channel's incremental
# conversions). Primary outcome per world: D = MAE(last click) - MAE(observational
# Shapley), the mean absolute error of channel shares against the truth. Secondary
# (sensitivity): the same D against normalised removal increments v*(N) - v*(N - c).
# PILOT GENERATOR (a planning assumption, not the trial's generator): 3-5 channels,
# random targeting strength, lifts, exposure and an optional two-channel synergy.
import itertools, math, random, statistics

# --- helpers repeated from earlier programs, so that this one runs on its own ---
def shapley(players, v):
    n, phi = len(players), {}
    for i in players:
        rest, s = [p for p in players if p != i], 0.0
        for k in range(n):
            w = math.factorial(k) * math.factorial(n - k - 1) / math.factorial(n)
            for T in itertools.combinations(rest, k):
                s += w * (v(frozenset(T) | {i}) - v(frozenset(T)))
        phi[i] = s
    return phi
# --- end of repeated helpers ---

Z = statistics.NormalDist().inv_cdf

def world(rng):
    K = rng.choice((3, 4, 5)); ch = "ABCDE"[:K]; ph = rng.uniform(0.1, 0.3)
    sel = rng.random()
    lo = {c: rng.uniform(0.05, 0.5) for c in ch}
    hi = {c: min(0.95, lo[c] + sel * rng.uniform(0, 0.5)) for c in ch}
    lift = {c: rng.uniform(0.0, 0.03) for c in ch}
    when = {c: rng.random() for c in ch}                  # typical position in a path
    syn = (rng.sample(ch, 2), rng.uniform(0, 0.04)) if rng.random() < 0.5 else (None, 0)
    base = (rng.uniform(0.03, 0.10), rng.uniform(0.002, 0.01))
    return ch, ph, lo, hi, lift, when, syn, base

def v_star(Wd, S):
    ch, ph, lo, hi, lift, when, syn, base = Wd
    out = 0.0
    for w, e, b in ((ph, hi, base[0]), (1 - ph, lo, base[1])):
        x = b + sum(e[c] * lift[c] for c in S)
        if syn[0] and all(c in S for c in syn[0]):
            x += e[syn[0][0]] * e[syn[0][1]] * syn[1]
        out += w * x
    return out

def users(rng, Wd, n):
    ch, ph, lo, hi, lift, when, syn, base = Wd
    data = []
    for _ in range(n):
        h = rng.random() < ph; e = hi if h else lo
        seen = [c for c in ch if rng.random() < e[c]]
        path = sorted(seen, key=lambda c: when[c] + 0.3 * rng.random())
        p = base[0 if h else 1] + sum(lift[c] for c in seen)
        if syn[0] and all(c in seen for c in syn[0]):
            p += syn[1]
        data.append((path, rng.random() < p))
    return data

def mae(a, b, ch):
    return sum(abs(a[c] - b[c]) for c in ch) / len(ch)

def score(rng, n_users=20_000):
    Wd = world(rng); ch = Wd[0]
    tru = shapley(list(ch), lambda S: v_star(Wd, S) - v_star(Wd, ()))
    data = users(rng, Wd, n_users)
    last = {c: 0.0 for c in ch}
    for path, y in data:
        if y and path:
            last[path[-1]] += 1
    cnt, conv = {}, {}
    for path, y in data:
        S = frozenset(path); cnt[S] = cnt.get(S, 0) + 1; conv[S] = conv.get(S, 0) + y
    rate = {S: conv[S] / cnt[S] for S in cnt}
    v = lambda T: rate.get(frozenset(T), 0.0) - rate.get(frozenset(), 0.0)
    obs = {c: 0.0 for c in ch}
    for S, m in cnt.items():
        if S:
            for c, x in shapley(sorted(S), v).items():
                obs[c] += m * x
    norm = lambda d: {c: d[c] / sum(d.values()) if sum(d.values()) else 1 / len(ch)
                      for c in ch}
    t, l, s = norm({c: max(tru[c], 0.0) for c in ch}), norm(last), norm(obs)
    full = v_star(Wd, ch)                     # alternative truth: removal increments
    alt = norm({c: max(full - v_star(Wd, [d for d in ch if d != c]), 0.0) for c in ch})
    return mae(l, t, ch), mae(s, t, ch), mae(l, alt, ch) - mae(s, alt, ch)

if __name__ == "__main__":
    rng = random.Random(4800); n0 = 120
    R = [score(rng) for _ in range(n0)]
    D = [a - b for a, b, _ in R]
    m, sd = statistics.fmean(D), statistics.stdev(D)
    print(f"pilot ({n0} worlds, 20,000 users each): MAE last click"
          f" {statistics.fmean(r[0] for r in R):.1%},"
          f" Shapley {statistics.fmean(r[1] for r in R):.1%}")
    print(f"   D = MAE(last) - MAE(Shapley): mean {m:+.2%}, sd {sd:.2%};"
          f" Shapley better in {sum(d > 0 for d in D)}/{n0}")
    Da = [r[2] for r in R]
    print(f"   secondary, truth = normalised removal increments v*(N) - v*(N - c):"
          f"\n      mean D {statistics.fmean(Da):+.2%}, Shapley better in"
          f" {sum(d > 0 for d in Da)}/{n0}")
    sesoi = 0.02
    n = math.ceil(((Z(0.975) + Z(0.90)) * sd / sesoi) ** 2)
    print(f"SESOI 2 percentage points; 90 % power, two-sided alpha 0.05:")
    print(f"   {n} worlds (normal approximation, sd from the pilot)")
    hits, rs = 0, random.Random(9)
    for _ in range(4000):                                 # resampling check of power
        s = [d - m + sesoi for d in rs.choices(D, k=n)]   # pilot shape, mean = SESOI
        se = statistics.stdev(s) / math.sqrt(n)
        hits += statistics.fmean(s) - Z(0.975) * se > 0
    print(f"   simulated power at n = {n}: {hits / 4000:.3f} (4,000 resamples of the"
          f" pilot differences)")
    plan = math.ceil(n * 1.10 / 10) * 10
    print(f"   plus 10 % for failed or degenerate worlds, rounded up: plan {plan} worlds")
    print("sensitivity (worlds)        SESOI: 1 pt     2 pt     3 pt")
    for lab, f in (("sd as in the pilot", 1.0), ("sd 50 % higher", 1.5)):
        row = [math.ceil(((Z(0.975) + Z(0.90)) * sd * f / e) ** 2) for e in
               (0.01, 0.02, 0.03)]
        print(f"   {lab:28s}" + "".join(f"{x:8,d}" for x in row))
# pilot (120 worlds, 20,000 users each): MAE last click 13.0%, Shapley 4.7%
#    D = MAE(last) - MAE(Shapley): mean +8.28%, sd 5.48%; Shapley better in 112/120
#    secondary, truth = normalised removal increments v*(N) - v*(N - c):
#       mean D +8.02%, Shapley better in 112/120
# SESOI 2 percentage points; 90 % power, two-sided alpha 0.05:
#    79 worlds (normal approximation, sd from the pilot)
#    simulated power at n = 79: 0.899 (4,000 resamples of the pilot differences)
#    plus 10 % for failed or degenerate worlds, rounded up: plan 90 worlds
# sensitivity (worlds)        SESOI: 1 pt     2 pt     3 pt
#    sd as in the pilot               316      79      36
#    sd 50 % higher                   711     178      79

In the pilot (120 worlds), last click’s mean absolute error is 13.0 percentage points and observational Shapley’s 4.7; the mean of DD is +8.28 points with standard deviation 5.48, and Shapley is better in 112 worlds; with “normalised single-channel removal increments” as the truth (a secondary outcome), the mean of DD is +8.02 points. These numbers are planning assumptions, not evidence: the pilot generator was built by the author, half its worlds have no synergy and targeting strength is drawn uniformly between 0 and 1, and the difference on an independent generator may be much smaller, or even negative.

The smallest effect of interest is 2 percentage points. In budget terms: when a budget is allocated by shares, the fraction misplaced is 12∑c|sc−tc|=K2MAE\tfrac12\sum_c|s_c – t_c| = \tfrac{K}{2}\,\mathrm{MAE}, so with KK channels each percentage point of MAE misplaces about K/2K/2 percentage points of budget; D=2D = 2 means that last click misplaces about 4 % more of the budget than Shapley (with four channels; 3 %–5 % with three to five). A smaller difference is not worth the extra computation and explanation Shapley requires. With two-sided α=0.05\alpha = 0.05 and 90 % power, the normal approximation needs n=(z0.975+z0.90)2σ2/Δ2=79n = (z_{0.975} + z_{0.90})^2 \sigma^2/\Delta^2 = 79 worlds; resampling the pilot differences with their mean shifted to the smallest effect of interest gives a power of 0.899 at 79 worlds, in agreement with the normal approximation. Allowing 10 % of worlds to be excluded because generation fails or degenerates (for example no conversions, or a channel with no exposure), 90 worlds are planned. The sensitivity table shows that a smallest effect of 1 point would need 316 worlds, and a standard deviation 50 % higher than in the pilot would need 178 worlds for the 2-point effect. Synthetic worlds are cheap, so the sample size itself is not the bottleneck; the bottleneck is locking the generator and the methods in advance.

Following Chapter 6 of Book 33, “90 % power” means that if the true effect equals the smallest effect of interest exactly, the probability of reaching some conclusion that “Shapley is closer” (a confidence interval whose lower bound is above 0) is about 90 %; in that case the strongest conclusion, “better by at least the smallest effect of interest”, has only about a 2.5 % chance of appearing.

Decision rule

Calculation

# Decision rule, fixed before the third party runs the frozen generator. D = MAE(last
# click) - MAE(observational Shapley), in percentage points of channel share, averaged
# over worlds, with a 95 % CI from a bootstrap over worlds (10,000 resamples). SESOI
# 2 points. Gates, checked first: more than 10 % of worlds failed or degenerate; the
# independent re-run differing on more than 0.1 % of worlds; the generator's truth
# file opened before the results were locked -> NOT INTERPRETABLE.
# Pre-specified qualifier: Shapley's own MAE above 5 points ("better than last click,
# still far from the incremental truth").
SESOI = 2.0

def verdict(d, lo, hi, mae_sh, failed=0.02, mismatch=0.0, unblinded=False):
    for bad, why in ((failed > 0.10, f"{failed:.0%} of worlds failed (limit 10 %)"),
                     (mismatch > 0.001, f"re-run differs on {mismatch:.2%}"
                                        " (limit 0.1 %)"),
                     (unblinded, "truth file opened before lock")):
        if bad:
            return f"D = {d:+.1f}: NOT INTERPRETABLE - {why}"
    if lo >= SESOI:   v = "Shapley is closer to the truth by at least the SESOI"
    elif lo > 0:      v = "Shapley is closer; not shown to reach the SESOI"
    elif hi < 0:      v = "last click is closer to the truth"
    elif lo >= -SESOI and hi <= SESOI: v = "no difference worth acting on"
    else:             v = "unclear: the interval is too wide to decide"
    if mae_sh > 5.0:
        v += f"\n    qualifier: Shapley's own error is {mae_sh:.1f} points (limit 5)"
    return f"D = {d:+.1f} ({lo:+.1f} to {hi:+.1f}): {v}"

for case in ((8.3, 7.1, 9.5, 4.7), (3.0, 1.2, 4.8, 3.9), (0.4, -0.9, 1.7, 9.0),
             (-2.5, -3.9, -1.1, 11.0), (1.0, -2.6, 4.6, 6.0), (6.0, 4.4, 7.6, 8.2),
             (8.0, 6.8, 9.2, 4.0, 0.14), (8.0, 6.8, 9.2, 4.0, 0.02, 0.0, True)):
    print(verdict(*case))
print("secondary outcomes (Markov removal-effect MAE, MAE by targeting strength and")
print("  synergy, total credited against total incremental conversions) are reported")
print("  with intervals; they never decide the verdict")
# D = +8.3 (+7.1 to +9.5): Shapley is closer to the truth by at least the SESOI
# D = +3.0 (+1.2 to +4.8): Shapley is closer; not shown to reach the SESOI
# D = +0.4 (-0.9 to +1.7): no difference worth acting on
#     qualifier: Shapley's own error is 9.0 points (limit 5)
# D = -2.5 (-3.9 to -1.1): last click is closer to the truth
#     qualifier: Shapley's own error is 11.0 points (limit 5)
# D = +1.0 (-2.6 to +4.6): unclear: the interval is too wide to decide
#     qualifier: Shapley's own error is 6.0 points (limit 5)
# D = +6.0 (+4.4 to +7.6): Shapley is closer to the truth by at least the SESOI
#     qualifier: Shapley's own error is 8.2 points (limit 5)
# D = +8.0: NOT INTERPRETABLE - 14% of worlds failed (limit 10 %)
# D = +8.0: NOT INTERPRETABLE - truth file opened before lock
# secondary outcomes (Markov removal-effect MAE, MAE by targeting strength and
#   synergy, total credited against total incremental conversions) are reported
#   with intervals; they never decide the verdict

The example results are made up. If any gate fails (more than 10 % of worlds failed or degenerate, the independent re-run disagreeing with the registered code on more than 0.1 % of worlds, the file of true shares opened before results were locked), the report says “not interpretable” whatever the primary outcome. Only when the lower confidence bound is at least 2 percentage points does the report say “Shapley is closer by at least the smallest effect of interest”; a lower bound above 0 but below 2 gives “Shapley is closer, size not established”; an upper bound below 0 gives “last click is closer”; an interval within ±2 gives “no difference worth acting on”; anything else gives “unclear”. The qualifier does not change the category of the conclusion, only adds to it: in the sixth example, Shapley is closer by at least the smallest effect of interest, but its own error is still 8.2 points, and the report must say so as well.

What this study cannot show

First, the generator is not a real market: the conclusions hold only for its targeting, synergy and path structures, and real selection bias may be stronger, more complex and changing over time. Second, the generator’s design determines the result; the protocol registers the parameter distributions but cannot cover every form of targeting and interaction. Third, the definition of the “truth” builds in the Shapley equal-split rule: the true shares are the Shapley values of the interventional v*v^*, so synergy is split equally by definition; in untargeted worlds observational Shapley equals this “truth” in expectation while last click does not. The primary outcome therefore depends heavily on the generator’s distribution of targeting strength and path position, which is why a second definition of the truth is added as a sensitivity analysis. Fourth, it compares one Shapley implementation with last click, and tests neither other Shapley variants (such as the counterfactual adjusted Shapley value (Singal et al. 2022)) nor model-based attribution. Fifth, even if Shapley is clearly better in synthetic worlds, that does not show it delivers causal increments in real data; Chapters 1 and 2 showed that with targeting it does not. Sixth, it tests none of the models of Chapters 3–6, which need their own tests. Seventh, the large effect in the pilot comes from the author’s own generator, which is exactly why an independent generator and pre-registration are needed; if the independent study finds “no difference worth acting on”, that is the study’s conclusion. Eighth, it cannot replace randomised holdout experiments with real advertisers; in any concrete media decision, incrementality must still be measured by experiment.

Ethics and conflicts of interest

The study involves no human participants and uses no data from any real user, advertiser or customer, so it needs no ethics committee approval; but the protocol, generator, method code and analysis code must be registered publicly before any run. Whatever the result, the results, the code and summary data for every world will be made public. To limit the influence of interests: the protocol and code are registered publicly before the third party runs anything, the author has no access to seeds, generator parameters or the file of true shares, the study uses no data from the author’s businesses or their clients, and the results are published whatever they are.

Disclosure

The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check. The author founded and runs Asia Influencer X and Yafu Global, which work in influencer marketing, live-stream commerce and digital marketing (including ad buying, performance attribution and customer acquisition) and use the attribution and experimentation methods discussed in this book in their daily work, so he has a direct commercial interest in its subject; the book uses no data from these two companies or their clients as evidence and does not promote their services.

Summary

The four-step checklist reviews a quarter’s marketing reports from four sides: attribution and incrementality, stopping rules, customer value and the budget margin; the fictional example shows that last click can overstate a channel’s incrementality six-fold, that tests without a fixed stopping rule can have actual error rates of 28 %, that multiplied retention and a heterogeneity model differ by 58 %, and that a channel with an average return of 1.39 can have a marginal return of only 0.75, below cost. The pre-registered simulation benchmark compares observational Shapley with last click on their error against true incremental shares, on 90 synthetic worlds from a generator with known increments frozen by an independent third party, with a single primary outcome (the difference in mean absolute errors), a smallest effect of interest of 2 percentage points, 90 % power, a decision rule written as code and a pre-specified rider “still far from the incremental truth”; it is published whatever the result. It tests only how the methods behave in a simulated world whose truth is known; it cannot prove the book’s other models, nor replace real incrementality experiments.

References

Lewis, Randall A., and Justin M. Rao. 2015. “The Unfavorable Economics of Measuring the Returns to Advertising.” Quarterly Journal of Economics 130 (4): 1941–73. https://doi.org/10.1093/qje/qjv023.
Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115 (11): 2600–2606. https://doi.org/10.1073/pnas.1708274114.
Singal, Raghav, Omar Besbes, Antoine Desir, Vineet Goyal, and Garud Iyengar. 2022. “Shapley Meets Uniform: An Axiomatic Framework for Attribution in Online Advertising.” Management Science 68 (10): 7457–79. https://doi.org/10.1287/mnsc.2021.4263.

Fan, Y. (2026). Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels: From Thompson Sampling Bandits and Absorbing Markov Chains to BG/NBD Customer Lifetime Value, Read Beside the Strategic Dispositions of the Sunzi [算法营销、多触点沙普利归因与获客漏斗的随机控制] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836264 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).