5D Library › Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels · 中文

Chapter 1 Attribution Is Not Incrementality: Last Click, Potential Outcomes and Randomised Holdouts

Two different questions

The “attribution” in a marketing report often answers two different questions at once. The first is a bookkeeping question: how should the credit for a conversion that has already happened be divided among the channels that appeared on its path? The second is a causal question: if we stopped spending on a channel next month, how many fewer conversions would there be? The first can be answered by any rule; the second requires comparing two worlds, the one in which we spend and the one in which we do not, and we only ever see one of them.

Statistics writes the second question in terms of potential outcomes (Holland 1986). For user ii, let Yi(1)Y_i(1) be whether he converts when he sees a channel’s ad and Yi(0)Y_i(0) whether he converts when he does not. The channel’s causal effect on him is Yi(1)−Yi(0)Y_i(1) – Y_i(0), and its incrementality over the whole population is ∑i[Yi(1)−Yi(0)]\sum_i [Y_i(1) – Y_i(0)]. Holland called this the fundamental problem of causal inference: a person can be in only one state, so only one of Yi(1)Y_i(1) and Yi(0)Y_i(0) is ever observed. Randomisation matters because, under random assignment, the treated and control groups have the same distribution in every other respect, and the difference of their average outcomes is an unbiased estimate of the average causal effect.

Rule-based attribution (last click, first click, linear, time decay, position-based) answers the first question. These rules are not wrong, since a bookkeeping rule is neither right nor wrong; the error is to take the bookkeeping result as the answer to the second question and to allocate budgets by it.

A fictional world whose truth is known

To compare how far attribution methods are from the truth, we need a world in which the truth is known. Real data provide no such world, so what follows uses a fictional one; it is the author’s model, with illustrative parameters.

Of 200,000 fictional users, 20 % have high purchase intent. There are four channels: search, social, display and email. Exposure to search and email depends strongly on intent (only those who want to buy search for the brand name or sit on the member list), social is only lightly targeted, and display is not targeted at all. Each user’s conversion probability equals a base probability set by intent (8 % for high intent, 0.5 % for low), plus the causal lift of each channel he sees (1.0, 3.0, 0.8 and 1.2 percentage points for search, social, display and email). Because the lifts add, each channel’s true incrementality is uniquely defined and can be computed directly from the model: the number exposed times the lift.

Calculation

# A FICTIONAL world of customer journeys whose true incrementality is known.
# 20 % of users have high purchase intent. Four channels: Search (S), Social (F),
# Display (D), Email (E). Exposure to S and E depends strongly on intent (selection);
# D is untargeted. Conversion probability = base(intent) + sum of the causal lifts
# of the channels seen (additive, so each channel's incremental effect is unique).
# Touch times: D and F early, S late, E anywhere; a conversion follows the last touch.
# Rule-based attribution (last, first, linear click) is compared with the truth.
import random

CH = "SFDE"
P_HIGH = 0.20
EXPO = {"S": (0.60, 0.10), "F": (0.35, 0.30), "D": (0.50, 0.50), "E": (0.60, 0.15)}
BASE = (0.08, 0.005)                                   # (high intent, low intent)
LIFT = {"S": 0.010, "F": 0.030, "D": 0.008, "E": 0.012}
WHEN = {"S": (0.6, 1.0), "F": (0.0, 0.7), "D": (0.0, 0.5), "E": (0.2, 1.0)}

def user(rng, sel=1.0):
    """One journey: (intent, channels in time order, converted)."""
    hi = rng.random() < P_HIGH
    touches = []
    for c in CH:
        p_hi, p_lo = EXPO[c]
        p = p_lo + sel * (p_hi - p_lo) if hi else p_lo   # sel = 0: no selection
        if rng.random() < p:
            touches.append((rng.uniform(*WHEN[c]), c))
    path = [c for _, c in sorted(touches)]
    p_conv = BASE[0 if hi else 1] + sum(LIFT[c] for c in path)
    return hi, path, rng.random() < p_conv

def truth(n, sel=1.0):
    """Expected incremental conversions per channel (exact, from the model)."""
    out = {}
    for c in CH:
        p_hi, p_lo = EXPO[c]
        expo = P_HIGH * (p_lo + sel * (p_hi - p_lo)) + (1 - P_HIGH) * p_lo
        out[c] = n * expo * LIFT[c]
    return out

def shares(d):
    tot = sum(d.values())
    return {c: d[c] / tot for c in CH}

def rules(data):
    last, first, linear = ({c: 0.0 for c in CH} for _ in range(3))
    for _, path, conv in data:
        if conv and path:
            last[path[-1]] += 1; first[path[0]] += 1
            for c in path:
                linear[c] += 1 / len(path)
    return last, first, linear

if __name__ == "__main__":
    n = 200_000
    rng = random.Random(2026)
    data = [user(rng) for _ in range(n)]
    conv = sum(d[2] for d in data)
    untouched = sum(d[2] for d in data if not d[1])
    tr = truth(n)
    base = n * (P_HIGH * BASE[0] + (1 - P_HIGH) * BASE[1])
    print(f"{n:,} fictional users: {conv:,} conversions ({untouched:,} with no touch)")
    print(f"expected: {base:,.0f} would happen without any ad,"
          f" {sum(tr.values()):,.0f} are incremental")
    last, first, linear = rules(data)
    print("share of credit       truth    last click   first click   linear")
    T, L, F, Li = shares(tr), shares(last), shares(first), shares(linear)
    names = {"S": "Search", "F": "Social", "D": "Display", "E": "Email"}
    for c in CH:
        print(f"   {names[c]:8s}         {T[c]:6.1%}     {L[c]:6.1%}       {F[c]:6.1%}"
              f"     {Li[c]:6.1%}")
    for lab, s in (("last click", L), ("first click", F), ("linear", Li)):
        mae = sum(abs(s[c] - T[c]) for c in CH) / len(CH)
        print(f"   mean absolute error of {lab:11s} {mae:6.1%}")
    credited = sum(last.values())
    print(f"last click credits {credited:,.0f} conversions to ads;"
          f" the ads caused about {sum(tr.values()):,.0f}")
# 200,000 fictional users: 7,676 conversions (377 with no touch)
# expected: 4,000 would happen without any ad, 3,636 are incremental
# share of credit       truth    last click   first click   linear
#    Search            11.0%      38.5%         6.4%      20.3%
#    Social            51.2%      21.0%        31.2%      27.4%
#    Display           22.0%      16.1%        46.7%      30.4%
#    Email             15.8%      24.5%        15.7%      22.0%
#    mean absolute error of last click   18.1%
#    mean absolute error of first click  12.3%
#    mean absolute error of linear       11.9%
# last click credits 7,299 conversions to ads; the ads caused about 3,636

The world has 7,676 conversions, of which about 4,000 would have happened without any advertising (high-intent users buy anyway); the ads caused only about 3,636. Last click credits 7,299 conversions to the ads, twice the true incrementality. The shares matter more: social’s true incremental share is 51.2 %, and last click gives it 21.0 %; search’s true share is 11.0 %, and last click gives it 38.5 %. There are two reasons, independent of each other. First, search usually comes at the end of a path (people search once they have decided to buy), so the last-touch rule naturally favours it. Second, search and email mainly reach people who were going to buy anyway, so they “appear” on paths much more often than they “cause” conversions. First click and linear do little better, with mean absolute errors of 12.3 and 11.9 percentage points (last click 18.1).

Note

The example illustrates a mechanism, not proportions. In the real world the targeting of each channel, its position on paths and its causal lift are unknown, and last click may overstate or understate any given channel. The program shows only this: as long as exposure is correlated with purchase intent, any rule that looks only at paths credits the ads with what intent did.

The algebra of selection bias

To write the intuition as a formula, let exposure to channel cc be D∈{0,1}D \in \{0, 1\}. The observed difference “conversion rate of the exposed minus that of the unexposed” decomposes as

E[Y∣D=1]−E[Y∣D=0]=E[Y(1)−Y(0)∣D=1]⏟average causal effect on the exposed+E[Y(0)∣D=1]−E[Y(0)∣D=0]⏟selection bias.E[Y \mid D=1] – E[Y \mid D=0] = \underbrace{E[Y(1) – Y(0) \mid D=1]}_{\text{average causal effect on the exposed}} + \underbrace{E[Y(0) \mid D=1] – E[Y(0) \mid D=0]}_{\text{selection bias}}.

The second term is the extent to which the people who were shown ads would have been more likely to buy even without them. The more precise the targeting, the larger this term. Ad platforms’ targeting systems aim precisely at finding “the people most likely to convert”, so in digital advertising selection bias is the rule, not the exception. Rule-based attribution, like the Shapley and Markov methods of Chapter 2, uses only conditional expectations of the kind E[Y∣path]E[Y \mid \text{path}], from which the second term cannot be removed.

Evidence: the gap between experiments and observational methods

Evidence

Why advertising effects are hard to measure. Lewis and Rao analysed 25 large randomised advertising experiments run with major US retailers and brokerages, with about $2.8 million of advertising spend, most with more than a million unique users (Lewis and Rao 2015). They found that individual-level sales are extremely volatile relative to the per-capita cost of advertising (a coefficient of variation of about 10 is common), and the median width of the confidence interval for return on investment exceeded 100 percentage points; informative conclusions could require samples of more than ten million person-weeks. So even with a randomised experiment, the return of a single campaign is often poorly measured; without one, separating the effect of advertising from the effect of intent with observational data alone is harder still.

Evidence

Observational methods against large experiments. Gordon, Zettelmeyer, Bhargava and Chapsky used 15 US advertising experiments at Facebook (about 500 million user–experiment observations and 1.6 billion ad impressions) as a benchmark, and asked whether common observational methods — exact matching, stratification, propensity-score matching, regression adjustment and inverse-probability-weighted regression adjustment — could recover the experimental results (Gordon et al. 2019). Often they could not, even after conditioning on extensive demographic and behavioural variables; overall they mostly overestimated advertising effects, though in some cases they underestimated them; in half of the studies the estimated percentage lift in purchase outcomes was off by a factor of about three across all methods. Richer covariates usually moved the estimates closer to the experiment, but results were inconsistent across studies.

Evidence

Brand-keyword search ads. Blake, Nosko and Tadelis ran large-scale field experiments on paid search at eBay (Blake et al. 2015). For brand keywords, when the ads were switched off traffic moved almost entirely to the organic search results, and the ads had no measurable short-term benefit; for non-brand keywords, new and infrequent users were positively influenced, while the frequent users who account for most of the spending were barely affected. They estimated that the returns to paid search are a fraction of conventional non-experimental estimates, and that average returns on non-brand keywords were negative. This is the real-world version of “search comes at the end of the path and reaches people who were going to buy anyway”.

Measuring incrementality: holdouts and ghost ads

The most direct way to measure incrementality is a randomised holdout: a random subset of users is prevented from seeing a channel’s ads, and its conversion rate is compared with that of everyone else. It has three difficulties. First, it is noisy and, as Lewis and Rao showed, needs very large samples. Second, if the control group sees a substitute ad (a public-service ad, say), the platform’s delivery algorithm may show the substitute to different people from those who would have seen the real ad, and the two groups are no longer comparable. Third, the real ad reaches only part of the treated group (the platform decides through auctions and targeting); comparing only “those in the treated group who were actually exposed” with the whole control group reintroduces selection bias, while comparing the whole treated group with the whole control group (intention to treat) is unbiased but diluted by the many who would never be exposed, and so noisier still.

The ghost ads method of Johnson, Lewis and Nubbemeyer solves the last two problems (Johnson et al. 2017). In the control group, the ad system runs its auction and targeting logic as usual and records the moments at which “this user would have been shown the ad if he were in the treated group”, but actually shows something else. Exposed users in the treated group can then be compared with “ghost-exposed” users in the control group: both are selected by the same delivery logic and both exclude people who would never have been reached, so the comparison is unbiased and more precise, and no substitute ads need be paid for.

Summary

Attribution rules answer a bookkeeping question; incrementality answers a causal one. Exposure unrelated to purchase intent is necessary for the two to agree, but for last click it is not sufficient: in the untargeted version of the same fictional world, last click is still biased by path position. In a fictional world whose truth is known, last click credits the ads with twice their true incrementality and moves credit from early, untargeted channels to late, heavily targeted ones. Studies comparing observational methods with large randomised experiments show that, even with rich covariates, observational estimates are often off by a factor of several; experiments on brand-keyword search show that the return of an end-of-path channel can be close to zero. The causal benchmark is the randomised holdout or ghost-ad experiment, at the cost of very large samples.

Hypothesis · not established · Systematic overstatement of end-of-path channels

The author’s hypothesis (not established): among e-commerce advertisers who use last-click attribution and target search and email heavily, the share last click gives to brand search and email is systematically higher than the incremental share estimated by randomised holdouts. What result would show it to be wrong: in a pre-registered set of advertisers (for example at least 20, recruited by independent researchers), the 95 % confidence interval of the pooled estimate of “last-click share minus holdout incremental share” contains zero or lies entirely below zero.

References

Blake, Thomas, Chris Nosko, and Steven Tadelis. 2015. “Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment.” Econometrica 83 (1): 155–74.
Gordon, Brett R., Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky. 2019. “A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook.” Marketing Science 38 (2): 193–225. https://doi.org/10.1287/mksc.2018.1135.
Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81 (396): 945–60.
Johnson, Garrett A., Randall A. Lewis, and Elmar I. Nubbemeyer. 2017. “Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness.” Journal of Marketing Research 54 (6): 867–84.
Lewis, Randall A., and Justin M. Rao. 2015. “The Unfavorable Economics of Measuring the Returns to Advertising.” Quarterly Journal of Economics 130 (4): 1941–73. https://doi.org/10.1093/qje/qjv023.

Fan, Y. (2026). Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels: From Thompson Sampling Bandits and Absorbing Markov Chains to BG/NBD Customer Lifetime Value, Read Beside the Strategic Dispositions of the Sunzi [算法营销、多触点沙普利归因与获客漏斗的随机控制] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836264 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).