5D Library › 算法营销、多触点沙普利归因与获客漏斗的随机控制 · English

第五章 一位客户还值多少钱:BG/NBD、Gamma-Gamma 与留存率的假象

非契约关系中的客户

订阅、电信与保险是契约关系:客户什么时候离开,账上有记录。电商、餐饮与直播带货多是非契约关系:客户最后一次购买之后再也没出现,我们不知道他是已经流失,还是只是还没到下一次购买的时候。客户终身价值(CLV)的估计在这两种情形下需要不同的模型 (Gupta et al. 2006)。本章先讨论非契约情形的 BG/NBD 与 Gamma-Gamma 模型,再讨论契约情形下「留存率随时间上升」的现象,以及它与辛普森悖论的区别。

BG/NBD 模型

施密特莱因(Schmittlein)、莫里森(Morrison)与科伦坡(Colombo)1987 年提出的 Pareto/NBD 模型是这一领域的起点 (Schmittlein et al. 1987),但它的似然函数需要反复计算高斯超几何函数,数值上相当麻烦。费德(Fader)、哈迪(Hardie)与李(Lee)2005 年提出了一个更容易计算的替代模型 BG/NBD (Fader et al. 2005a),假设如下:

  1. 客户在「活跃」期间按泊松过程购买,购买率为 λ\lambda;
  2. 不同客户的 λ\lambda 服从伽马分布 Γ(r,α)\Gamma(r, \alpha);
  3. 每次购买之后,客户以概率 pp 永久变为不活跃;
  4. 不同客户的 pp 服从贝塔分布 B(a,b)B(a, b);
  5. λ\lambda 与 pp 在客户之间相互独立。

与 Pareto/NBD 的区别在于第 3 条:流失只能发生在购买之后,而不是在连续时间中的任何时刻。这使似然函数只含伽马函数与贝塔函数。对一个在 (0,T](0, T] 内重复购买 xx 次、最后一次在 txt_x 的客户,

L=B(a,b+x)B(a,b)Γ(r+x)αrΓ(r)(α+T)r+x+δx>0B(a+1,b+x−1)B(a,b)Γ(r+x)αrΓ(r)(α+tx)r+x,L = \frac{B(a, b+x)}{B(a, b)}\,\frac{\Gamma(r+x)\,\alpha^r}{\Gamma(r)\,(\alpha+T)^{r+x}} + \delta_{x>0}\,\frac{B(a+1, b+x-1)}{B(a, b)}\,\frac{\Gamma(r+x)\,\alpha^r}{\Gamma(r)\,(\alpha+t_x)^{r+x}},

只依赖 (x,tx,T)(x, t_x, T) 三个数,即营销中常说的频率与最近一次购买(RFM 中的 F 与 R)。期望购买次数为

E[X(t)]=a+b−1a−1[1−(αα+t)r2F1(r,b;a+b−1;tα+t)].E[X(t)] = \frac{a+b-1}{a-1}\left[1 – \left(\frac{\alpha}{\alpha+t}\right)^r {}_2F_1\!\left(r, b; a+b-1; \frac{t}{\alpha+t}\right)\right].

由似然的两项之比,可以得到一个客户在 TT 时仍然活跃的后验概率:x>0x > 0 时为 1/[1+ab+x−1(α+Tα+tx)r+x]1/\bigl[1 + \tfrac{a}{b+x-1}\bigl(\tfrac{\alpha+T}{\alpha+t_x}\bigr)^{r+x}\bigr],x=0x = 0 时为 1(模型中流失只发生在购买之后)。

证据

原文的数据与估计。 费德等人用 CDNOW 网站 1997 年第一季度首购客户的 1/10 系统抽样(2,357 人)、前 39 周为校准期、后 39 周为保留期,估计得 r=0.243r = 0.243、α=4.414\alpha = 4.414、a=0.793a = 0.793、b=2.426b = 2.426,对数似然 −9,582.4;同一数据上 Pareto/NBD 的对数似然为 −9,595.0 (Fader et al. 2005a)。他们指出,对给定参数计算一次高斯超几何函数并不难,即使在电子表格中也可以用多项式级数近似。

计算 2F1_2F_1:级数与积分

高斯超几何函数 2F1(a,b;c;z)=∑k(a)k(b)k(c)kk!zk_2F_1(a, b; c; z) = \sum_{k} \frac{(a)_k (b)_k}{(c)_k\, k!} z^k 在 |z|<1|z| < 1 时收敛,但 zz 越接近 1 收敛越慢。在 E[X(t)]E[X(t)] 中 z=t/(α+t)z = t/(\alpha + t),预测期越长越接近 1。另一种算法是欧拉积分表示 (Abramowitz and Stegun 1964)

2F1(a,b;c;z)=Γ(c)Γ(a)Γ(c−a)∫01sa−1(1−s)c−a−1(1−zs)−bds,c>a>0.{}_2F_1(a, b; c; z) = \frac{\Gamma(c)}{\Gamma(a)\Gamma(c-a)} \int_0^1 s^{a-1}(1-s)^{c-a-1}(1-zs)^{-b}\,ds, \qquad c > a > 0.

这里有两个数值陷阱。第一,在 BG/NBD 中 a=r=0.243<1a = r = 0.243 < 1,被积函数在 s=0s = 0 处发散;第二,zz 接近 1 时,(1−zs)−b(1-zs)^{-b} 在 s=1s = 1 附近急剧变大。程序的处理是:先用普法夫(Pfaff)变换 2F1(a,b;c;z)=(1−z)−a2F1(a,c−b;c;z/(z−1))_2F_1(a, b; c; z) = (1-z)^{-a}\,{}_2F_1(a, c-b; c; z/(z-1)) 把自变量移到负半轴,此时 (1−ws)b−c(1 – ws)^{b-c} 在 [0,1][0, 1] 上有界;再令 s=u1/as = u^{1/a},把 sa−1dss^{a-1}ds 变成 du/adu/a,消去端点的奇异性,然后用辛普森法则积分。

推导

# BG/NBD (Fader, Hardie & Lee 2005). While alive, a customer buys as Poisson(lambda),
# lambda ~ Gamma(r, alpha); after each purchase she drops out with probability p,
# p ~ Beta(a, b). A: E[X(t)] = (a+b-1)/(a-1) [1 - (alpha/(alpha+t))^r 2F1(r, b; a+b-1;
# t/(alpha+t))]; 2F1 by its power series and by Euler's integral after Pfaff's
# transformation and a change of variable, compared as z -> 1, and checked against a
# simulation. B: a FICTIONAL cohort of 2,357 customers simulated with the published
# CDNOW estimates, refitted by maximum likelihood over 39 weeks; holdout forecast for
# weeks 40-78; P(alive) for two customers with the same frequency and recency apart.
import math, random, statistics

def f21_series(a, b, c, z, tol=1e-12, kmax=200_000):
    term, s, k = 1.0, 1.0, 0
    while abs(term) > tol * abs(s) and k < kmax:
        term *= (a + k) * (b + k) / ((c + k) * (k + 1)) * z; s += term; k += 1
    return s, k

def f21_euler(a, b, c, z, n=200):
    """For 0 < a < c, z < 1: Pfaff's transformation 2F1(a,b;c;z) = (1-z)^(-a)
    2F1(a, c-b; c; w), w = z/(z-1) <= 0, then Euler's integral G(c)/(G(a)G(c-a))
    int_0^1 s^(a-1) (1-s)^(c-a-1) (1-ws)^(b-c) ds, whose integrand stays bounded;
    with s = u^(1/a) the weight s^(a-1) ds becomes du / a. Simpson's rule in u."""
    w = z / (z - 1)
    g = lambda u: ((1 - u ** (1 / a)) ** (c - a - 1)
                   * (1 - w * u ** (1 / a)) ** (b - c) / a)
    h = 1 / n; tot = g(0) + g(1)
    tot += sum((4 if i % 2 else 2) * g(i * h) for i in range(1, n))
    lg = math.lgamma(c) - math.lgamma(a) - math.lgamma(c - a)
    return (1 - z) ** (-a) * math.exp(lg) * tot * h / 3

def ex(t, r, al, a, b, f21=f21_euler):
    z = t / (al + t)
    return (a + b - 1) / (a - 1) * (1 - (al / (al + t)) ** r * f21(r, b, a + b - 1, z))

def simulate(rng, n, r, al, a, b, T):
    out = []
    for _ in range(n):
        lam, p = rng.gammavariate(r, 1 / al), rng.betavariate(a, b)
        t, times = 0.0, []
        while True:
            t += rng.expovariate(lam)
            if t > T:
                break
            times.append(t)
            if rng.random() < p:
                break
        out.append(times)
    return out

def loglik(th, data):
    r, al, a, b = th
    if min(th) <= 0:
        return -1e18
    c = math.lgamma(r) - r * math.log(al)
    lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    ll = 0.0
    for x, tx, T in data:
        A1 = math.lgamma(r + x) - c
        t1 = lb(a, b + x) - lb(a, b) + A1 - (r + x) * math.log(al + T)
        if x > 0:
            t2 = lb(a + 1, b + x - 1) - lb(a, b) + A1 - (r + x) * math.log(al + tx)
            m = max(t1, t2); ll += m + math.log(math.exp(t1 - m) + math.exp(t2 - m))
        else:
            ll += t1
    return ll

def p_alive(x, tx, T, r, al, a, b):
    if x == 0:
        return 1.0
    return 1 / (1 + a / (b + x - 1) * ((al + T) / (al + tx)) ** (r + x))

def nelder_mead(f, x0, it=300):
    n = len(x0)
    pts = [list(x0)] + [[x * (1.2 if i == j else 1) for j, x in enumerate(x0)]
                        for i in range(n)]
    for _ in range(it):
        pts.sort(key=f); c = [sum(p[i] for p in pts[:-1]) / n for i in range(n)]
        w = pts[-1]; r = [c[i] + (c[i] - w[i]) for i in range(n)]
        if f(r) < f(pts[0]):
            e = [c[i] + 2 * (c[i] - w[i]) for i in range(n)]
            pts[-1] = e if f(e) < f(r) else r
        elif f(r) < f(pts[-2]):
            pts[-1] = r
        else:
            k = [c[i] + 0.5 * (w[i] - c[i]) for i in range(n)]
            if f(k) < f(w):
                pts[-1] = k
            else:
                pts = [pts[0]] + [[pts[0][i] + 0.5 * (p[i] - pts[0][i]) for i in range(n)]
                                  for p in pts[1:]]
    return min(pts, key=f)

if __name__ == "__main__":
    PUB = (0.243, 4.414, 0.793, 2.426)              # CDNOW estimates in FHL (2005)
    r, al, a, b = PUB
    print("A. 2F1(r, b; a+b-1; z) with the published parameters")
    print("   t (weeks)     z       series (terms)          Euler integral     E[X(t)]")
    for t in (39, 78, 520, 5200):
        z = t / (al + t); s, k = f21_series(r, b, a + b - 1, z)
        e = f21_euler(r, b, a + b - 1, z)
        print(f"   {t:5d}     {z:.5f}   {s:12.6f} ({k:6,d})   {e:12.6f}"
              f"     {ex(t, *PUB):6.3f}")
    rng = random.Random(2005)
    sim = simulate(rng, 20_000, *PUB, 78)
    x39 = statistics.fmean(sum(x <= 39 for x in s) for s in sim)
    x78 = statistics.fmean(len(s) for s in sim)
    print(f"   simulation, 20,000 customers: E[X(39)] {x39:.3f}, E[X(78)] {x78:.3f}")
    print("B. fictional cohort of 2,357 simulated with the published parameters")
    cohort = simulate(random.Random(1997), 2357, *PUB, 78)
    cal = [(sum(t <= 39 for t in s), max([t for t in s if t <= 39], default=0.0), 39.0)
           for s in cohort]
    th = nelder_mead(lambda p: -loglik(p, cal), [0.5, 5.0, 0.5, 2.0])
    print("   refitted r {:.3f}, alpha {:.3f}, a {:.3f}, b {:.3f}; log-likelihood {:.1f}"
          .format(*th, loglik(th, cal)))
    hold = sum(sum(39 < t <= 78 for t in s) for s in cohort)
    pred = 2357 * (ex(78, *th) - ex(39, *th))
    print(f"   repeat purchases in weeks 40-78: actual {hold:,},"
          f" BG/NBD forecast {pred:,.0f}")
    for tx in (38.0, 10.0):
        print(f"   4 purchases, last in week {tx:4.1f}: P(alive at week 39)"
              f" {p_alive(4, tx, 39.0, *th):.3f}")
# A. 2F1(r, b; a+b-1; z) with the published parameters
#    t (weeks)     z       series (terms)          Euler integral     E[X(t)]
#       39     0.89833       1.937370 (   211)       1.937370      1.197
#       78     0.94644       2.390006 (   401)       2.390005      1.861
#      520     0.99158       4.674606 ( 2,409)       4.674604      4.975
#     5200     0.99915      11.896629 (21,479)      11.896622     12.148
#    simulation, 20,000 customers: E[X(39)] 1.174, E[X(78)] 1.830
# B. fictional cohort of 2,357 simulated with the published parameters
#    refitted r 0.253, alpha 5.014, a 1.066, b 3.065; log-likelihood -10396.1
#    repeat purchases in weeks 40-78: actual 1,446, BG/NBD forecast 1,329
#    4 purchases, last in week 38.0: P(alive at week 39) 0.838
#    4 purchases, last in week 10.0: P(alive at week 39) 0.055

A 部分中,两种算法在 t=39t = 39 周时都给出 1.937370;t=5,200t = 5{,}200 周(100 年)时 z=0.99915z = 0.99915,级数需要 21,479 项才收敛,积分只用 201 个节点,二者相差不到百万分之一(独立核查用 1−z1-z 连接公式计算,表明此处级数值更准确)。E[X(39)]=1.197E[X(39)] = 1.197、E[X(78)]=1.861E[X(78)] = 1.861,2 万个模拟客户的平均值为 1.174 与 1.830,在抽样误差之内。

B 部分用原文参数模拟了一个虚构的 2,357 人队列,再用前 39 周的数据重新估计参数。rr 与 α\alpha 估得较准(0.253、5.014),aa 与 bb 偏差较大(1.066、3.065),因为在只有 39 周的数据中,「流失」与「购买率低」难以区分。保留期(第 40–78 周)的实际重复购买为 1,446 次,模型预测 1,329 次,低估约 8%。最后两行说明 P(活跃)P(\text{活跃}) 的用处:两个客户都买了 4 次,最后一次分别在第 38 周与第 10 周,前者仍活跃的概率为 0.838,后者只有 0.055。频率相同而最近一次购买不同,价值相差很大,这是「留存率相乘」类方法看不到的。

Gamma-Gamma 客单价模型

BG/NBD 预测购买次数,不预测金额。费德、哈迪与李在另一篇 2005 年的论文中把金额模型与购买模型结合起来,计算客户终身价值 (Fader et al. 2005b);所用的客单价模型的思想来自科伦坡与江(Jiang)1999 年 (Colombo and Jiang 1999)。设一位客户每笔交易的金额服从 Γ(p,ν)\Gamma(p, \nu),ν\nu 在客户之间服从 Γ(q,γ)\Gamma(q, \gamma);给定这位客户 xx 笔交易的平均金额 z‾\bar z,他未来每笔交易的期望金额为

E(Z∣p,q,γ;z‾,x)=p(γ+xz‾)px+q−1,E(Z \mid p, q, \gamma; \bar z, x) = \frac{p(\gamma + x\bar z)}{px + q – 1},

即总体均值 pγ/(q−1)p\gamma/(q-1) 与个人平均 z‾\bar z 的加权平均,交易次数越多,个人平均的权重越大 (Fader and Hardie 2013)。在 CDNOW 数据上(2,357 人中有重复购买的 946 人),估计值为 p=6.25p = 6.25、q=3.74q = 3.74、γ=15.44\gamma = 15.44。

模型的一条关键假设是:客户之间平均交易金额的分布与购买过程相互独立 (Fader and Hardie 2013)。这样才能把「未来购买次数」与「每笔金额」分开估计再相乘。

推导

# Gamma-Gamma spend model (Colombo & Jiang 1999; Fader, Hardie & Lee 2005b): each
# transaction value ~ Gamma(p, nu), nu ~ Gamma(q, gamma) across customers, ASSUMED
# independent of the purchase process. A: fictional repeat buyers simulated with the
# published CDNOW estimates (p 6.25, q 3.74, gamma 15.44), refitted by maximum
# likelihood. B: expected spend E(Z | x, zbar) and a 104-week discounted CLV from the
# BG/NBD conditional expectation (FHL 2005a, eq. 10) for three fictional customers.
# C: when heavy buyers spend less per order, multiplying expected transactions by
# expected spend (the independence assumption) overstates revenue.
import math, random, statistics

# --- helpers repeated from earlier programs, so that this one runs on its own ---
def f21_series(a, b, c, z, tol=1e-12, kmax=200_000):
    term, s, k = 1.0, 1.0, 0
    while abs(term) > tol * abs(s) and k < kmax:
        term *= (a + k) * (b + k) / ((c + k) * (k + 1)) * z; s += term; k += 1
    return s, k

def nelder_mead(f, x0, it=300):
    n = len(x0)
    pts = [list(x0)] + [[x * (1.2 if i == j else 1) for j, x in enumerate(x0)]
                        for i in range(n)]
    for _ in range(it):
        pts.sort(key=f); c = [sum(p[i] for p in pts[:-1]) / n for i in range(n)]
        w = pts[-1]; r = [c[i] + (c[i] - w[i]) for i in range(n)]
        if f(r) < f(pts[0]):
            e = [c[i] + 2 * (c[i] - w[i]) for i in range(n)]
            pts[-1] = e if f(e) < f(r) else r
        elif f(r) < f(pts[-2]):
            pts[-1] = r
        else:
            k = [c[i] + 0.5 * (w[i] - c[i]) for i in range(n)]
            if f(k) < f(w):
                pts[-1] = k
            else:
                pts = [pts[0]] + [[pts[0][i] + 0.5 * (p[i] - pts[0][i]) for i in range(n)]
                                  for p in pts[1:]]
    return min(pts, key=f)
# --- end of repeated helpers ---

BG = (0.243, 4.414, 0.793, 2.426)

def gg_ll(th, data):
    p, q, g = th
    if min(th) <= 0:
        return -1e18
    ll = 0.0
    for x, zbar in data:
        ll += (math.lgamma(p * x + q) - math.lgamma(p * x) - math.lgamma(q)
               + q * math.log(g) + (p * x - 1) * math.log(zbar) + p * x * math.log(x)
               - (p * x + q) * math.log(g + x * zbar))
    return ll

def e_spend(x, zbar, p, q, g):
    return p * (g + x * zbar) / (p * x + q - 1)

def ey(t, x, tx, T, r, al, a, b):
    z = t / (al + T + t)
    num = (a + b + x - 1) / (a - 1) * (1 - ((al + T) / (al + T + t)) ** (r + x)
                                       * f21_series(r + x, b + x, a + b + x - 1, z)[0])
    den = 1 + (x > 0) * a / (b + x - 1) * ((al + T) / (al + tx)) ** (r + x)
    return num / den

def clv(x, tx, T, m, weeks=104, annual=0.10):
    d = (1 + annual) ** (1 / 52); prev, tot = 0.0, 0.0
    for w in range(1, weeks + 1):
        cur = ey(w, x, tx, T, *BG); tot += (cur - prev) / d ** w; prev = cur
    return tot * m, prev

def spenders(rng, n, p, q, g, dep=0.0):
    out = []
    for _ in range(n):
        x = 1 + min(int(rng.expovariate(0.4)), 20)        # repeat transactions
        nu = rng.gammavariate(q, 1 / g) * math.exp(dep * (x - 3))
        out.append((x, statistics.fmean(rng.gammavariate(p, 1 / nu) for _ in range(x)),
                    p / nu))                              # (x, zbar, true mean spend)
    return out

def pearson(x, y):
    """Pearson correlation (statistics.correlation needs Python 3.10+)."""
    mx, my = statistics.fmean(x), statistics.fmean(y)
    sxy = math.fsum((a - mx) * (b - my) for a, b in zip(x, y))
    sxx = math.fsum((a - mx) ** 2 for a in x); syy = math.fsum((b - my) ** 2 for b in y)
    return sxy / math.sqrt(sxx * syy)

if __name__ == "__main__":
    PUB = (6.25, 3.74, 15.44)
    data = spenders(random.Random(1999), 946, *PUB)
    th = nelder_mead(lambda t: -gg_ll(t, [d[:2] for d in data]), [3.0, 3.0, 10.0], 400)
    mean_pop = th[0] * th[2] / (th[1] - 1)
    print(f"A. 946 fictional repeat buyers: refitted p {th[0]:.2f}, q {th[1]:.2f},"
          f" gamma {th[2]:.2f}")
    print(f"   population mean spend {mean_pop:.2f}"
          f" (published parameters: {PUB[0] * PUB[2] / (PUB[1] - 1):.2f})")
    print(f"   correlation of x with average spend: "
          f"{pearson([d[0] for d in data], [d[1] for d in data]):+.3f}")
    print("B. 104-week CLV, 10 % a year, published BG/NBD and Gamma-Gamma parameters")
    print("   customer (x, t_x, zbar)     E[spend]   E[orders, 104 wk]  CLV")
    for x, tx, zb in ((1, 30.0, 60.0), (6, 38.0, 25.0), (6, 12.0, 25.0)):
        m = e_spend(x, zb, *PUB); v, n = clv(x, tx, 39.0, m)
        print(f"   ({x}, {tx:4.1f}, {zb:4.1f})              {m:6.2f}       {n:6.2f}"
              f"        {v:7.2f}")
    print("C. 1,500 fictional buyers: revenue per buyer E[x m] (x repeat orders, m mean")
    print("   spend per order) against E[x] E[m], the product used when spend is")
    print("   modelled separately from frequency")
    for dep, lab in ((0.0, "independent"), (0.25, "heavy buyers spend less")):
        d2 = spenders(random.Random(7), 1500, *PUB, dep)
        xm = statistics.fmean(d[0] * d[2] for d in d2)
        pm = statistics.fmean(d[0] for d in d2) * statistics.fmean(d[2] for d in d2)
        print(f"   {lab:24s} E[x m] {xm:6.1f}   E[x] E[m] {pm:6.1f}"
              f"   ({pm / xm - 1:+.0%})")
# A. 946 fictional repeat buyers: refitted p 7.72, q 3.50, gamma 11.88
#    population mean spend 36.60 (published parameters: 35.22)
#    correlation of x with average spend: -0.075
# B. 104-week CLV, 10 % a year, published BG/NBD and Gamma-Gamma parameters
#    customer (x, t_x, zbar)     E[spend]   E[orders, 104 wk]  CLV
#    (1, 30.0, 60.0)               52.45         1.48          71.41
#    (6, 38.0, 25.0)               25.70         8.40         198.96
#    (6, 12.0, 25.0)               25.70         0.20           4.73
# C. 1,500 fictional buyers: revenue per buyer E[x m] (x repeat orders, m mean
#    spend per order) against E[x] E[m], the product used when spend is
#    modelled separately from frequency
#    independent              E[x m]  106.6   E[x] E[m]  109.4   (+3%)
#    heavy buyers spend less  E[x m]   85.3   E[x] E[m]  126.7   (+49%)

A 部分用原文参数模拟 946 个虚构的重复购买者,重新估计得 p=7.72p = 7.72、q=3.50q = 3.50、γ=11.88\gamma = 11.88;三个参数单独看差别不小,但它们决定的总体平均客单价相近(36.60 对原文参数的 35.22),这是这类模型中参数弱可识别的常见情形。B 部分结合 BG/NBD 的条件期望 E[Y(t)∣x,tx,T]E[Y(t) \mid x, t_x, T] 与 Gamma-Gamma 的期望金额,按年 10% 贴现计算三个虚构客户未来 104 周的价值:只有一次重复购买、但那一次金额很高(60)的客户,期望客单价被收缩到 52.45,价值 71.41;买过 6 次、最后一次在第 38 周的客户价值 198.96;同样买过 6 次但最后一次在第 12 周的客户,期望只剩 0.20 次购买,价值 4.73。

C 部分检验独立性假设。如果常买的客户每笔花得少(作者判断这在促销驱动的电商中可能出现,见本章末的假说),收入的真实期望 E[xm]E[xm] 为 85.3,而把频率与金额分开估计再相乘得到的 E[x]E[m]E[x]E[m] 为 126.7,高估 49%;独立时二者只差 3%。所以在使用 Gamma-Gamma 之前,应当检查交易次数与平均金额的相关性(原文的 CDNOW 数据中二者相关很弱);相关明显时,需要联合建模。

「留存率随时间上升」:异质性,而不是辛普森悖论

契约业务中常见这样一张表:第一年续约率 63%,第二年 74%,第三年 82%……看起来客户「越来越忠诚」。作者框架把它称为辛普森悖论造成的假象。更准确的说法是:这是异质性造成的分选效应。费德与哈迪 2007 年用移位贝塔几何(sBG)模型说明了这一点 (Fader and Hardie 2007):每个客户每期以自己的概率 θ\theta 流失,θ\theta 在客户之间服从 B(α,β)B(\alpha, \beta);个人的流失概率不随时间变化。第 tt 期的留存率为

rt=β+t−1α+β+t−1,r_t = \frac{\beta + t – 1}{\alpha + \beta + t – 1},

随 tt 单调上升,只是因为流失倾向高的人先走了。他们写道,在这个模型中个人层面没有任何时间动态,观察到的留存率上升「只是异质人群中的分选效应」。沃佩尔(Vaupel)与亚辛(Yashin)在人口学中系统讨论过同样的现象:群体的死亡率曲线可以与每一个个体的死亡率曲线形状完全不同 (Vaupel and Yashin 1985)。

推导

# Rising retention from heterogeneity, and why multiplying retention rates misleads.
# A: the shifted-beta-geometric (sBG) model of Fader & Hardie (2007): each customer
#    churns each period with her own probability theta ~ Beta(alpha, beta); fitted by
#    maximum likelihood to the published survival of their "regular" segment, years
#    1-7. Implied retention r_t = (beta + t - 1)/(alpha + beta + t - 1) rises with t
#    although no individual's churn probability changes.
# B: projected survival and discounted expected lifetime (10 % a year) under the sBG
#    and under the naive "year-1 retention multiplied" rule.
# C: a different phenomenon, a true Simpson's paradox: retention falls in both
#    acquisition channels, yet the pooled rate rises because the channel mix shifted.
import math

SURV = [1.0, 0.631, 0.468, 0.382, 0.326, 0.289, 0.262, 0.241]   # FH 2007, Table 1

def sbg_surv(t, al, be):
    return math.exp(math.lgamma(be + t) - math.lgamma(be) + math.lgamma(al + be)
                    - math.lgamma(al + be + t))

def ll(al, be, n=1000):
    if al <= 0 or be <= 0:
        return -1e18
    s = [sbg_surv(t, al, be) for t in range(8)]
    lost = [n * (SURV[t - 1] - SURV[t]) for t in range(1, 8)]
    return (sum(k * math.log(s[t] - s[t + 1]) for t, k in enumerate(lost))
            + n * SURV[7] * math.log(s[7]))

def fit():
    best = (-1e18, 0, 0)
    for i in range(1, 300):                       # coarse grid, then refine
        for j in range(1, 300):
            v = ll(i / 100, j / 100)
            best = max(best, (v, i / 100, j / 100))
    _, al, be = best; step = 0.005
    while step > 1e-6:
        moved = False
        for da, db in ((step, 0), (-step, 0), (0, step), (0, -step)):
            if ll(al + da, be + db) > ll(al, be):
                al, be, moved = al + da, be + db, True
        if not moved:
            step /= 2
    return al, be

if __name__ == "__main__":
    al, be = fit()
    print(f"A. sBG fitted to years 1-7: alpha {al:.3f}, beta {be:.3f}"
          f" (paper: 0.704, 1.182)")
    print("   year   observed survival   sBG survival   sBG retention r_t")
    for t in range(1, 8):
        r = (be + t - 1) / (al + be + t - 1)
        print(f"   {t:4d}        {SURV[t]:6.1%}           {sbg_surv(t, al, be):6.1%}"
              f"          {r:6.1%}")
    r1 = SURV[1]
    print("B. projections from the same first-year retention of 63.1 %")
    for t in (8, 12):
        print(f"   survival at year {t:2d}: sBG {sbg_surv(t, al, be):6.1%},"
              f" naive 0.631^{t} = {r1 ** t:6.1%}")
    d = 0.10
    life_s = sum(sbg_surv(t, al, be) / (1 + d) ** t for t in range(0, 400))
    life_n = sum(r1 ** t / (1 + d) ** t for t in range(0, 400))
    print(f"   discounted expected lifetime (years, incl. year 0, 10 %):"
          f" sBG {life_s:.2f},"
          f" naive {life_n:.2f} ({life_s / life_n - 1:+.0%})")
    print("C. Simpson's paradox: 12-month retention by acquisition channel")
    cohorts = {"2025": {"referral": (800, 0.60), "paid social": (3200, 0.20)},
               "2026": {"referral": (3000, 0.55), "paid social": (1000, 0.15)}}
    for y, ch in cohorts.items():
        tot = sum(n for n, _ in ch.values()); kept = sum(n * p for n, p in ch.values())
        print(f"   cohort {y}: " + ", ".join(f"{k} {p:.0%} of {n:,}"
                                            for k, (n, p) in ch.items())
              + f"; pooled {kept / tot:.1%}")
# A. sBG fitted to years 1-7: alpha 0.704, beta 1.182 (paper: 0.704, 1.182)
#    year   observed survival   sBG survival   sBG retention r_t
#       1         63.1%            62.7%           62.7%
#       2         46.8%            47.4%           75.6%
#       3         38.2%            38.8%           81.9%
#       4         32.6%            33.2%           85.6%
#       5         28.9%            29.2%           88.0%
#       6         26.2%            26.2%           89.8%
#       7         24.1%            23.9%           91.1%
# B. projections from the same first-year retention of 63.1 %
#    survival at year  8: sBG  22.0%, naive 0.631^8 =   2.5%
#    survival at year 12: sBG  17.0%, naive 0.631^12 =   0.4%
#    discounted expected lifetime (years, incl. year 0, 10 %): sBG 3.69, naive 2.35 (+58%)
# C. Simpson's paradox: 12-month retention by acquisition channel
#    cohort 2025: referral 60% of 800, paid social 20% of 3,200; pooled 28.0%
#    cohort 2026: referral 55% of 3,000, paid social 15% of 1,000; pooled 45.0%

A 部分用原文表 1 中「普通」客户前 7 年的存活比例拟合 sBG,得到 α=0.704\alpha = 0.704、β=1.182\beta = 1.182,与原文完全一致;隐含留存率从第一年的 62.7% 升到第七年的 91.1%。B 部分比较两种外推:如果用第一年的留存率 63.1% 连乘,第 8 年只剩 2.5% 的客户,第 12 年 0.4%;sBG 给出 22.0% 与 17.0%。这两个数可以做样本外核对:Fader 等 2018 年公布了同一数据集后续年份的实际存活,第 8–12 年为 22.3%、20.7%、19.4%、18.3%、17.3% (Fader et al. 2018),sBG 只用前 7 年拟合,预测与之相当接近。按年 10% 贴现的期望存续年数,sBG 为 3.69 年,连乘只有 2.35 年;前者比后者高 58%,换言之连乘低估了约 36%。作者框架说「简单留存率相乘」估算 LTV 有误,这一点成立,而且方向通常是低估;费德与哈迪 2010 年专门讨论了在契约情形下忽略异质性给客户群估值带来的危害 (Fader and Hardie 2010)。图 4 画出了这两条曲线。

左:「普通」客户的存活比例(圆点为原文表 1 的观测值)、sBG 拟合与用第一年留存率连乘的外推。右:观测与 sBG 隐含的逐年留存率,虚线为个人平均留存率 \beta/(\alpha+\beta)。

那么辛普森悖论是什么?它指的是:在每一个子群中,两个变量的关系方向相同,把子群合并后方向却反转 (Simpson 1951)。C 部分构造了一个营销中的例子:2026 年的队列中,推荐渠道的 12 个月留存率从 60% 降到 55%,付费社交渠道从 20% 降到 15%,两个渠道都变差了;但合并后的留存率却从 28.0% 升到 45.0%,因为 2026 年的新客户中推荐渠道的比例从 20% 升到了 75%。两种现象的共同点是「组成变化」;区别是:sBG 中没有任何子群的留存率在变化,变化的是存活者的组成;辛普森悖论中每个子群都在变差,变化的是新进入者的组成。前者的对策是用异质性模型外推,后者的对策是按渠道分层报告。

本章小结

BG/NBD 用四个参数描述非契约客户的购买与流失,只依赖频率与最近一次购买;它的期望公式中的 2F1_2F_1 在长期预测时级数收敛很慢,用普法夫变换加变量代换后的欧拉积分可以稳定计算,两种算法互相核对。Gamma-Gamma 模型给出收缩后的期望客单价,但假设金额与频率独立;在常客花得少的虚构例子中,违反这一假设使收入高估 49%。逐期留存率随时间上升,通常是异质性造成的分选效应:sBG 精确重现了费德与哈迪的参数,并显示「留存率连乘」把期望存续年数低估了约三分之一。辛普森悖论是另一件事:子群都变差而合并后变好。

假说 · 尚未验证 · 直播电商客户的金额与频率负相关

作者的假说(尚未确立): 在以限时折扣为主的直播电商中,客户的重复购买次数与其平均订单金额负相关,使 Gamma-Gamma 与 BG/NBD 分开估计再相乘的客户价值系统性高于联合模型的估计。什么样的结果能证明它是错的: 在事先登记的、由独立研究者取得的若干个直播电商客户队列中,重复购买次数与平均订单金额的等级相关系数的合并估计,其 95% 置信区间包含零或为正。

参考文献

Abramowitz, Milton, and Irene A. Stegun, eds. 1964. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. National Bureau of Standards Applied Mathematics Series 55. US Government Printing Office.
Colombo, Richard, and Weina Jiang. 1999. “A Stochastic RFM Model.” Journal of Interactive Marketing 13 (3): 2–12.
Fader, Peter S., and Bruce G. S. Hardie. 2007. “How to Project Customer Retention.” Journal of Interactive Marketing 21 (1): 76–90.
Fader, Peter S., and Bruce G. S. Hardie. 2010. “Customer-Base Valuation in a Contractual Setting: The Perils of Ignoring Heterogeneity.” Marketing Science 29 (1): 85–93.
Fader, Peter S., and Bruce G. S. Hardie. 2013. The Gamma-Gamma Model of Monetary Value. Technical note, brucehardie.com.
Fader, Peter S., Bruce G. S. Hardie, and Ka Lok Lee. 2005a. “‘Counting Your Customers’ the Easy Way: An Alternative to the Pareto/NBD Model.” Marketing Science 24 (2): 275–84. https://doi.org/10.1287/mksc.1040.0098.
Fader, Peter S., Bruce G. S. Hardie, and Ka Lok Lee. 2005b. “RFM and CLV: Using Iso-Value Curves for Customer Base Analysis.” Journal of Marketing Research 42 (4): 415–30.
Fader, Peter S., Bruce G. S. Hardie, Yuzhou Liu, Joseph Davin, and Thomas Steenburgh. 2018. “How to Project Customer Retention” Revisited: The Role of Duration Dependence. Revised manuscript, January 2018, brucehardie.com (papers/037).
Gupta, Sunil, Dominique Hanssens, Bruce Hardie, et al. 2006. “Modeling Customer Lifetime Value.” Journal of Service Research 9 (2): 139–55.
Schmittlein, David C., Donald G. Morrison, and Richard Colombo. 1987. “Counting Your Customers: Who Are They and What Will They Do Next?” Management Science 33 (1): 1–24.
Simpson, E. H. 1951. “The Interpretation of Interaction in Contingency Tables.” Journal of the Royal Statistical Society, Series B 13 (2): 238–41.
Vaupel, James W., and Anatoli I. Yashin. 1985. “Heterogeneity’s Ruses: Some Surprising Effects of Selection on Population Dynamics.” American Statistician 39 (3): 176–85.

Fan, Y. (2026). Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels: From Thompson Sampling Bandits and Absorbing Markov Chains to BG/NBD Customer Lifetime Value, Read Beside the Strategic Dispositions of the Sunzi [算法营销、多触点沙普利归因与获客漏斗的随机控制] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836264 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).