5D Library › Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels · 中文

Chapter 5 What Is a Customer Still Worth? BG/NBD, Gamma-Gamma and the Illusion of Rising Retention

Customers in non-contractual relationships

Subscriptions, telecoms and insurance are contractual relationships: when a customer leaves, the books show it. E-commerce, restaurants and live-stream selling are mostly non-contractual: after a customer’s last purchase we never see him again, and we do not know whether he has churned or simply has not yet reached his next purchase. Estimating customer lifetime value (CLV) needs different models in the two settings (Gupta et al. 2006). This chapter first covers the BG/NBD and Gamma-Gamma models for the non-contractual case, then the “rising retention” seen in contractual settings and how it differs from Simpson’s paradox.

The BG/NBD model

The Pareto/NBD model of Schmittlein, Morrison and Colombo (1987) is the starting point of the field (Schmittlein et al. 1987), but its likelihood requires repeated evaluation of Gauss hypergeometric functions and is numerically awkward. Fader, Hardie and Lee proposed in 2005 an easier-to-compute alternative, the BG/NBD model (Fader et al. 2005a), with these assumptions:

  1. while active, a customer buys according to a Poisson process with rate λ\lambda;
  2. λ\lambda varies across customers as a gamma distribution Γ(r,α)\Gamma(r, \alpha);
  3. after each purchase a customer becomes permanently inactive with probability pp;
  4. pp varies across customers as a beta distribution B(a,b)B(a, b);
  5. λ\lambda and pp are independent across customers.

The difference from Pareto/NBD lies in assumption 3: dropout can happen only just after a purchase, not at any moment in continuous time. This leaves only gamma and beta functions in the likelihood. For a customer with xx repeat purchases in (0,T](0, T], the last at txt_x,

L=B(a,b+x)B(a,b)Γ(r+x)αrΓ(r)(α+T)r+x+δx>0B(a+1,b+x−1)B(a,b)Γ(r+x)αrΓ(r)(α+tx)r+x,L = \frac{B(a, b+x)}{B(a, b)}\,\frac{\Gamma(r+x)\,\alpha^r}{\Gamma(r)\,(\alpha+T)^{r+x}} + \delta_{x>0}\,\frac{B(a+1, b+x-1)}{B(a, b)}\,\frac{\Gamma(r+x)\,\alpha^r}{\Gamma(r)\,(\alpha+t_x)^{r+x}},

which depends only on (x,tx,T)(x, t_x, T), the frequency and recency of marketing’s RFM. The expected number of purchases is

E[X(t)]=a+b−1a−1[1−(αα+t)r2F1(r,b;a+b−1;tα+t)].E[X(t)] = \frac{a+b-1}{a-1}\left[1 – \left(\frac{\alpha}{\alpha+t}\right)^r {}_2F_1\!\left(r, b; a+b-1; \frac{t}{\alpha+t}\right)\right].

The ratio of the two terms of the likelihood gives the posterior probability that a customer is still active at TT: 1/[1+ab+x−1(α+Tα+tx)r+x]1/\bigl[1 + \tfrac{a}{b+x-1}\bigl(\tfrac{\alpha+T}{\alpha+t_x}\bigr)^{r+x}\bigr] for x>0x > 0, and 1 for x=0x = 0 (in the model, dropout happens only after a purchase).

Evidence

The original data and estimates. Fader and co-authors used a 1/10 systematic sample (2,357 people) of the customers who made their first purchase at the CDNOW website in the first quarter of 1997, with the first 39 weeks for calibration and the next 39 as a holdout, and estimated r=0.243r = 0.243, α=4.414\alpha = 4.414, a=0.793a = 0.793, b=2.426b = 2.426, with log-likelihood −9,582.4; Pareto/NBD on the same data gave −9,595.0 (Fader et al. 2005a). They noted that a single evaluation of the Gauss hypergeometric function for given parameters is straightforward, and can be closely approximated with a polynomial series even in a spreadsheet.

Computing 2F1_2F_1: series and integral

The Gauss hypergeometric function 2F1(a,b;c;z)=∑k(a)k(b)k(c)kk!zk_2F_1(a, b; c; z) = \sum_{k} \frac{(a)_k (b)_k}{(c)_k\, k!} z^k converges for |z|<1|z| < 1, but ever more slowly as zz approaches 1. In E[X(t)]E[X(t)], z=t/(α+t)z = t/(\alpha + t) approaches 1 as the forecast horizon grows. Another algorithm is Euler’s integral representation (Abramowitz and Stegun 1964)

2F1(a,b;c;z)=Γ(c)Γ(a)Γ(c−a)∫01sa−1(1−s)c−a−1(1−zs)−bds,c>a>0.{}_2F_1(a, b; c; z) = \frac{\Gamma(c)}{\Gamma(a)\Gamma(c-a)} \int_0^1 s^{a-1}(1-s)^{c-a-1}(1-zs)^{-b}\,ds, \qquad c > a > 0.

There are two numerical traps. First, in BG/NBD a=r=0.243<1a = r = 0.243 < 1, so the integrand diverges at s=0s = 0; second, when zz is near 1, (1−zs)−b(1-zs)^{-b} grows sharply near s=1s = 1. The program first applies Pfaff’s transformation 2F1(a,b;c;z)=(1−z)−a2F1(a,c−b;c;z/(z−1))_2F_1(a, b; c; z) = (1-z)^{-a}\,{}_2F_1(a, c-b; c; z/(z-1)), which moves the argument to the negative half-line where (1−ws)b−c(1 – ws)^{b-c} is bounded on [0,1][0, 1]; then substitutes s=u1/as = u^{1/a}, which turns sa−1dss^{a-1}ds into du/adu/a and removes the endpoint singularity; and integrates by Simpson’s rule.

Calculation

# BG/NBD (Fader, Hardie & Lee 2005). While alive, a customer buys as Poisson(lambda),
# lambda ~ Gamma(r, alpha); after each purchase she drops out with probability p,
# p ~ Beta(a, b). A: E[X(t)] = (a+b-1)/(a-1) [1 - (alpha/(alpha+t))^r 2F1(r, b; a+b-1;
# t/(alpha+t))]; 2F1 by its power series and by Euler's integral after Pfaff's
# transformation and a change of variable, compared as z -> 1, and checked against a
# simulation. B: a FICTIONAL cohort of 2,357 customers simulated with the published
# CDNOW estimates, refitted by maximum likelihood over 39 weeks; holdout forecast for
# weeks 40-78; P(alive) for two customers with the same frequency and recency apart.
import math, random, statistics

def f21_series(a, b, c, z, tol=1e-12, kmax=200_000):
    term, s, k = 1.0, 1.0, 0
    while abs(term) > tol * abs(s) and k < kmax:
        term *= (a + k) * (b + k) / ((c + k) * (k + 1)) * z; s += term; k += 1
    return s, k

def f21_euler(a, b, c, z, n=200):
    """For 0 < a < c, z < 1: Pfaff's transformation 2F1(a,b;c;z) = (1-z)^(-a)
    2F1(a, c-b; c; w), w = z/(z-1) <= 0, then Euler's integral G(c)/(G(a)G(c-a))
    int_0^1 s^(a-1) (1-s)^(c-a-1) (1-ws)^(b-c) ds, whose integrand stays bounded;
    with s = u^(1/a) the weight s^(a-1) ds becomes du / a. Simpson's rule in u."""
    w = z / (z - 1)
    g = lambda u: ((1 - u ** (1 / a)) ** (c - a - 1)
                   * (1 - w * u ** (1 / a)) ** (b - c) / a)
    h = 1 / n; tot = g(0) + g(1)
    tot += sum((4 if i % 2 else 2) * g(i * h) for i in range(1, n))
    lg = math.lgamma(c) - math.lgamma(a) - math.lgamma(c - a)
    return (1 - z) ** (-a) * math.exp(lg) * tot * h / 3

def ex(t, r, al, a, b, f21=f21_euler):
    z = t / (al + t)
    return (a + b - 1) / (a - 1) * (1 - (al / (al + t)) ** r * f21(r, b, a + b - 1, z))

def simulate(rng, n, r, al, a, b, T):
    out = []
    for _ in range(n):
        lam, p = rng.gammavariate(r, 1 / al), rng.betavariate(a, b)
        t, times = 0.0, []
        while True:
            t += rng.expovariate(lam)
            if t > T:
                break
            times.append(t)
            if rng.random() < p:
                break
        out.append(times)
    return out

def loglik(th, data):
    r, al, a, b = th
    if min(th) <= 0:
        return -1e18
    c = math.lgamma(r) - r * math.log(al)
    lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    ll = 0.0
    for x, tx, T in data:
        A1 = math.lgamma(r + x) - c
        t1 = lb(a, b + x) - lb(a, b) + A1 - (r + x) * math.log(al + T)
        if x > 0:
            t2 = lb(a + 1, b + x - 1) - lb(a, b) + A1 - (r + x) * math.log(al + tx)
            m = max(t1, t2); ll += m + math.log(math.exp(t1 - m) + math.exp(t2 - m))
        else:
            ll += t1
    return ll

def p_alive(x, tx, T, r, al, a, b):
    if x == 0:
        return 1.0
    return 1 / (1 + a / (b + x - 1) * ((al + T) / (al + tx)) ** (r + x))

def nelder_mead(f, x0, it=300):
    n = len(x0)
    pts = [list(x0)] + [[x * (1.2 if i == j else 1) for j, x in enumerate(x0)]
                        for i in range(n)]
    for _ in range(it):
        pts.sort(key=f); c = [sum(p[i] for p in pts[:-1]) / n for i in range(n)]
        w = pts[-1]; r = [c[i] + (c[i] - w[i]) for i in range(n)]
        if f(r) < f(pts[0]):
            e = [c[i] + 2 * (c[i] - w[i]) for i in range(n)]
            pts[-1] = e if f(e) < f(r) else r
        elif f(r) < f(pts[-2]):
            pts[-1] = r
        else:
            k = [c[i] + 0.5 * (w[i] - c[i]) for i in range(n)]
            if f(k) < f(w):
                pts[-1] = k
            else:
                pts = [pts[0]] + [[pts[0][i] + 0.5 * (p[i] - pts[0][i]) for i in range(n)]
                                  for p in pts[1:]]
    return min(pts, key=f)

if __name__ == "__main__":
    PUB = (0.243, 4.414, 0.793, 2.426)              # CDNOW estimates in FHL (2005)
    r, al, a, b = PUB
    print("A. 2F1(r, b; a+b-1; z) with the published parameters")
    print("   t (weeks)     z       series (terms)          Euler integral     E[X(t)]")
    for t in (39, 78, 520, 5200):
        z = t / (al + t); s, k = f21_series(r, b, a + b - 1, z)
        e = f21_euler(r, b, a + b - 1, z)
        print(f"   {t:5d}     {z:.5f}   {s:12.6f} ({k:6,d})   {e:12.6f}"
              f"     {ex(t, *PUB):6.3f}")
    rng = random.Random(2005)
    sim = simulate(rng, 20_000, *PUB, 78)
    x39 = statistics.fmean(sum(x <= 39 for x in s) for s in sim)
    x78 = statistics.fmean(len(s) for s in sim)
    print(f"   simulation, 20,000 customers: E[X(39)] {x39:.3f}, E[X(78)] {x78:.3f}")
    print("B. fictional cohort of 2,357 simulated with the published parameters")
    cohort = simulate(random.Random(1997), 2357, *PUB, 78)
    cal = [(sum(t <= 39 for t in s), max([t for t in s if t <= 39], default=0.0), 39.0)
           for s in cohort]
    th = nelder_mead(lambda p: -loglik(p, cal), [0.5, 5.0, 0.5, 2.0])
    print("   refitted r {:.3f}, alpha {:.3f}, a {:.3f}, b {:.3f}; log-likelihood {:.1f}"
          .format(*th, loglik(th, cal)))
    hold = sum(sum(39 < t <= 78 for t in s) for s in cohort)
    pred = 2357 * (ex(78, *th) - ex(39, *th))
    print(f"   repeat purchases in weeks 40-78: actual {hold:,},"
          f" BG/NBD forecast {pred:,.0f}")
    for tx in (38.0, 10.0):
        print(f"   4 purchases, last in week {tx:4.1f}: P(alive at week 39)"
              f" {p_alive(4, tx, 39.0, *th):.3f}")
# A. 2F1(r, b; a+b-1; z) with the published parameters
#    t (weeks)     z       series (terms)          Euler integral     E[X(t)]
#       39     0.89833       1.937370 (   211)       1.937370      1.197
#       78     0.94644       2.390006 (   401)       2.390005      1.861
#      520     0.99158       4.674606 ( 2,409)       4.674604      4.975
#     5200     0.99915      11.896629 (21,479)      11.896622     12.148
#    simulation, 20,000 customers: E[X(39)] 1.174, E[X(78)] 1.830
# B. fictional cohort of 2,357 simulated with the published parameters
#    refitted r 0.253, alpha 5.014, a 1.066, b 3.065; log-likelihood -10396.1
#    repeat purchases in weeks 40-78: actual 1,446, BG/NBD forecast 1,329
#    4 purchases, last in week 38.0: P(alive at week 39) 0.838
#    4 purchases, last in week 10.0: P(alive at week 39) 0.055

In part A, both algorithms give 1.937370 at t=39t = 39 weeks; at t=5,200t = 5{,}200 weeks (100 years) z=0.99915z = 0.99915, the series needs 21,479 terms to converge, the integral uses 201 nodes, and they differ by less than one part in a million (an independent check with the 1−z1-z connection formula shows that here the series value is the more accurate). E[X(39)]=1.197E[X(39)] = 1.197 and E[X(78)]=1.861E[X(78)] = 1.861; the averages over 20,000 simulated customers are 1.174 and 1.830, within sampling error.

Part B simulates a fictional cohort of 2,357 with the published parameters and re-estimates the parameters from the first 39 weeks. rr and α\alpha are recovered fairly well (0.253, 5.014), aa and bb less well (1.066, 3.065), because with only 39 weeks of data “has churned” and “buys rarely” are hard to tell apart. Actual repeat purchases in the holdout (weeks 40–78) are 1,446, the model forecasts 1,329, about 8 % low. The last two lines show what P(alive)P(\text{alive}) is for: two customers have each bought 4 times, the last time in week 38 and in week 10 respectively; the first is still active with probability 0.838, the second only 0.055. Same frequency, different recency, very different value: something that methods based on multiplying retention rates cannot see.

The Gamma-Gamma spend model

BG/NBD forecasts the number of purchases, not their value. In another 2005 paper Fader, Hardie and Lee combined a spend model with the purchase model to compute customer lifetime value (Fader et al. 2005b); the spend model builds on an idea of Colombo and Jiang (1999) (Colombo and Jiang 1999). Suppose the value of each of a customer’s transactions is Γ(p,ν)\Gamma(p, \nu), with ν\nu distributed across customers as Γ(q,γ)\Gamma(q, \gamma); given the average z‾\bar z of a customer’s xx transactions, his expected value per future transaction is

E(Z∣p,q,γ;z‾,x)=p(γ+xz‾)px+q−1,E(Z \mid p, q, \gamma; \bar z, x) = \frac{p(\gamma + x\bar z)}{px + q – 1},

a weighted average of the population mean pγ/(q−1)p\gamma/(q-1) and his own average z‾\bar z, with more weight on his own average the more transactions he has made (Fader and Hardie 2013). On the CDNOW data (the 946 of 2,357 customers with repeat purchases) the estimates are p=6.25p = 6.25, q=3.74q = 3.74, γ=15.44\gamma = 15.44.

A key assumption of the model is that the distribution of average transaction values across customers is independent of the transaction process (Fader and Hardie 2013). Only then can “future number of purchases” and “value per purchase” be estimated separately and multiplied.

Calculation

# Gamma-Gamma spend model (Colombo & Jiang 1999; Fader, Hardie & Lee 2005b): each
# transaction value ~ Gamma(p, nu), nu ~ Gamma(q, gamma) across customers, ASSUMED
# independent of the purchase process. A: fictional repeat buyers simulated with the
# published CDNOW estimates (p 6.25, q 3.74, gamma 15.44), refitted by maximum
# likelihood. B: expected spend E(Z | x, zbar) and a 104-week discounted CLV from the
# BG/NBD conditional expectation (FHL 2005a, eq. 10) for three fictional customers.
# C: when heavy buyers spend less per order, multiplying expected transactions by
# expected spend (the independence assumption) overstates revenue.
import math, random, statistics

# --- helpers repeated from earlier programs, so that this one runs on its own ---
def f21_series(a, b, c, z, tol=1e-12, kmax=200_000):
    term, s, k = 1.0, 1.0, 0
    while abs(term) > tol * abs(s) and k < kmax:
        term *= (a + k) * (b + k) / ((c + k) * (k + 1)) * z; s += term; k += 1
    return s, k

def nelder_mead(f, x0, it=300):
    n = len(x0)
    pts = [list(x0)] + [[x * (1.2 if i == j else 1) for j, x in enumerate(x0)]
                        for i in range(n)]
    for _ in range(it):
        pts.sort(key=f); c = [sum(p[i] for p in pts[:-1]) / n for i in range(n)]
        w = pts[-1]; r = [c[i] + (c[i] - w[i]) for i in range(n)]
        if f(r) < f(pts[0]):
            e = [c[i] + 2 * (c[i] - w[i]) for i in range(n)]
            pts[-1] = e if f(e) < f(r) else r
        elif f(r) < f(pts[-2]):
            pts[-1] = r
        else:
            k = [c[i] + 0.5 * (w[i] - c[i]) for i in range(n)]
            if f(k) < f(w):
                pts[-1] = k
            else:
                pts = [pts[0]] + [[pts[0][i] + 0.5 * (p[i] - pts[0][i]) for i in range(n)]
                                  for p in pts[1:]]
    return min(pts, key=f)
# --- end of repeated helpers ---

BG = (0.243, 4.414, 0.793, 2.426)

def gg_ll(th, data):
    p, q, g = th
    if min(th) <= 0:
        return -1e18
    ll = 0.0
    for x, zbar in data:
        ll += (math.lgamma(p * x + q) - math.lgamma(p * x) - math.lgamma(q)
               + q * math.log(g) + (p * x - 1) * math.log(zbar) + p * x * math.log(x)
               - (p * x + q) * math.log(g + x * zbar))
    return ll

def e_spend(x, zbar, p, q, g):
    return p * (g + x * zbar) / (p * x + q - 1)

def ey(t, x, tx, T, r, al, a, b):
    z = t / (al + T + t)
    num = (a + b + x - 1) / (a - 1) * (1 - ((al + T) / (al + T + t)) ** (r + x)
                                       * f21_series(r + x, b + x, a + b + x - 1, z)[0])
    den = 1 + (x > 0) * a / (b + x - 1) * ((al + T) / (al + tx)) ** (r + x)
    return num / den

def clv(x, tx, T, m, weeks=104, annual=0.10):
    d = (1 + annual) ** (1 / 52); prev, tot = 0.0, 0.0
    for w in range(1, weeks + 1):
        cur = ey(w, x, tx, T, *BG); tot += (cur - prev) / d ** w; prev = cur
    return tot * m, prev

def spenders(rng, n, p, q, g, dep=0.0):
    out = []
    for _ in range(n):
        x = 1 + min(int(rng.expovariate(0.4)), 20)        # repeat transactions
        nu = rng.gammavariate(q, 1 / g) * math.exp(dep * (x - 3))
        out.append((x, statistics.fmean(rng.gammavariate(p, 1 / nu) for _ in range(x)),
                    p / nu))                              # (x, zbar, true mean spend)
    return out

def pearson(x, y):
    """Pearson correlation (statistics.correlation needs Python 3.10+)."""
    mx, my = statistics.fmean(x), statistics.fmean(y)
    sxy = math.fsum((a - mx) * (b - my) for a, b in zip(x, y))
    sxx = math.fsum((a - mx) ** 2 for a in x); syy = math.fsum((b - my) ** 2 for b in y)
    return sxy / math.sqrt(sxx * syy)

if __name__ == "__main__":
    PUB = (6.25, 3.74, 15.44)
    data = spenders(random.Random(1999), 946, *PUB)
    th = nelder_mead(lambda t: -gg_ll(t, [d[:2] for d in data]), [3.0, 3.0, 10.0], 400)
    mean_pop = th[0] * th[2] / (th[1] - 1)
    print(f"A. 946 fictional repeat buyers: refitted p {th[0]:.2f}, q {th[1]:.2f},"
          f" gamma {th[2]:.2f}")
    print(f"   population mean spend {mean_pop:.2f}"
          f" (published parameters: {PUB[0] * PUB[2] / (PUB[1] - 1):.2f})")
    print(f"   correlation of x with average spend: "
          f"{pearson([d[0] for d in data], [d[1] for d in data]):+.3f}")
    print("B. 104-week CLV, 10 % a year, published BG/NBD and Gamma-Gamma parameters")
    print("   customer (x, t_x, zbar)     E[spend]   E[orders, 104 wk]  CLV")
    for x, tx, zb in ((1, 30.0, 60.0), (6, 38.0, 25.0), (6, 12.0, 25.0)):
        m = e_spend(x, zb, *PUB); v, n = clv(x, tx, 39.0, m)
        print(f"   ({x}, {tx:4.1f}, {zb:4.1f})              {m:6.2f}       {n:6.2f}"
              f"        {v:7.2f}")
    print("C. 1,500 fictional buyers: revenue per buyer E[x m] (x repeat orders, m mean")
    print("   spend per order) against E[x] E[m], the product used when spend is")
    print("   modelled separately from frequency")
    for dep, lab in ((0.0, "independent"), (0.25, "heavy buyers spend less")):
        d2 = spenders(random.Random(7), 1500, *PUB, dep)
        xm = statistics.fmean(d[0] * d[2] for d in d2)
        pm = statistics.fmean(d[0] for d in d2) * statistics.fmean(d[2] for d in d2)
        print(f"   {lab:24s} E[x m] {xm:6.1f}   E[x] E[m] {pm:6.1f}"
              f"   ({pm / xm - 1:+.0%})")
# A. 946 fictional repeat buyers: refitted p 7.72, q 3.50, gamma 11.88
#    population mean spend 36.60 (published parameters: 35.22)
#    correlation of x with average spend: -0.075
# B. 104-week CLV, 10 % a year, published BG/NBD and Gamma-Gamma parameters
#    customer (x, t_x, zbar)     E[spend]   E[orders, 104 wk]  CLV
#    (1, 30.0, 60.0)               52.45         1.48          71.41
#    (6, 38.0, 25.0)               25.70         8.40         198.96
#    (6, 12.0, 25.0)               25.70         0.20           4.73
# C. 1,500 fictional buyers: revenue per buyer E[x m] (x repeat orders, m mean
#    spend per order) against E[x] E[m], the product used when spend is
#    modelled separately from frequency
#    independent              E[x m]  106.6   E[x] E[m]  109.4   (+3%)
#    heavy buyers spend less  E[x m]   85.3   E[x] E[m]  126.7   (+49%)

Part A simulates 946 fictional repeat buyers with the published parameters and re-estimates p=7.72p = 7.72, q=3.50q = 3.50, γ=11.88\gamma = 11.88; the three parameters differ noticeably one by one, but the population mean spend they imply is similar (36.60 against 35.22 for the published parameters), the weak identification common in models of this kind. Part B combines the BG/NBD conditional expectation E[Y(t)∣x,tx,T]E[Y(t) \mid x, t_x, T] with the Gamma-Gamma expected spend, discounted at 10 % a year, to value three fictional customers over the next 104 weeks: a customer with a single repeat purchase on which he spent a lot (60) has his expected spend shrunk to 52.45 and is worth 71.41; one who bought 6 times, the last in week 38, is worth 198.96; one who also bought 6 times but last in week 12 is expected to make only 0.20 more purchases and is worth 4.73.

Part C tests the independence assumption. If frequent buyers spend less per order (in the author’s judgement this may occur in promotion-driven e-commerce; see the hypothesis at the end of the chapter), true expected revenue E[xm]E[xm] is 85.3, while estimating frequency and spend separately and multiplying gives E[x]E[m]=126.7E[x]E[m] = 126.7, 49 % too high; under independence the two differ by only 3 %. So before using Gamma-Gamma, one should check the correlation between number of transactions and average spend (it is very weak in the original CDNOW data); when it is clearly present, the two must be modelled jointly.

“Retention rising over time”: heterogeneity, not Simpson’s paradox

Contractual businesses often show a table like this: first-year renewal 63 %, second-year 74 %, third-year 82 % … customers seem to grow “ever more loyal”. The author’s framework calls this an illusion produced by Simpson’s paradox. A more accurate description is a sorting effect of heterogeneity. Fader and Hardie (2007) showed this with the shifted-beta-geometric (sBG) model (Fader and Hardie 2007): each customer churns each period with her own probability θ\theta, distributed across customers as B(α,β)B(\alpha, \beta); no individual’s churn probability changes over time. The retention rate in period tt is

rt=β+t−1α+β+t−1,r_t = \frac{\beta + t – 1}{\alpha + \beta + t – 1},

which rises monotonically with tt only because the customers most prone to churn leave first. In their words, there are no underlying time dynamics at the level of the individual customer; the observed rise in retention rates is simply a sorting effect in a heterogeneous population. Vaupel and Yashin discussed the same phenomenon systematically in demography: the mortality curve of a population can have a shape quite unlike that of every individual in it (Vaupel and Yashin 1985).

Calculation

# Rising retention from heterogeneity, and why multiplying retention rates misleads.
# A: the shifted-beta-geometric (sBG) model of Fader & Hardie (2007): each customer
#    churns each period with her own probability theta ~ Beta(alpha, beta); fitted by
#    maximum likelihood to the published survival of their "regular" segment, years
#    1-7. Implied retention r_t = (beta + t - 1)/(alpha + beta + t - 1) rises with t
#    although no individual's churn probability changes.
# B: projected survival and discounted expected lifetime (10 % a year) under the sBG
#    and under the naive "year-1 retention multiplied" rule.
# C: a different phenomenon, a true Simpson's paradox: retention falls in both
#    acquisition channels, yet the pooled rate rises because the channel mix shifted.
import math

SURV = [1.0, 0.631, 0.468, 0.382, 0.326, 0.289, 0.262, 0.241]   # FH 2007, Table 1

def sbg_surv(t, al, be):
    return math.exp(math.lgamma(be + t) - math.lgamma(be) + math.lgamma(al + be)
                    - math.lgamma(al + be + t))

def ll(al, be, n=1000):
    if al <= 0 or be <= 0:
        return -1e18
    s = [sbg_surv(t, al, be) for t in range(8)]
    lost = [n * (SURV[t - 1] - SURV[t]) for t in range(1, 8)]
    return (sum(k * math.log(s[t] - s[t + 1]) for t, k in enumerate(lost))
            + n * SURV[7] * math.log(s[7]))

def fit():
    best = (-1e18, 0, 0)
    for i in range(1, 300):                       # coarse grid, then refine
        for j in range(1, 300):
            v = ll(i / 100, j / 100)
            best = max(best, (v, i / 100, j / 100))
    _, al, be = best; step = 0.005
    while step > 1e-6:
        moved = False
        for da, db in ((step, 0), (-step, 0), (0, step), (0, -step)):
            if ll(al + da, be + db) > ll(al, be):
                al, be, moved = al + da, be + db, True
        if not moved:
            step /= 2
    return al, be

if __name__ == "__main__":
    al, be = fit()
    print(f"A. sBG fitted to years 1-7: alpha {al:.3f}, beta {be:.3f}"
          f" (paper: 0.704, 1.182)")
    print("   year   observed survival   sBG survival   sBG retention r_t")
    for t in range(1, 8):
        r = (be + t - 1) / (al + be + t - 1)
        print(f"   {t:4d}        {SURV[t]:6.1%}           {sbg_surv(t, al, be):6.1%}"
              f"          {r:6.1%}")
    r1 = SURV[1]
    print("B. projections from the same first-year retention of 63.1 %")
    for t in (8, 12):
        print(f"   survival at year {t:2d}: sBG {sbg_surv(t, al, be):6.1%},"
              f" naive 0.631^{t} = {r1 ** t:6.1%}")
    d = 0.10
    life_s = sum(sbg_surv(t, al, be) / (1 + d) ** t for t in range(0, 400))
    life_n = sum(r1 ** t / (1 + d) ** t for t in range(0, 400))
    print(f"   discounted expected lifetime (years, incl. year 0, 10 %):"
          f" sBG {life_s:.2f},"
          f" naive {life_n:.2f} ({life_s / life_n - 1:+.0%})")
    print("C. Simpson's paradox: 12-month retention by acquisition channel")
    cohorts = {"2025": {"referral": (800, 0.60), "paid social": (3200, 0.20)},
               "2026": {"referral": (3000, 0.55), "paid social": (1000, 0.15)}}
    for y, ch in cohorts.items():
        tot = sum(n for n, _ in ch.values()); kept = sum(n * p for n, p in ch.values())
        print(f"   cohort {y}: " + ", ".join(f"{k} {p:.0%} of {n:,}"
                                            for k, (n, p) in ch.items())
              + f"; pooled {kept / tot:.1%}")
# A. sBG fitted to years 1-7: alpha 0.704, beta 1.182 (paper: 0.704, 1.182)
#    year   observed survival   sBG survival   sBG retention r_t
#       1         63.1%            62.7%           62.7%
#       2         46.8%            47.4%           75.6%
#       3         38.2%            38.8%           81.9%
#       4         32.6%            33.2%           85.6%
#       5         28.9%            29.2%           88.0%
#       6         26.2%            26.2%           89.8%
#       7         24.1%            23.9%           91.1%
# B. projections from the same first-year retention of 63.1 %
#    survival at year  8: sBG  22.0%, naive 0.631^8 =   2.5%
#    survival at year 12: sBG  17.0%, naive 0.631^12 =   0.4%
#    discounted expected lifetime (years, incl. year 0, 10 %): sBG 3.69, naive 2.35 (+58%)
# C. Simpson's paradox: 12-month retention by acquisition channel
#    cohort 2025: referral 60% of 800, paid social 20% of 3,200; pooled 28.0%
#    cohort 2026: referral 55% of 3,000, paid social 15% of 1,000; pooled 45.0%

Part A fits the sBG to the surviving shares of the “regular” customers in the first 7 years of the paper’s Table 1 and obtains α=0.704\alpha = 0.704, β=1.182\beta = 1.182, identical to the paper; the implied retention rate rises from 62.7 % in year 1 to 91.1 % in year 7. Part B compares two extrapolations: multiplying the first-year retention of 63.1 % forward leaves 2.5 % of customers in year 8 and 0.4 % in year 12; the sBG gives 22.0 % and 17.0 %. These two numbers allow an out-of-sample check: Fader and co-authors (2018) published the actual survival of the same data set in later years, 22.3 %, 20.7 %, 19.4 %, 18.3 % and 17.3 % for years 8–12 (Fader et al. 2018); the sBG, fitted to the first 7 years only, comes quite close. The expected discounted lifetime at 10 % a year is 3.69 years under the sBG and only 2.35 years by multiplication; the former is 58 % higher, in other words multiplication understates by about 36 %. The author’s framework is right that “simply multiplying retention rates” misestimates LTV, and the error is usually an understatement; Fader and Hardie (2010) discussed specifically the perils of ignoring heterogeneity when valuing a contractual customer base (Fader and Hardie 2010). Figure 4 shows both curves.

Left: surviving share of “regular” customers (dots are the observed values of the paper’s Table 1), the sBG fit, and extrapolation by multiplying first-year retention. Right: observed and sBG-implied year-by-year retention; the dotted line is the mean individual retention \beta/(\alpha+\beta).

What, then, is Simpson’s paradox? It is an association that has the same direction within every subgroup and reverses when the subgroups are pooled (Simpson 1951). Part C builds a marketing example: in the 2026 cohort, 12-month retention of the referral channel falls from 60 % to 55 % and of the paid-social channel from 20 % to 15 %, so both channels got worse; yet pooled retention rises from 28.0 % to 45.0 %, because the referral share of new customers rose from 20 % to 75 %. What the two phenomena share is “a change of composition”; they differ in that in the sBG no subgroup’s retention changes and what changes is the composition of the survivors, whereas in Simpson’s paradox every subgroup gets worse and what changes is the composition of the new entrants. The remedy for the first is to extrapolate with a heterogeneity model; for the second, to report by channel.

Summary

BG/NBD describes the purchasing and dropout of non-contractual customers with four parameters and depends only on frequency and recency; the series for the 2F1_2F_1 in its expectation converges very slowly for long horizons, and Euler’s integral after Pfaff’s transformation and a change of variable computes it stably, the two algorithms checking each other. The Gamma-Gamma model gives a shrunken expected spend but assumes spend and frequency are independent; in a fictional example in which frequent buyers spend less, violating the assumption overstates revenue by 49 %. Period retention rising over time is usually a sorting effect of heterogeneity: the sBG reproduces Fader and Hardie’s parameters exactly and shows that “multiplying retention rates” understates expected lifetime by about a third. Simpson’s paradox is a different thing: every subgroup gets worse while the pooled figure improves.

Hypothesis · not established · Spend and frequency are negatively related in live-stream commerce

The author’s hypothesis (not established): in live-stream commerce built mainly on flash discounts, customers’ number of repeat purchases is negatively correlated with their average order value, so that customer values computed by estimating Gamma-Gamma and BG/NBD separately and multiplying are systematically higher than joint-model estimates. What result would show it to be wrong: in several pre-registered live-stream commerce customer cohorts obtained by independent researchers, the 95 % confidence interval of the pooled rank correlation between number of repeat purchases and average order value contains zero or is positive.

References

Abramowitz, Milton, and Irene A. Stegun, eds. 1964. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. National Bureau of Standards Applied Mathematics Series 55. US Government Printing Office.
Colombo, Richard, and Weina Jiang. 1999. “A Stochastic RFM Model.” Journal of Interactive Marketing 13 (3): 2–12.
Fader, Peter S., and Bruce G. S. Hardie. 2007. “How to Project Customer Retention.” Journal of Interactive Marketing 21 (1): 76–90.
Fader, Peter S., and Bruce G. S. Hardie. 2010. “Customer-Base Valuation in a Contractual Setting: The Perils of Ignoring Heterogeneity.” Marketing Science 29 (1): 85–93.
Fader, Peter S., and Bruce G. S. Hardie. 2013. The Gamma-Gamma Model of Monetary Value. Technical note, brucehardie.com.
Fader, Peter S., Bruce G. S. Hardie, and Ka Lok Lee. 2005a. “‘Counting Your Customers’ the Easy Way: An Alternative to the Pareto/NBD Model.” Marketing Science 24 (2): 275–84. https://doi.org/10.1287/mksc.1040.0098.
Fader, Peter S., Bruce G. S. Hardie, and Ka Lok Lee. 2005b. “RFM and CLV: Using Iso-Value Curves for Customer Base Analysis.” Journal of Marketing Research 42 (4): 415–30.
Fader, Peter S., Bruce G. S. Hardie, Yuzhou Liu, Joseph Davin, and Thomas Steenburgh. 2018. “How to Project Customer Retention” Revisited: The Role of Duration Dependence. Revised manuscript, January 2018, brucehardie.com (papers/037).
Gupta, Sunil, Dominique Hanssens, Bruce Hardie, et al. 2006. “Modeling Customer Lifetime Value.” Journal of Service Research 9 (2): 139–55.
Schmittlein, David C., Donald G. Morrison, and Richard Colombo. 1987. “Counting Your Customers: Who Are They and What Will They Do Next?” Management Science 33 (1): 1–24.
Simpson, E. H. 1951. “The Interpretation of Interaction in Contingency Tables.” Journal of the Royal Statistical Society, Series B 13 (2): 238–41.
Vaupel, James W., and Anatoli I. Yashin. 1985. “Heterogeneity’s Ruses: Some Surprising Effects of Selection on Population Dynamics.” American Statistician 39 (3): 176–85.

Fan, Y. (2026). Algorithmic Marketing, Multi-Touch Shapley Attribution and Stochastic Control of Acquisition Funnels: From Thompson Sampling Bandits and Absorbing Markov Chains to BG/NBD Customer Lifetime Value, Read Beside the Strategic Dispositions of the Sunzi [算法营销、多触点沙普利归因与获客漏斗的随机控制] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836264 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).