What does Paywall Recall and Upsell Ratchets conclude, in numbers, about “Testing Message Variants: Sequential Tests, Multi-Armed Bandits and the False Discovery Rate”?

Answer: The upper boundary is a log-likelihood ratio of 2.7726 and the lower one -1.5581. A one-sided fixed-sample test with the same error rates needs 1,985 messages. When the true recovery rate is 0.030, the sequential test stops after only 982 messages on average, with a median of 697 and a ninetieth percentile of 2,078, and wrongly declares an effect in 0.045 of runs, below 0.05. When the true rate is 0.040, it needs 1,262 messages on average, with a median of 1,008, and correctly declares an effect in 0.795 of runs, close to the 0.80 that was set.

Scope of the numbers: The site is treated as a paid report product, with no claim of predictive accuracy and no fear; all visitor and order data are synthetic or illustrative. Marked boxes separate published evidence, runnable Python calculations (25 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: -1.5581, 0.030, 0.040, 0.045, 0.795, 1008

boundaries: accept H1 at log LR 2.7726, accept H0 at -1.5581
fixed-sample one-sided test with the same alpha and power: n = 1,985
truth H0, p = 0.030mean n982median69790th pct2,078says H1 0.045
truth H1, p = 0.040mean n1,262median1,00890th pct2,478says H1 0.795
truth between, p = 0.035mean n1,498median1,10690th pct3,241says H1 0.396
Show the full program
# Chapter 6. p61: Wald's sequential probability ratio test for one recall
# variant against a known baseline recovery rate.
# H0: wave-1 recovery p = 0.030; H1: p = 0.040 (a third better).
# alpha = 0.05, beta = 0.20 (power 0.80); Wald's boundaries A = (1 - beta) / alpha and
# B = beta / (1 - alpha). Simulated 2,000 times under each truth (fixed
# seed), against the fixed-sample size with the same error rates.
import math
import random
from statistics import NormalDist

P0, P1, ALPHA, BETA = 0.030, 0.040, 0.05, 0.20
LA, LB = math.log((1 - BETA) / ALPHA), math.log(BETA / (1 - ALPHA))
S1, S0 = math.log(P1 / P0), math.log((1 - P1) / (1 - P0))


def sprt(rng, p, cap=60000):
    k = 0
    for n in range(1, cap + 1):
        k = k + (1 if rng.random() < p else 0)
        llr = k * S1 + (n - k) * S0
        if llr >= LA:
            return n, 1
        if llr <= LB:
            return n, 0
    return cap, -1


def fixed_n():
    z = NormalDist()
    za, zb = z.inv_cdf(1 - ALPHA), z.inv_cdf(1 - BETA)
    num = za * math.sqrt(P0 * (1 - P0)) + zb * math.sqrt(P1 * (1 - P1))
    return math.ceil((num / (P1 - P0)) ** 2)


if __name__ == "__main__":
    rng = random.Random(20261120)
    print("boundaries: accept H1 at log LR {:.4f}, accept H0 at {:.4f}".format(LA, LB))
    print("fixed-sample one-sided test with the same alpha and power: n = {:,}".format(fixed_n()))
    for name, p in (("truth H0, p = 0.030", P0), ("truth H1, p = 0.040", P1),
                    ("truth between, p = 0.035", 0.035)):
        runs = [sprt(rng, p) for _ in range(2000)]
        ns = sorted(r[0] for r in runs)
        ok = len([1 for r in runs if r[1] == 1]) / len(runs)
        print("{:<26} mean n {:>6,.0f}  median {:>6,}  90th pct {:>6,}  says H1 {:.3f}".format(
            name, math.fsum(ns) / len(ns), ns[len(ns) // 2], ns[int(0.9 * len(ns))], ok))

Single source and citation

Fan, Y. (2026). Paywall Abandonment Recovery Timing, Personalised Curiosity Recall and High-Ticket Upsell Ratchets (Version 1.0.0), Chapter 6: Testing Message Variants: Sequential Tests, Multi-Armed Bandits and the False Discovery Rate. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-349 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026349,
  author    = {Fan, Yafu},
  title     = {Paywall Abandonment Recovery Timing, Personalised Curiosity Recall and High-Ticket Upsell Ratchets},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book