What does Click-to-Chat Closing and Attribution conclude, in numbers, about “Increments and Sample Sizes: Why Most Increments Cannot Be Measured”?

Answer: Increments can only be measured by holding out a control group, and the samples required are often startling. Detecting a relative increment of 0.10 in a cold audience requires 822,401 people per group; comparing close rates of 0.050 and 0.060 at the conversation level requires 8,158 conversations per group, and with 2,000 per group the power is only 0.287. Revenue is noisier still: with a coefficient of variation of 10, detecting an increment of 0.05 requires 627,911 people per group. What a small or mid-sized seller can do is measure upstream metrics, look only for large effects, alternate by week, borrow the platform's scale and register in advance.

Scope of the numbers: Its conversion rates, latencies, reply rates, sample sizes and amounts are illustrative, and all calculations are generated by its programs. It is general business guidance, not legal or marketing-compliance advice, and promises no income.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.05, 0.050, 0.060, 0.10, 0.287, 2000

CV lift people per arm
3.00.02 353,200
3.00.05 56,512
3.00.10 14,128
10.00.02 3,924,440
10.00.05 627,911
10.00.10 156,978
chats per arm simulated power (0.050 vs 0.060)
2,0000.287
4,0000.520
8,1580.805
12,0000.928
Show the full program
# Chapter 4. p42: why noisy sales make ad lift so costly to measure.
# Lewis and Rao (2015) report that a coefficient of variation of 10 in
# individual sales is common. For a lift d (share of mean sales) the
# per-arm size is 2 * (za + zb)^2 * CV^2 / d^2. Then a fixed-seed check
# of power for a chat-level close-rate test, 0.050 vs 0.060, with binomial
# counts drawn by the normal approximation, 4,000 replications.
import math
import random
from statistics import NormalDist

Z = NormalDist()
ZA, ZB = Z.inv_cdf(0.975), Z.inv_cdf(0.80)
print("CV   lift   people per arm")
for cv in (3.0, 10.0):
    for d in (0.02, 0.05, 0.10):
        n = math.ceil(2 * (ZA + ZB) ** 2 * cv * cv / (d * d))
        print("%4.1f %5.2f %16s" % (cv, d, format(n, ",")))


def power(n, p0, p1, reps, rng):
    hits = 0
    for _ in range(reps):
        x0 = round(rng.gauss(n * p0, math.sqrt(n * p0 * (1 - p0))))
        x1 = round(rng.gauss(n * p1, math.sqrt(n * p1 * (1 - p1))))
        pb = (x0 + x1) / (2 * n)
        se = math.sqrt(2 * pb * (1 - pb) / n)
        if se > 0 and abs(x1 - x0) / n / se > ZA:
            hits += 1
    return hits / reps


rng = random.Random(20261120)
print("chats per arm  simulated power (0.050 vs 0.060)")
for n in (2000, 4000, 8158, 12000):
    print("%13s %10.3f" % (format(n, ","), power(n, 0.05, 0.06, 4000,
                                                   rng)))

Single source and citation

Fan, Y. (2026). Click-to-Messenger Conversion Funnels, Chat Attribution and Voice-Note Closing in Southeast Asian Commerce (Version 1.0.0), Chapter 4: Increments and Sample Sizes: Why Most Increments Cannot Be Measured. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-366 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026366,
  author    = {Fan, Yafu},
  title     = {Click-to-Messenger Conversion Funnels, Chat Attribution and Voice-Note Closing in Southeast Asian Commerce},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book