What does Click-to-Chat Closing and Attribution conclude, in numbers, about “Increments and Sample Sizes: Why Most Increments Cannot Be Measured”?
Answer: Increments can only be measured by holding out a control group, and the samples required are often startling. Detecting a relative increment of 0.10 in a cold audience requires 822,401 people per group; comparing close rates of 0.050 and 0.060 at the conversation level requires 8,158 conversations per group, and with 2,000 per group the power is only 0.287. Revenue is noisier still: with a coefficient of variation of 10, detecting an increment of 0.05 requires 627,911 people per group. What a small or mid-sized seller can do is measure upstream metrics, look only for large effects, alternate by week, borrow the platform's scale and register in advance.
Scope of the numbers: Its conversion rates, latencies, reply rates, sample sizes and amounts are illustrative, and all calculations are generated by its programs. It is general business guidance, not legal or marketing-compliance advice, and promises no income.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.05, 0.050, 0.060, 0.10, 0.287, 2000
| CV lift people per arm | |
| 3.0 | 0.02 353,200 |
| 3.0 | 0.05 56,512 |
| 3.0 | 0.10 14,128 |
| 10.0 | 0.02 3,924,440 |
| 10.0 | 0.05 627,911 |
| 10.0 | 0.10 156,978 |
| chats per arm simulated power (0.050 vs 0.060) | |
| 2,000 | 0.287 |
| 4,000 | 0.520 |
| 8,158 | 0.805 |
| 12,000 | 0.928 |
Show the full program
# Chapter 4. p42: why noisy sales make ad lift so costly to measure.
# Lewis and Rao (2015) report that a coefficient of variation of 10 in
# individual sales is common. For a lift d (share of mean sales) the
# per-arm size is 2 * (za + zb)^2 * CV^2 / d^2. Then a fixed-seed check
# of power for a chat-level close-rate test, 0.050 vs 0.060, with binomial
# counts drawn by the normal approximation, 4,000 replications.
import math
import random
from statistics import NormalDist
Z = NormalDist()
ZA, ZB = Z.inv_cdf(0.975), Z.inv_cdf(0.80)
print("CV lift people per arm")
for cv in (3.0, 10.0):
for d in (0.02, 0.05, 0.10):
n = math.ceil(2 * (ZA + ZB) ** 2 * cv * cv / (d * d))
print("%4.1f %5.2f %16s" % (cv, d, format(n, ",")))
def power(n, p0, p1, reps, rng):
hits = 0
for _ in range(reps):
x0 = round(rng.gauss(n * p0, math.sqrt(n * p0 * (1 - p0))))
x1 = round(rng.gauss(n * p1, math.sqrt(n * p1 * (1 - p1))))
pb = (x0 + x1) / (2 * n)
se = math.sqrt(2 * pb * (1 - pb) / n)
if se > 0 and abs(x1 - x0) / n / se > ZA:
hits += 1
return hits / reps
rng = random.Random(20261120)
print("chats per arm simulated power (0.050 vs 0.060)")
for n in (2000, 4000, 8158, 12000):
print("%13s %10.3f" % (format(n, ","), power(n, 0.05, 0.06, 4000,
rng)))Single source and citation
Fan, Y. (2026). Click-to-Messenger Conversion Funnels, Chat Attribution and Voice-Note Closing in Southeast Asian Commerce (Version 1.0.0), Chapter 4: Increments and Sample Sizes: Why Most Increments Cannot Be Measured. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-366 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026366,
author = {Fan, Yafu},
title = {Click-to-Messenger Conversion Funnels, Chat Attribution and Voice-Note Closing in Southeast Asian Commerce},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Funnel Products and Halving Friction Loss: the Break-Even of Direct-to-Chat Ads
- Chapter 2 · The Decay Curve of First-Response Latency: Triage Before Staff
- Chapter 3 · Chat Attribution: Last Click, the Shapley Value and Severed Sources
- Chapter 5 · Three Labels and the Optimal Stop Point: the Mathematics of Consent-Based Follow-Up
- Chapter 6 · The Length Boundary of Real-Person Voice Notes and One-Tap Payment in the Chat
- Chapter 7 · Classical Sources East and West: Silence in the Guiguzi, Watching Deeds in the Analects, and the Voice in Classical Rhetoric
- Chapter 8 · Actuarial Tools and Execution Checklist: the Weekly Chat Ledger, the Cadence Checker and Pre-Registered Tests