What does Copy and Checkout Arithmetic conclude, in numbers, about “Cognitive Fluency and the Statistical Limits of Testing”?

Answer: Chapter 4 measures the fluency of copy and computes A/B sample sizes (13,914 visitors per arm for 3.0% against 3.6%) and the cost of peeking (false-positive rate 0.256 with 20 looks).

Scope of the numbers: Conversion rates, refund rates and costs not attributed to a source are illustrative; all data are generated by its programs; the book is not legal advice. Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.256, 13914, 20

baselinetargetvisitors per arm
0.0300.03353,211
0.0300.03613,914
0.0300.0452,518
0.1000.11014,751
identical pages,1 looks: false-positive rate 0.044
identical pages,5 looks: false-positive rate 0.134
identical pages, 20 looks: false-positive rate 0.256
Show the full program
# Chapter 4. p08: How many visitors to see a lift - and the peeking trap.
# Two-proportion z test, two-sided alpha 0.05, power 0.80:
#   n per arm = (z_a sqrt(2 p q) + z_b sqrt(p1 q1 + p2 q2))^2 / (p2 - p1)^2.
# Peeking: checking the p-value after every 500 visitors per arm and
# stopping at the first p < 0.05 inflates the false-positive rate even
# when the two pages are identical. Simulated with a fixed seed.
import random
from math import sqrt, ceil
from statistics import NormalDist

N = NormalDist()

def n_per_arm(p1, p2, alpha=0.05, power=0.80):
    za, zb = N.inv_cdf(1 - alpha / 2), N.inv_cdf(power)
    p = (p1 + p2) / 2
    num = (za * sqrt(2 * p * (1 - p))
           + zb * sqrt(p1 * (1 - p1) + p2 * (1 - p2)))
    return ceil(num ** 2 / (p2 - p1) ** 2)

def peeking_fpr(p, looks, step, trials, rng):
    hits = 0
    for _ in range(trials):
        a = b = n = 0
        for _ in range(looks):
            a += sum(rng.random() < p for _ in range(step))
            b += sum(rng.random() < p for _ in range(step))
            n += step
            pool = (a + b) / (2 * n)
            se = sqrt(2 * pool * (1 - pool) / n) or 1
            if abs(a - b) / n / se > 1.959964:
                hits += 1
                break
    return hits / trials

if __name__ == "__main__":
    print("   baseline  target   visitors per arm")
    for p1, p2 in ((0.03, 0.033), (0.03, 0.036), (0.03, 0.045),
                   (0.10, 0.11)):
        print("   {:8.3f}  {:6.3f}   {:,}".format(p1, p2, n_per_arm(p1, p2)))
    rng = random.Random(77008)
    for looks in (1, 5, 20):
        print("   identical pages, {:2d} looks: false-positive rate {:.3f}"
              .format(looks, peeking_fpr(0.05, looks, 500, 1000, rng)))

Single source and citation

Fan, Y. (2026). Persuasive Copywriting Algebra, Loss-Framing Asymmetry and Checkout Micro-Architecture (Version 1.0.0), Chapter 4: Cognitive Fluency and the Statistical Limits of Testing. 5D Library. https://doi.org/10.5281/zenodo.24090205

DOI: 10.5281/zenodo.24090205 · Read the monograph · 5DL-B-077 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026077,
  author    = {Fan, Yafu},
  title     = {Persuasive Copywriting Algebra, Loss-Framing Asymmetry and Checkout Micro-Architecture},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090205},
  url       = {https://doi.org/10.5281/zenodo.24090205}
}

Other chapter answers in this book