5D Library › Persuasive Copywriting Algebra, Loss-Framing Asymmetry and Checkout Micro-Architecture · 中文

Chapter 4. Cognitive Fluency and the Statistical Limits of Testing

Fluency is a measurable property of text

Cognitive fluency is the ease with which information is processed; more fluent statements are more readily judged true and liked (Alter and Oppenheimer 2009). Part of it can be measured directly. The Flesch reading ease score uses average sentence length and average syllables per word (Flesch 1948); with a crude syllable counter, the program compares three English versions of the same offer (Chinese copy has no equally established formula, but the structure is the same).

Calculation

# Chapter 4. p07: Cognitive fluency as a measurable text property.
# Flesch reading ease = 206.835 - 1.015 (words / sentences)
#                       - 84.6 (syllables / words),
# with a crude vowel-group syllable counter. Three versions of the same
# offer; fluency is a property of the text, independent of its truth.
import re

TEXTS = {
    "dense": ("Utilising our proprietary multidimensional analytical "
              "methodology, the comprehensive personalised report "
              "elucidates contemporaneous configurations affecting "
              "prospective financial outcomes."),
    "plain": ("Your full report shows the dates that matter for money "
              "this month. It takes two minutes to read."),
    "short": ("See your dates for this month. Two minutes. "
              "Full report inside."),
}

def syllables(word):
    w = word.lower().strip(".,")
    groups = re.findall(r"[aeiouy]+", w)
    n = len(groups) - (1 if w.endswith("e") and len(groups) > 1 else 0)
    return max(1, n)

def flesch(text):
    sentences = max(1, len(re.findall(r"[.!?]", text)))
    words = re.findall(r"[A-Za-z']+", text)
    syl = sum(syllables(w) for w in words)
    return (206.835 - 1.015 * len(words) / sentences
            - 84.6 * syl / len(words)), len(words), sentences, syl

if __name__ == "__main__":
    print("   text    words  sentences  syllables  reading ease")
    for name, t in TEXTS.items():
        score, w, s, y = flesch(t)
        print("   {:6s}  {:5d}  {:9d}  {:9d}  {:12.1f}".format(
            name, w, s, y, score))
    print("\n   higher = easier; 60-70 is plain English, below 30 is"
          " academic.")
#    text    words  sentences  syllables  reading ease
#    dense      17          1         64        -128.9
#    plain      18          2         25          80.2
#    short      11          3         16          80.1
#
#    higher = easier; 60-70 is plain English, below 30 is academic.

The version stuffed with long words scores −128.9, nearly unreadable; the plain version 80.2 and the short version 80.1. The two readable versions score almost the same, showing that beyond a point further shortening no longer improves readability. Fluency describes only how easily a text is processed, not whether it is true; precisely because fluency makes things easier to believe, it works just as well for false statements, which is the ledger of Chapter 5.

How many visitors to see a lift

Calculation

# Chapter 4. p08: How many visitors to see a lift - and the peeking trap.
# Two-proportion z test, two-sided alpha 0.05, power 0.80:
#   n per arm = (z_a sqrt(2 p q) + z_b sqrt(p1 q1 + p2 q2))^2 / (p2 - p1)^2.
# Peeking: checking the p-value after every 500 visitors per arm and
# stopping at the first p < 0.05 inflates the false-positive rate even
# when the two pages are identical. Simulated with a fixed seed.
import random
from math import sqrt, ceil
from statistics import NormalDist

N = NormalDist()

def n_per_arm(p1, p2, alpha=0.05, power=0.80):
    za, zb = N.inv_cdf(1 - alpha / 2), N.inv_cdf(power)
    p = (p1 + p2) / 2
    num = (za * sqrt(2 * p * (1 - p))
           + zb * sqrt(p1 * (1 - p1) + p2 * (1 - p2)))
    return ceil(num ** 2 / (p2 - p1) ** 2)

def peeking_fpr(p, looks, step, trials, rng):
    hits = 0
    for _ in range(trials):
        a = b = n = 0
        for _ in range(looks):
            a += sum(rng.random() < p for _ in range(step))
            b += sum(rng.random() < p for _ in range(step))
            n += step
            pool = (a + b) / (2 * n)
            se = sqrt(2 * pool * (1 - pool) / n) or 1
            if abs(a - b) / n / se > 1.959964:
                hits += 1
                break
    return hits / trials

if __name__ == "__main__":
    print("   baseline  target   visitors per arm")
    for p1, p2 in ((0.03, 0.033), (0.03, 0.036), (0.03, 0.045),
                   (0.10, 0.11)):
        print("   {:8.3f}  {:6.3f}   {:,}".format(p1, p2, n_per_arm(p1, p2)))
    rng = random.Random(77008)
    for looks in (1, 5, 20):
        print("   identical pages, {:2d} looks: false-positive rate {:.3f}"
              .format(looks, peeking_fpr(0.05, looks, 500, 1000, rng)))
#    baseline  target   visitors per arm
#       0.030   0.033   53,211
#       0.030   0.036   13,914
#       0.030   0.045   2,518
#       0.100   0.110   14,751
#    identical pages,  1 looks: false-positive rate 0.044
#    identical pages,  5 looks: false-positive rate 0.134
#    identical pages, 20 looks: false-positive rate 0.256

At a 3% baseline, detecting a lift from 3.0% to 3.3% with power 0.80 at two-sided 0.05 needs 53,211 visitors per arm; from 3.0% to 3.6%, 13,914; from 3.0% to 4.5%, only 2,518. At a 10% baseline a one-point lift needs 14,751. For most copy tests in a small shop, the sample is simply too small to detect relative lifts below the mid-teens.

A more common error is “peeking”: checking the p-value every 500 visitors and declaring a winner as soon as it falls below 0.05. Even with two identical pages, the false-positive rate is 0.044 with one look, 0.134 with 5 looks and 0.256 with 20 (Johari et al. 2017).

Note

Two disciplines. First, compute the sample size in advance and fix the stopping rule, or use sequential tests designed for continuous monitoring (Kohavi et al. 2020). Second, for low-traffic shops, put the effort into changes like those of Chapter 3, whose direction is fixed by structure and needs no statistical test.

Summary

Fluency can be measured, but it concerns ease of processing, not truth. Small lifts need tens of thousands of visitors to detect; peeking every 500 visitors for 20 looks pushes the false-positive rate from 0.05 to 0.256.

References

Alter, Adam L., and Daniel M. Oppenheimer. 2009. “Uniting the Tribes of Fluency to Form a Metacognitive Nation.” Personality and Social Psychology Review 13 (3): 219–35.
Flesch, Rudolf. 1948. “A New Readability Yardstick.” Journal of Applied Psychology 32 (3): 221–33.
Johari, Ramesh, Pete Koomen, Leonid Pekelis, and David Walsh. 2017. “Peeking at a/b Tests: Why It Matters, and What to Do about It.” Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1517–25.
Kohavi, Ron, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to a/b Testing. Cambridge University Press.

Fan, Y. (2026). Persuasive Copywriting Algebra, Loss-Framing Asymmetry and Checkout Micro-Architecture: Prospect-Theory Valuation Ratios, Numerical Specificity Credibility and Cognitive-Fluency Conversion Bounds [说服文案代数、损失框架不对称性与结账微界面工程] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090205 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).