What does Compression Distance and Rewrite Detection conclude, in numbers, about “Short-Text Bias and Threshold Sensitivity: One Line Cannot Rule Every Length”?

Answer: Chapter 3 measures short-text bias: the power to separate is still 0.9894, yet a fixed threshold gives false positives at a rate of 0.900 at twenty words and catches only 0.030 at 640 words.

Scope of the numbers: Its corpora, thresholds and parameters are illustrative, every sample text is generated by program or written by the author, and all calculations are generated by its programs. It is not individual business advice and not legal advice, and every flag raised by its small tool needs human review.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.030, 0.900, 0.9894, 640

wordsAUCFPR@0.55 TPR@0.55FPR@0.65 TPR@0.65FPR@0.75 TPR@0.75
200.98940.0500.9370.9001.0001.0001.000
400.99890.0000.7470.0500.9930.9801.000
801.00000.0000.3300.0000.9400.2601.000
1601.00000.0000.0470.0000.8000.0001.000
6401.00000.0000.0000.0000.0300.0001.000
Show the full program
# Chapter 3. p32: one fixed NCD threshold, five text lengths.
# A pair is flagged as "same origin" when NCD < t. For each length we
# draw 300 unrelated pairs and 300 rewrites at a 60 % substitution rate
# (corpus of p11) and report, for t = 0.55, 0.65, 0.75, the false-flag
# rate on unrelated pairs (FPR) and the catch rate on rewrites (TPR),
# plus the threshold-free AUC (chance that a rewrite scores below an
# unrelated pair, ties count one half). Corpus illustrative.
import bisect
import math
import random
import zlib

CONS, VOWS = "bdfgklmnprstvz", "aeiou"


def lexicon(seed, n_content=400, n_forms=3, n_func=30):
    rng = random.Random(seed)
    seen = set()

    def word(k):
        while True:
            w = "".join(rng.choice(CONS) + rng.choice(VOWS)
                        for _ in range(k))
            if w not in seen:
                seen.add(w)
                return w
    func = [word(1) for _ in range(n_func)]
    content = [[word(rng.randint(2, 3)) for _ in range(n_forms)]
               for _ in range(n_content)]
    return func, content


def zipf_cum(n):
    w = [1.0 / (k + 1) for k in range(n)]
    tot = math.fsum(w)
    return [math.fsum(w[:k + 1]) / tot for k in range(n)]


FUNC, CONTENT = lexicon(293)
FCUM, CCUM = zipf_cum(len(FUNC)), zipf_cum(len(CONTENT))


def draw(rng, cum):
    return min(bisect.bisect_left(cum, rng.random()), len(cum) - 1)


def script(rng, n_words, p_func=0.4):
    toks = []
    while len(toks) < n_words:
        for _ in range(rng.randint(8, 14)):
            if rng.random() < p_func:
                toks.append(("f", draw(rng, FCUM), 0))
            else:
                toks.append(("c", draw(rng, CCUM), 0))
        toks.append(("p", 0, 0))
    return toks


def render(toks):
    out = []
    for kind, i, form in toks:
        if kind == "f":
            out.append(FUNC[i])
        elif kind == "c":
            out.append(CONTENT[i][form])
        elif out:
            out[-1] += "."
    return " ".join(out)


def substitute(rng, toks, r):
    out = []
    for kind, i, form in toks:
        if kind == "c" and rng.random() < r:
            form = rng.choice([k for k in range(3) if k != form])
        out.append((kind, i, form))
    return out


def C(b):
    return len(zlib.compress(b, 9))


def ncd(bx, by):
    cx, cy, cxy = C(bx), C(by), C(bx + by)
    return (cxy - min(cx, cy)) / max(cx, cy)


def auc(pos, neg):
    win = 0.0
    for p in pos:
        lo = bisect.bisect_left(neg, p)
        hi = bisect.bisect_right(neg, p)
        win += (len(neg) - hi) + 0.5 * (hi - lo)  # step
    return win / (len(pos) * len(neg))


rng = random.Random(20261113)
P = 300
TS = (0.55, 0.65, 0.75)
print("words  AUC     " + "  ".join("FPR@%.2f TPR@%.2f" % (t, t)
                                    for t in TS))
for words in (20, 40, 80, 160, 640):
    neg, pos = [], []
    for _ in range(P):
        t = script(rng, words)
        x = render(t).encode()
        pos.append(ncd(x, render(substitute(rng, t, 0.6)).encode()))
        neg.append(ncd(x, render(script(rng, words)).encode()))
    neg.sort()
    cells = []
    for th in TS:
        fpr = len([v for v in neg if v < th]) / P
        tpr = len([v for v in pos if v < th]) / P
        cells.append("%.3f    %.3f" % (fpr, tpr))
    print("%4d   %.4f  %s" % (words, auc(pos, neg), "    ".join(cells)))

Single source and citation

Fan, Y. (2026). Normalised Compression Distance and Kolmogorov Complexity: Detecting Rewritten Copies and Clustering Families of Shared-Origin Sales Scripts (Version 1.0.0), Chapter 3: Short-Text Bias and Threshold Sensitivity: One Line Cannot Rule Every Length. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-293 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026293,
  author    = {Fan, Yafu},
  title     = {Normalised Compression Distance and Kolmogorov Complexity: Detecting Rewritten Copies and Clustering Families of Shared-Origin Sales Scripts},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book