What does Compression Distance and Rewrite Detection conclude, in numbers, about “Short-Text Bias and Threshold Sensitivity: One Line Cannot Rule Every Length”?
Answer: Chapter 3 measures short-text bias: the power to separate is still 0.9894, yet a fixed threshold gives false positives at a rate of 0.900 at twenty words and catches only 0.030 at 640 words.
Scope of the numbers: Its corpora, thresholds and parameters are illustrative, every sample text is generated by program or written by the author, and all calculations are generated by its programs. It is not individual business advice and not legal advice, and every flag raised by its small tool needs human review.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.030, 0.900, 0.9894, 640
| words | AUC | FPR@0.55 TPR@0.55 | FPR@0.65 TPR@0.65 | FPR@0.75 TPR@0.75 | |||
| 20 | 0.9894 | 0.050 | 0.937 | 0.900 | 1.000 | 1.000 | 1.000 |
| 40 | 0.9989 | 0.000 | 0.747 | 0.050 | 0.993 | 0.980 | 1.000 |
| 80 | 1.0000 | 0.000 | 0.330 | 0.000 | 0.940 | 0.260 | 1.000 |
| 160 | 1.0000 | 0.000 | 0.047 | 0.000 | 0.800 | 0.000 | 1.000 |
| 640 | 1.0000 | 0.000 | 0.000 | 0.000 | 0.030 | 0.000 | 1.000 |
Show the full program
# Chapter 3. p32: one fixed NCD threshold, five text lengths.
# A pair is flagged as "same origin" when NCD < t. For each length we
# draw 300 unrelated pairs and 300 rewrites at a 60 % substitution rate
# (corpus of p11) and report, for t = 0.55, 0.65, 0.75, the false-flag
# rate on unrelated pairs (FPR) and the catch rate on rewrites (TPR),
# plus the threshold-free AUC (chance that a rewrite scores below an
# unrelated pair, ties count one half). Corpus illustrative.
import bisect
import math
import random
import zlib
CONS, VOWS = "bdfgklmnprstvz", "aeiou"
def lexicon(seed, n_content=400, n_forms=3, n_func=30):
rng = random.Random(seed)
seen = set()
def word(k):
while True:
w = "".join(rng.choice(CONS) + rng.choice(VOWS)
for _ in range(k))
if w not in seen:
seen.add(w)
return w
func = [word(1) for _ in range(n_func)]
content = [[word(rng.randint(2, 3)) for _ in range(n_forms)]
for _ in range(n_content)]
return func, content
def zipf_cum(n):
w = [1.0 / (k + 1) for k in range(n)]
tot = math.fsum(w)
return [math.fsum(w[:k + 1]) / tot for k in range(n)]
FUNC, CONTENT = lexicon(293)
FCUM, CCUM = zipf_cum(len(FUNC)), zipf_cum(len(CONTENT))
def draw(rng, cum):
return min(bisect.bisect_left(cum, rng.random()), len(cum) - 1)
def script(rng, n_words, p_func=0.4):
toks = []
while len(toks) < n_words:
for _ in range(rng.randint(8, 14)):
if rng.random() < p_func:
toks.append(("f", draw(rng, FCUM), 0))
else:
toks.append(("c", draw(rng, CCUM), 0))
toks.append(("p", 0, 0))
return toks
def render(toks):
out = []
for kind, i, form in toks:
if kind == "f":
out.append(FUNC[i])
elif kind == "c":
out.append(CONTENT[i][form])
elif out:
out[-1] += "."
return " ".join(out)
def substitute(rng, toks, r):
out = []
for kind, i, form in toks:
if kind == "c" and rng.random() < r:
form = rng.choice([k for k in range(3) if k != form])
out.append((kind, i, form))
return out
def C(b):
return len(zlib.compress(b, 9))
def ncd(bx, by):
cx, cy, cxy = C(bx), C(by), C(bx + by)
return (cxy - min(cx, cy)) / max(cx, cy)
def auc(pos, neg):
win = 0.0
for p in pos:
lo = bisect.bisect_left(neg, p)
hi = bisect.bisect_right(neg, p)
win += (len(neg) - hi) + 0.5 * (hi - lo) # step
return win / (len(pos) * len(neg))
rng = random.Random(20261113)
P = 300
TS = (0.55, 0.65, 0.75)
print("words AUC " + " ".join("FPR@%.2f TPR@%.2f" % (t, t)
for t in TS))
for words in (20, 40, 80, 160, 640):
neg, pos = [], []
for _ in range(P):
t = script(rng, words)
x = render(t).encode()
pos.append(ncd(x, render(substitute(rng, t, 0.6)).encode()))
neg.append(ncd(x, render(script(rng, words)).encode()))
neg.sort()
cells = []
for th in TS:
fpr = len([v for v in neg if v < th]) / P
tpr = len([v for v in pos if v < th]) / P
cells.append("%.3f %.3f" % (fpr, tpr))
print("%4d %.4f %s" % (words, auc(pos, neg), " ".join(cells)))Single source and citation
Fan, Y. (2026). Normalised Compression Distance and Kolmogorov Complexity: Detecting Rewritten Copies and Clustering Families of Shared-Origin Sales Scripts (Version 1.0.0), Chapter 3: Short-Text Bias and Threshold Sensitivity: One Line Cannot Rule Every Length. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-293 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026293,
author = {Fan, Yafu},
title = {Normalised Compression Distance and Kolmogorov Complexity: Detecting Rewritten Copies and Clustering Families of Shared-Origin Sales Scripts},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Sweeping the Substitution Rate: Where Literal Matching Fails
- Chapter 2 · Kolmogorov Complexity and Real Compressors: Upper Bounds, Normality and the Window
- Chapter 4 · N-grams, MinHash and Locality-Sensitive Hashing: a First Screen over Millions of Pages
- Chapter 5 · Deep Paraphrase: Number Fingerprints and Argument-Sequence Fingerprints
- Chapter 6 · Hierarchical Clustering of Shared-Origin Script Families
- Chapter 7 · The Classics Examined: the Colophon of the Cheng House at Meishan and the Statute of Anne
- Chapter 8 · Closing the Loop: Weekly Monitoring for Rewritten Copies, De-duplicating Direct-Answer Pages and a Pre-registered Test