关于「短文本偏差与阈值敏感度:一条线管不了所有长度」,《压缩距离与洗稿识别》给出的定量结论是什么?
结论:第三章量化短文本偏差:区分能力仍有 0.9894,固定阈值却在二十词上误报 0.900、在 640 词上只检出 0.030。
数据口径:书中的语料、阈值与参数均为示意,所有样本文本由程序生成或作者自拟;全部计算由书中程序生成。 本书的用途是保护原创与发现抄袭,不讨论如何改写他人内容以逃避查重;本书是非个人化经营建议,不是法律意见,书中小工具的每一条标记都须人工复核。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.030, 0.900, 0.9894, 640
| words | AUC | FPR@0.55 TPR@0.55 | FPR@0.65 TPR@0.65 | FPR@0.75 TPR@0.75 | |||
| 20 | 0.9894 | 0.050 | 0.937 | 0.900 | 1.000 | 1.000 | 1.000 |
| 40 | 0.9989 | 0.000 | 0.747 | 0.050 | 0.993 | 0.980 | 1.000 |
| 80 | 1.0000 | 0.000 | 0.330 | 0.000 | 0.940 | 0.260 | 1.000 |
| 160 | 1.0000 | 0.000 | 0.047 | 0.000 | 0.800 | 0.000 | 1.000 |
| 640 | 1.0000 | 0.000 | 0.000 | 0.000 | 0.030 | 0.000 | 1.000 |
查看完整验算程序
# Chapter 3. p32: one fixed NCD threshold, five text lengths.
# A pair is flagged as "same origin" when NCD < t. For each length we
# draw 300 unrelated pairs and 300 rewrites at a 60 % substitution rate
# (corpus of p11) and report, for t = 0.55, 0.65, 0.75, the false-flag
# rate on unrelated pairs (FPR) and the catch rate on rewrites (TPR),
# plus the threshold-free AUC (chance that a rewrite scores below an
# unrelated pair, ties count one half). Corpus illustrative.
import bisect
import math
import random
import zlib
CONS, VOWS = "bdfgklmnprstvz", "aeiou"
def lexicon(seed, n_content=400, n_forms=3, n_func=30):
rng = random.Random(seed)
seen = set()
def word(k):
while True:
w = "".join(rng.choice(CONS) + rng.choice(VOWS)
for _ in range(k))
if w not in seen:
seen.add(w)
return w
func = [word(1) for _ in range(n_func)]
content = [[word(rng.randint(2, 3)) for _ in range(n_forms)]
for _ in range(n_content)]
return func, content
def zipf_cum(n):
w = [1.0 / (k + 1) for k in range(n)]
tot = math.fsum(w)
return [math.fsum(w[:k + 1]) / tot for k in range(n)]
FUNC, CONTENT = lexicon(293)
FCUM, CCUM = zipf_cum(len(FUNC)), zipf_cum(len(CONTENT))
def draw(rng, cum):
return min(bisect.bisect_left(cum, rng.random()), len(cum) - 1)
def script(rng, n_words, p_func=0.4):
toks = []
while len(toks) < n_words:
for _ in range(rng.randint(8, 14)):
if rng.random() < p_func:
toks.append(("f", draw(rng, FCUM), 0))
else:
toks.append(("c", draw(rng, CCUM), 0))
toks.append(("p", 0, 0))
return toks
def render(toks):
out = []
for kind, i, form in toks:
if kind == "f":
out.append(FUNC[i])
elif kind == "c":
out.append(CONTENT[i][form])
elif out:
out[-1] += "."
return " ".join(out)
def substitute(rng, toks, r):
out = []
for kind, i, form in toks:
if kind == "c" and rng.random() < r:
form = rng.choice([k for k in range(3) if k != form])
out.append((kind, i, form))
return out
def C(b):
return len(zlib.compress(b, 9))
def ncd(bx, by):
cx, cy, cxy = C(bx), C(by), C(bx + by)
return (cxy - min(cx, cy)) / max(cx, cy)
def auc(pos, neg):
win = 0.0
for p in pos:
lo = bisect.bisect_left(neg, p)
hi = bisect.bisect_right(neg, p)
win += (len(neg) - hi) + 0.5 * (hi - lo) # step
return win / (len(pos) * len(neg))
rng = random.Random(20261113)
P = 300
TS = (0.55, 0.65, 0.75)
print("words AUC " + " ".join("FPR@%.2f TPR@%.2f" % (t, t)
for t in TS))
for words in (20, 40, 80, 160, 640):
neg, pos = [], []
for _ in range(P):
t = script(rng, words)
x = render(t).encode()
pos.append(ncd(x, render(substitute(rng, t, 0.6)).encode()))
neg.append(ncd(x, render(script(rng, words)).encode()))
neg.sort()
cells = []
for th in TS:
fpr = len([v for v in neg if v < th]) / P
tpr = len([v for v in pos if v < th]) / P
cells.append("%.3f %.3f" % (fpr, tpr))
print("%4d %.4f %s" % (words, auc(pos, neg), " ".join(cells)))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《归一化压缩距离与柯尔莫哥洛夫复杂度、洗稿伪原创识别与同源话术家族聚类母典》第 3 章「短文本偏差与阈值敏感度:一条线管不了所有长度」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-293 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026293,
author = {Fan, Yafu},
title = {Normalised Compression Distance and Kolmogorov Complexity: Detecting Rewritten Copies and Clustering Families of Shared-Origin Sales Scripts},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}