关于「换词率扫描:字面查重在哪里失效」,《压缩距离与洗稿识别》给出的定量结论是什么?
结论:第一章发现,词三元组杰卡德指数在换词之初下降最陡,到三成时已丢掉 0.641 的可跌幅度,固定报警线 0.5 在换词率 0.211 处被越过;按无关文本校准后,实词全换时字面片段仍有 0.957 的检出率,压缩距离为 0.000,它只在「保留比例」上下降更缓(0.551 对 0.385)。
数据口径:书中的语料、阈值与参数均为示意,所有样本文本由程序生成或作者自拟;全部计算由书中程序生成。 本书的用途是保护原创与发现抄袭,不讨论如何改写他人内容以逃避查重;本书是非个人化经营建议,不是法律意见,书中小工具的每一条标记都须人工复核。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.000, 0.385, 0.5, 0.551, 0.957
| 150-word scripts, 300 rewrites per rate, 400 null pairs | ||||||
| null J3: mean 0.0012 | p99 0.0109 | null NCD: median 0.8140 | p01 0.7723 | ||||
| rate | J3 | NCD | J3 kept | NCD kept | det J3 | det NCD |
| 0.0 | 1.000 | 0.041 | 1.000 | 1.000 | 1.000 | 1.000 |
| 0.1 | 0.719 | 0.173 | 0.719 | 0.829 | 1.000 | 1.000 |
| 0.2 | 0.517 | 0.292 | 0.517 | 0.675 | 1.000 | 1.000 |
| 0.3 | 0.385 | 0.388 | 0.385 | 0.551 | 1.000 | 1.000 |
| 0.4 | 0.281 | 0.479 | 0.280 | 0.433 | 1.000 | 1.000 |
| 0.5 | 0.209 | 0.550 | 0.208 | 0.341 | 1.000 | 1.000 |
| 0.6 | 0.156 | 0.615 | 0.155 | 0.257 | 1.000 | 1.000 |
| 0.7 | 0.109 | 0.681 | 0.108 | 0.172 | 1.000 | 1.000 |
| 0.8 | 0.078 | 0.736 | 0.077 | 0.101 | 1.000 | 0.910 |
| 0.9 | 0.051 | 0.787 | 0.050 | 0.035 | 0.993 | 0.293 |
| 1.0 | 0.033 | 0.838 | 0.032 | -0.031 | 0.957 | 0.000 |
| fixed bar 0.5: J3 first below at r = 0.3, 1 - NCD at r = 0.5 |
核心公式
\[\mathrm{NCD}(x,y)=\frac{C(xy)-\min\{C(x),C(y)\}}{\max\{C(x),C(y)\}}.\]
查看完整验算程序
# Chapter 1. p11: substitution-rate sweep, word 3-gram Jaccard vs zlib NCD.
# Corpus (illustrative, self-made): pseudo-words built from syllables; 30
# function words with one form, 400 content concepts with 3 synonymous
# forms each, Zipf frequencies. A script is 150 words; a rewrite replaces
# a share r of content words by another synonym of the same concept.
# NCD(x, y) = (C(xy) - min(C(x), C(y))) / max(C(x), C(y)), C = zlib -9.
# Thresholds are calibrated on 400 unrelated pairs (null): J3 above the
# null 99th percentile, NCD below the null 1st percentile.
import bisect
import math
import random
import zlib
CONS, VOWS = "bdfgklmnprstvz", "aeiou"
def lexicon(seed, n_content=400, n_forms=3, n_func=30):
# pseudo-word lexicon (illustrative, self-made)
rng = random.Random(seed)
seen = set()
def word(k):
while True:
w = "".join(rng.choice(CONS) + rng.choice(VOWS)
for _ in range(k))
if w not in seen:
seen.add(w)
return w
func = [word(1) for _ in range(n_func)]
content = [[word(rng.randint(2, 3)) for _ in range(n_forms)]
for _ in range(n_content)]
return func, content
def zipf_cum(n):
w = [1.0 / (k + 1) for k in range(n)]
tot = math.fsum(w)
return [math.fsum(w[:k + 1]) / tot for k in range(n)]
FUNC, CONTENT = lexicon(293)
FCUM, CCUM = zipf_cum(len(FUNC)), zipf_cum(len(CONTENT))
def draw(rng, cum):
return min(bisect.bisect_left(cum, rng.random()), len(cum) - 1)
def script(rng, n_words, p_func=0.4):
toks = []
while len(toks) < n_words:
for _ in range(rng.randint(8, 14)):
if rng.random() < p_func:
toks.append(("f", draw(rng, FCUM), 0))
else:
toks.append(("c", draw(rng, CCUM), 0))
toks.append(("p", 0, 0))
return toks
def render(toks):
out = []
for kind, i, form in toks:
if kind == "f":
out.append(FUNC[i])
elif kind == "c":
out.append(CONTENT[i][form])
else:
out[-1] += "."
return " ".join(out)
def substitute(rng, toks, r):
out = []
for kind, i, form in toks:
if kind == "c" and rng.random() < r:
form = rng.choice([k for k in range(3) if k != form])
out.append((kind, i, form))
return out
def clen(b):
return len(zlib.compress(b, 9))
def ncd(x, y):
bx, by = x.encode(), y.encode()
cx, cy, cxy = clen(bx), clen(by), clen(bx + b" " + by)
return (cxy - min(cx, cy)) / max(cx, cy)
def shingles(text, n=3):
w = text.split()
return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
def jac(a, b):
return len(a & b) / len(a | b)
rng = random.Random(20261113)
NW, NULL, PAIRS = 150, 400, 300
nj, nn = [], []
for _ in range(NULL):
a, b = render(script(rng, NW)), render(script(rng, NW))
nj.append(jac(shingles(a), shingles(b)))
nn.append(ncd(a, b))
nj.sort()
nn.sort()
tj, tn = nj[int(0.99 * NULL)], nn[int(0.01 * NULL)]
mj, mn = math.fsum(nj) / NULL, nn[NULL // 2]
print("150-word scripts, %d rewrites per rate, %d null pairs"
% (PAIRS, NULL))
print("null J3: mean %.4f p99 %.4f | null NCD: median %.4f p01 %.4f"
% (mj, tj, mn, tn))
print("rate J3 NCD J3 kept NCD kept det J3 det NCD")
rows = []
for r10 in range(11):
r = r10 / 10.0
js, ns = [], []
for _ in range(PAIRS):
t = script(rng, NW)
a, b = render(t), render(substitute(rng, t, r))
js.append(jac(shingles(a), shingles(b)))
ns.append(ncd(a, b))
rows.append((r, math.fsum(js) / PAIRS, math.fsum(ns) / PAIRS,
len([x for x in js if x > tj]) / PAIRS,
len([x for x in ns if x < tn]) / PAIRS))
n0 = rows[0][2]
for r, j, n, dj, dn in rows:
kj = (j - mj) / (1.0 - mj)
kn = (mn - n) / (mn - n0)
print("%.1f %.3f %.3f %.3f %.3f %.3f %.3f"
% (r, j, n, kj, kn, dj, dn))
cj = [r for r, j, n, dj, dn in rows if j < 0.5][0]
cn = [r for r, j, n, dj, dn in rows if 1.0 - n < 0.5][0]
print("fixed bar 0.5: J3 first below at r = %.1f, 1 - NCD at r = %.1f"
% (cj, cn))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《归一化压缩距离与柯尔莫哥洛夫复杂度、洗稿伪原创识别与同源话术家族聚类母典》第 1 章「换词率扫描:字面查重在哪里失效」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-293 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026293,
author = {Fan, Yafu},
title = {Normalised Compression Distance and Kolmogorov Complexity: Detecting Rewritten Copies and Clustering Families of Shared-Origin Sales Scripts},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}