关于「规范化与哈希索引:跨卷定理冲突检测的误报与漏报」,《书库冲突检测、可复现与纠删码》给出的定量结论是什么?
结论:第二章比较四种比对器:只规范化术语的比对器精确率只有 0.073,改用区间相交规则后误报为零。
数据口径:书中的书库规模、错误率、风险率与故障率均为示意参数;全部计算由书中程序生成;本书不是法律、财务、采购或存储服务建议。 第一章把书库画成知识图谱,说明卷数越多错误越难藏身:示意书库到 200 卷时,0.9821 的偏差定义处在可以被另一卷反驳的位置。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.073
| statements 1,584, terms 400, terms with a true conflict 30 | |
| detector flagged TP FP FN precision recall | |
| L0 raw strings | 290 24 266 6 0.083 0.800 |
| L1 NFKC, case, spacing | 331 24 307 6 0.073 0.800 |
| L2 + alias table, value at 3 dp | 227 27 200 3 0.119 0.900 |
| L3 + interval overlap | 22 22 0 8 1.000 0.733 |
| L3 misses | 2 with the conflicting statement under an unlisted |
| alias, 6 with a change hidden below the printed precision |
查看完整验算程序
# Chapter 2. p21: cross-volume conflict detection on a synthetic bilingual
# corpus with four detectors of rising care; term-level false positives
# and false negatives against the known truth.
# Corpus: 400 terms, each stated 2 to 6 times across volumes, written as
# English, case or spacing variants, full-width forms or a Chinese alias;
# values printed at 2, 3 or 4 decimals, sometimes as a percentage. In
# 0.08 of terms one statement carries a true conflict (relative change
# between 0.001 and 0.3). In 0.05 of statements the alias is missing from
# the alias table. Every rate is illustrative; the seed is fixed.
import hashlib
import math
import random
import unicodedata
from decimal import Decimal, ROUND_HALF_EVEN
rng = random.Random(20261112)
TERMS, P_CONFLICT, P_UNLISTED = 400, 0.08, 0.05
ASCII = "0123456789.%abcdefghijklmnopqrstuvwxyz"
FULL = {ord(c): ord(c) + 0xFEE0 for c in ASCII}
FULL[ord(" ")] = 0x3000
ZH = chr(0x672F) + chr(0x8BED)
ALT = chr(0x522B) + chr(0x79F0)
def surface(i, unlisted):
if unlisted:
return ZH + ALT + str(i)
pick = rng.random()
if pick < 0.4:
return "term-{}".format(i)
if pick < 0.55:
return "Term {}".format(i)
if pick < 0.7:
return "term-{}".format(i).translate(FULL)
return ZH + str(i)
def printed(val):
d = rng.choice((2, 3, 3, 4))
q = val.quantize(Decimal(1).scaleb(-d), ROUND_HALF_EVEN)
if rng.random() < 0.3:
pq = (q * 100).quantize(Decimal(1).scaleb(2 - d))
txt = "{} %".format(pq)
else:
txt = str(q)
if rng.random() < 0.1:
txt = txt.translate(FULL)
return txt
rows, truth, lost = [], {}, {}
for i in range(TERMS):
val = Decimal(rng.randint(1000, 9999)).scaleb(-4)
reps = rng.randint(2, 6)
bad = rng.random() < P_CONFLICT
truth[i] = bad
for r in range(reps):
v = val
if bad and r == reps - 1:
d = math.exp(rng.uniform(math.log(0.001), math.log(0.3)))
d = d if rng.random() < 0.5 else -d
v = (val * Decimal(1 + d)).quantize(Decimal("0.0001"))
unl = rng.random() < P_UNLISTED
if bad and r == reps - 1:
lost[i] = unl
rows.append((i, surface(i, unl), printed(v)))
def key1(s):
s = unicodedata.normalize("NFKC", s).casefold()
return "-".join(s.replace("-", " ").split())
def key2(s):
k = key1(s)
if k.startswith(ZH) and k[2:].isdigit():
k = "term-" + k[2:]
return k
def interval(txt):
t = unicodedata.normalize("NFKC", txt).replace(" ", "")
scale = Decimal(1)
if t.endswith("%"):
t, scale = t[:-1], Decimal("0.01")
x = Decimal(t)
half = Decimal(1).scaleb(x.as_tuple().exponent) / 2
return x * scale, half * scale
def detect(level):
buckets = {}
for i, term, txt in rows:
k = term if level == 0 else key1(term) if level == 1 else key2(term)
h = hashlib.sha256(k.encode("utf-8")).hexdigest()[:16]
buckets.setdefault(h, []).append((i, txt))
flagged = set()
for h in sorted(buckets):
items = buckets[h]
if level <= 1:
vals = {unicodedata.normalize("NFKC", t) if level else t
for _, t in items}
hit = len(vals) > 1
elif level == 2:
vals = {interval(t)[0].quantize(Decimal("0.001"))
for _, t in items}
hit = len(vals) > 1
else:
iv = [interval(t) for _, t in items]
hit = max(x - u for x, u in iv) > min(x + u for x, u in iv)
if hit:
flagged |= {i for i, _ in items}
return flagged
real = {i for i in truth if truth[i]}
print("statements {:,}, terms {}, terms with a true conflict {}".format(
len(rows), TERMS, len(real)))
print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9} {:>6}".format(
"detector", "flagged", "TP", "FP", "FN", "precision", "recall"))
names = ("L0 raw strings", "L1 NFKC, case, spacing",
"L2 + alias table, value at 3 dp", "L3 + interval overlap")
for lvl, name in enumerate(names):
f = detect(lvl)
tp, fp, fn = len(f & real), len(f - real), len(real - f)
print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9.3f} {:>6.3f}".format(
name, len(f), tp, fp, fn, tp / len(f) if f else 0.0,
tp / len(real)))
missed = sorted(real - detect(3))
alias = len([i for i in missed if lost[i]])
print("L3 misses: {} with the conflicting statement under an unlisted"
.format(alias))
print(" alias, {} with a change hidden below the printed precision"
.format(len(missed) - alias))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《超大规模双语专著库跨卷定理冲突检测、代码可复现性衰减防腐与冷存储纠删码》第 2 章「规范化与哈希索引:跨卷定理冲突检测的误报与漏报」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-182 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026182,
author = {Fan, Yafu},
title = {Cross-Volume Theorem-Conflict Detection in Large Bilingual Monograph Libraries, Anti-Decay of Code Reproducibility and Erasure-Coded Cold Storage},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}