关于「规范化与哈希索引:跨卷定理冲突检测的误报与漏报」,《书库冲突检测、可复现与纠删码》给出的定量结论是什么?

结论:第二章比较四种比对器:只规范化术语的比对器精确率只有 0.073,改用区间相交规则后误报为零。

数据口径:书中的书库规模、错误率、风险率与故障率均为示意参数;全部计算由书中程序生成;本书不是法律、财务、采购或存储服务建议。 第一章把书库画成知识图谱,说明卷数越多错误越难藏身:示意书库到 200 卷时,0.9821 的偏差定义处在可以被另一卷反驳的位置。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.073

statements 1,584, terms 400, terms with a true conflict 30
detector flagged TP FP FN precision recall
L0 raw strings290 24 266 6 0.083 0.800
L1 NFKC, case, spacing331 24 307 6 0.073 0.800
L2 + alias table, value at 3 dp227 27 200 3 0.119 0.900
L3 + interval overlap22 22 0 8 1.000 0.733
L3 misses2 with the conflicting statement under an unlisted
alias, 6 with a change hidden below the printed precision
查看完整验算程序
# Chapter 2. p21: cross-volume conflict detection on a synthetic bilingual
# corpus with four detectors of rising care; term-level false positives
# and false negatives against the known truth.
# Corpus: 400 terms, each stated 2 to 6 times across volumes, written as
# English, case or spacing variants, full-width forms or a Chinese alias;
# values printed at 2, 3 or 4 decimals, sometimes as a percentage. In
# 0.08 of terms one statement carries a true conflict (relative change
# between 0.001 and 0.3). In 0.05 of statements the alias is missing from
# the alias table. Every rate is illustrative; the seed is fixed.
import hashlib
import math
import random
import unicodedata
from decimal import Decimal, ROUND_HALF_EVEN

rng = random.Random(20261112)
TERMS, P_CONFLICT, P_UNLISTED = 400, 0.08, 0.05
ASCII = "0123456789.%abcdefghijklmnopqrstuvwxyz"
FULL = {ord(c): ord(c) + 0xFEE0 for c in ASCII}
FULL[ord(" ")] = 0x3000
ZH = chr(0x672F) + chr(0x8BED)
ALT = chr(0x522B) + chr(0x79F0)


def surface(i, unlisted):
    if unlisted:
        return ZH + ALT + str(i)
    pick = rng.random()
    if pick < 0.4:
        return "term-{}".format(i)
    if pick < 0.55:
        return "Term  {}".format(i)
    if pick < 0.7:
        return "term-{}".format(i).translate(FULL)
    return ZH + str(i)


def printed(val):
    d = rng.choice((2, 3, 3, 4))
    q = val.quantize(Decimal(1).scaleb(-d), ROUND_HALF_EVEN)
    if rng.random() < 0.3:
        pq = (q * 100).quantize(Decimal(1).scaleb(2 - d))
        txt = "{} %".format(pq)
    else:
        txt = str(q)
    if rng.random() < 0.1:
        txt = txt.translate(FULL)
    return txt


rows, truth, lost = [], {}, {}
for i in range(TERMS):
    val = Decimal(rng.randint(1000, 9999)).scaleb(-4)
    reps = rng.randint(2, 6)
    bad = rng.random() < P_CONFLICT
    truth[i] = bad
    for r in range(reps):
        v = val
        if bad and r == reps - 1:
            d = math.exp(rng.uniform(math.log(0.001), math.log(0.3)))
            d = d if rng.random() < 0.5 else -d
            v = (val * Decimal(1 + d)).quantize(Decimal("0.0001"))
        unl = rng.random() < P_UNLISTED
        if bad and r == reps - 1:
            lost[i] = unl
        rows.append((i, surface(i, unl), printed(v)))


def key1(s):
    s = unicodedata.normalize("NFKC", s).casefold()
    return "-".join(s.replace("-", " ").split())


def key2(s):
    k = key1(s)
    if k.startswith(ZH) and k[2:].isdigit():
        k = "term-" + k[2:]
    return k


def interval(txt):
    t = unicodedata.normalize("NFKC", txt).replace(" ", "")
    scale = Decimal(1)
    if t.endswith("%"):
        t, scale = t[:-1], Decimal("0.01")
    x = Decimal(t)
    half = Decimal(1).scaleb(x.as_tuple().exponent) / 2
    return x * scale, half * scale


def detect(level):
    buckets = {}
    for i, term, txt in rows:
        k = term if level == 0 else key1(term) if level == 1 else key2(term)
        h = hashlib.sha256(k.encode("utf-8")).hexdigest()[:16]
        buckets.setdefault(h, []).append((i, txt))
    flagged = set()
    for h in sorted(buckets):
        items = buckets[h]
        if level <= 1:
            vals = {unicodedata.normalize("NFKC", t) if level else t
                    for _, t in items}
            hit = len(vals) > 1
        elif level == 2:
            vals = {interval(t)[0].quantize(Decimal("0.001"))
                    for _, t in items}
            hit = len(vals) > 1
        else:
            iv = [interval(t) for _, t in items]
            hit = max(x - u for x, u in iv) > min(x + u for x, u in iv)
        if hit:
            flagged |= {i for i, _ in items}
    return flagged


real = {i for i in truth if truth[i]}
print("statements {:,}, terms {}, terms with a true conflict {}".format(
    len(rows), TERMS, len(real)))
print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9} {:>6}".format(
    "detector", "flagged", "TP", "FP", "FN", "precision", "recall"))
names = ("L0 raw strings", "L1 NFKC, case, spacing",
         "L2 + alias table, value at 3 dp", "L3 + interval overlap")
for lvl, name in enumerate(names):
    f = detect(lvl)
    tp, fp, fn = len(f & real), len(f - real), len(real - f)
    print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9.3f} {:>6.3f}".format(
        name, len(f), tp, fp, fn, tp / len(f) if f else 0.0,
        tp / len(real)))
missed = sorted(real - detect(3))
alias = len([i for i in missed if lost[i]])
print("L3 misses: {} with the conflicting statement under an unlisted"
      .format(alias))
print("  alias, {} with a change hidden below the printed precision"
      .format(len(missed) - alias))

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《超大规模双语专著库跨卷定理冲突检测、代码可复现性衰减防腐与冷存储纠删码》第 2 章「规范化与哈希索引:跨卷定理冲突检测的误报与漏报」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-182 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026182,
  author    = {Fan, Yafu},
  title     = {Cross-Volume Theorem-Conflict Detection in Large Bilingual Monograph Libraries, Anti-Decay of Code Reproducibility and Erasure-Coded Cold Storage},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论