What does Library Conflicts, Reproducibility and Erasure Codes conclude, in numbers, about “Normalization and Hash Indexing: False Alarms and Misses in Cross-Volume Conflict Detection”?

Answer: Chapter 2 compares four comparators: one that only normalizes terms has a precision of 0.073, while the interval-overlap rule has no false alarms.

Scope of the numbers: Library sizes, error rates, hazards and failure rates are illustrative; all calculations are generated by the programs in the book; the book is not legal, financial, procurement or storage-service advice. Chapter 1 draws the library as a knowledge graph and shows that the more volumes there are, the fewer places an error can hide: at 200 volumes, 0.9821 of deviant definitions in the illustrative library sit where another volume can contradict them.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.073

statements 1,584, terms 400, terms with a true conflict 30
detector flagged TP FP FN precision recall
L0 raw strings290 24 266 6 0.083 0.800
L1 NFKC, case, spacing331 24 307 6 0.073 0.800
L2 + alias table, value at 3 dp227 27 200 3 0.119 0.900
L3 + interval overlap22 22 0 8 1.000 0.733
L3 misses2 with the conflicting statement under an unlisted
alias, 6 with a change hidden below the printed precision
Show the full program
# Chapter 2. p21: cross-volume conflict detection on a synthetic bilingual
# corpus with four detectors of rising care; term-level false positives
# and false negatives against the known truth.
# Corpus: 400 terms, each stated 2 to 6 times across volumes, written as
# English, case or spacing variants, full-width forms or a Chinese alias;
# values printed at 2, 3 or 4 decimals, sometimes as a percentage. In
# 0.08 of terms one statement carries a true conflict (relative change
# between 0.001 and 0.3). In 0.05 of statements the alias is missing from
# the alias table. Every rate is illustrative; the seed is fixed.
import hashlib
import math
import random
import unicodedata
from decimal import Decimal, ROUND_HALF_EVEN

rng = random.Random(20261112)
TERMS, P_CONFLICT, P_UNLISTED = 400, 0.08, 0.05
ASCII = "0123456789.%abcdefghijklmnopqrstuvwxyz"
FULL = {ord(c): ord(c) + 0xFEE0 for c in ASCII}
FULL[ord(" ")] = 0x3000
ZH = chr(0x672F) + chr(0x8BED)
ALT = chr(0x522B) + chr(0x79F0)


def surface(i, unlisted):
    if unlisted:
        return ZH + ALT + str(i)
    pick = rng.random()
    if pick < 0.4:
        return "term-{}".format(i)
    if pick < 0.55:
        return "Term  {}".format(i)
    if pick < 0.7:
        return "term-{}".format(i).translate(FULL)
    return ZH + str(i)


def printed(val):
    d = rng.choice((2, 3, 3, 4))
    q = val.quantize(Decimal(1).scaleb(-d), ROUND_HALF_EVEN)
    if rng.random() < 0.3:
        pq = (q * 100).quantize(Decimal(1).scaleb(2 - d))
        txt = "{} %".format(pq)
    else:
        txt = str(q)
    if rng.random() < 0.1:
        txt = txt.translate(FULL)
    return txt


rows, truth, lost = [], {}, {}
for i in range(TERMS):
    val = Decimal(rng.randint(1000, 9999)).scaleb(-4)
    reps = rng.randint(2, 6)
    bad = rng.random() < P_CONFLICT
    truth[i] = bad
    for r in range(reps):
        v = val
        if bad and r == reps - 1:
            d = math.exp(rng.uniform(math.log(0.001), math.log(0.3)))
            d = d if rng.random() < 0.5 else -d
            v = (val * Decimal(1 + d)).quantize(Decimal("0.0001"))
        unl = rng.random() < P_UNLISTED
        if bad and r == reps - 1:
            lost[i] = unl
        rows.append((i, surface(i, unl), printed(v)))


def key1(s):
    s = unicodedata.normalize("NFKC", s).casefold()
    return "-".join(s.replace("-", " ").split())


def key2(s):
    k = key1(s)
    if k.startswith(ZH) and k[2:].isdigit():
        k = "term-" + k[2:]
    return k


def interval(txt):
    t = unicodedata.normalize("NFKC", txt).replace(" ", "")
    scale = Decimal(1)
    if t.endswith("%"):
        t, scale = t[:-1], Decimal("0.01")
    x = Decimal(t)
    half = Decimal(1).scaleb(x.as_tuple().exponent) / 2
    return x * scale, half * scale


def detect(level):
    buckets = {}
    for i, term, txt in rows:
        k = term if level == 0 else key1(term) if level == 1 else key2(term)
        h = hashlib.sha256(k.encode("utf-8")).hexdigest()[:16]
        buckets.setdefault(h, []).append((i, txt))
    flagged = set()
    for h in sorted(buckets):
        items = buckets[h]
        if level <= 1:
            vals = {unicodedata.normalize("NFKC", t) if level else t
                    for _, t in items}
            hit = len(vals) > 1
        elif level == 2:
            vals = {interval(t)[0].quantize(Decimal("0.001"))
                    for _, t in items}
            hit = len(vals) > 1
        else:
            iv = [interval(t) for _, t in items]
            hit = max(x - u for x, u in iv) > min(x + u for x, u in iv)
        if hit:
            flagged |= {i for i, _ in items}
    return flagged


real = {i for i in truth if truth[i]}
print("statements {:,}, terms {}, terms with a true conflict {}".format(
    len(rows), TERMS, len(real)))
print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9} {:>6}".format(
    "detector", "flagged", "TP", "FP", "FN", "precision", "recall"))
names = ("L0 raw strings", "L1 NFKC, case, spacing",
         "L2 + alias table, value at 3 dp", "L3 + interval overlap")
for lvl, name in enumerate(names):
    f = detect(lvl)
    tp, fp, fn = len(f & real), len(f - real), len(real - f)
    print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9.3f} {:>6.3f}".format(
        name, len(f), tp, fp, fn, tp / len(f) if f else 0.0,
        tp / len(real)))
missed = sorted(real - detect(3))
alias = len([i for i in missed if lost[i]])
print("L3 misses: {} with the conflicting statement under an unlisted"
      .format(alias))
print("  alias, {} with a change hidden below the printed precision"
      .format(len(missed) - alias))

Single source and citation

Fan, Y. (2026). Cross-Volume Theorem-Conflict Detection in Large Bilingual Monograph Libraries, Anti-Decay of Code Reproducibility and Erasure-Coded Cold Storage (Version 1.0.0), Chapter 2: Normalization and Hash Indexing: False Alarms and Misses in Cross-Volume Conflict Detection. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-182 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026182,
  author    = {Fan, Yafu},
  title     = {Cross-Volume Theorem-Conflict Detection in Large Bilingual Monograph Libraries, Anti-Decay of Code Reproducibility and Erasure-Coded Cold Storage},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book