What does Library Conflicts, Reproducibility and Erasure Codes conclude, in numbers, about “Normalization and Hash Indexing: False Alarms and Misses in Cross-Volume Conflict Detection”?
Answer: Chapter 2 compares four comparators: one that only normalizes terms has a precision of 0.073, while the interval-overlap rule has no false alarms.
Scope of the numbers: Library sizes, error rates, hazards and failure rates are illustrative; all calculations are generated by the programs in the book; the book is not legal, financial, procurement or storage-service advice. Chapter 1 draws the library as a knowledge graph and shows that the more volumes there are, the fewer places an error can hide: at 200 volumes, 0.9821 of deviant definitions in the illustrative library sit where another volume can contradict them.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.073
| statements 1,584, terms 400, terms with a true conflict 30 | |
| detector flagged TP FP FN precision recall | |
| L0 raw strings | 290 24 266 6 0.083 0.800 |
| L1 NFKC, case, spacing | 331 24 307 6 0.073 0.800 |
| L2 + alias table, value at 3 dp | 227 27 200 3 0.119 0.900 |
| L3 + interval overlap | 22 22 0 8 1.000 0.733 |
| L3 misses | 2 with the conflicting statement under an unlisted |
| alias, 6 with a change hidden below the printed precision |
Show the full program
# Chapter 2. p21: cross-volume conflict detection on a synthetic bilingual
# corpus with four detectors of rising care; term-level false positives
# and false negatives against the known truth.
# Corpus: 400 terms, each stated 2 to 6 times across volumes, written as
# English, case or spacing variants, full-width forms or a Chinese alias;
# values printed at 2, 3 or 4 decimals, sometimes as a percentage. In
# 0.08 of terms one statement carries a true conflict (relative change
# between 0.001 and 0.3). In 0.05 of statements the alias is missing from
# the alias table. Every rate is illustrative; the seed is fixed.
import hashlib
import math
import random
import unicodedata
from decimal import Decimal, ROUND_HALF_EVEN
rng = random.Random(20261112)
TERMS, P_CONFLICT, P_UNLISTED = 400, 0.08, 0.05
ASCII = "0123456789.%abcdefghijklmnopqrstuvwxyz"
FULL = {ord(c): ord(c) + 0xFEE0 for c in ASCII}
FULL[ord(" ")] = 0x3000
ZH = chr(0x672F) + chr(0x8BED)
ALT = chr(0x522B) + chr(0x79F0)
def surface(i, unlisted):
if unlisted:
return ZH + ALT + str(i)
pick = rng.random()
if pick < 0.4:
return "term-{}".format(i)
if pick < 0.55:
return "Term {}".format(i)
if pick < 0.7:
return "term-{}".format(i).translate(FULL)
return ZH + str(i)
def printed(val):
d = rng.choice((2, 3, 3, 4))
q = val.quantize(Decimal(1).scaleb(-d), ROUND_HALF_EVEN)
if rng.random() < 0.3:
pq = (q * 100).quantize(Decimal(1).scaleb(2 - d))
txt = "{} %".format(pq)
else:
txt = str(q)
if rng.random() < 0.1:
txt = txt.translate(FULL)
return txt
rows, truth, lost = [], {}, {}
for i in range(TERMS):
val = Decimal(rng.randint(1000, 9999)).scaleb(-4)
reps = rng.randint(2, 6)
bad = rng.random() < P_CONFLICT
truth[i] = bad
for r in range(reps):
v = val
if bad and r == reps - 1:
d = math.exp(rng.uniform(math.log(0.001), math.log(0.3)))
d = d if rng.random() < 0.5 else -d
v = (val * Decimal(1 + d)).quantize(Decimal("0.0001"))
unl = rng.random() < P_UNLISTED
if bad and r == reps - 1:
lost[i] = unl
rows.append((i, surface(i, unl), printed(v)))
def key1(s):
s = unicodedata.normalize("NFKC", s).casefold()
return "-".join(s.replace("-", " ").split())
def key2(s):
k = key1(s)
if k.startswith(ZH) and k[2:].isdigit():
k = "term-" + k[2:]
return k
def interval(txt):
t = unicodedata.normalize("NFKC", txt).replace(" ", "")
scale = Decimal(1)
if t.endswith("%"):
t, scale = t[:-1], Decimal("0.01")
x = Decimal(t)
half = Decimal(1).scaleb(x.as_tuple().exponent) / 2
return x * scale, half * scale
def detect(level):
buckets = {}
for i, term, txt in rows:
k = term if level == 0 else key1(term) if level == 1 else key2(term)
h = hashlib.sha256(k.encode("utf-8")).hexdigest()[:16]
buckets.setdefault(h, []).append((i, txt))
flagged = set()
for h in sorted(buckets):
items = buckets[h]
if level <= 1:
vals = {unicodedata.normalize("NFKC", t) if level else t
for _, t in items}
hit = len(vals) > 1
elif level == 2:
vals = {interval(t)[0].quantize(Decimal("0.001"))
for _, t in items}
hit = len(vals) > 1
else:
iv = [interval(t) for _, t in items]
hit = max(x - u for x, u in iv) > min(x + u for x, u in iv)
if hit:
flagged |= {i for i, _ in items}
return flagged
real = {i for i in truth if truth[i]}
print("statements {:,}, terms {}, terms with a true conflict {}".format(
len(rows), TERMS, len(real)))
print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9} {:>6}".format(
"detector", "flagged", "TP", "FP", "FN", "precision", "recall"))
names = ("L0 raw strings", "L1 NFKC, case, spacing",
"L2 + alias table, value at 3 dp", "L3 + interval overlap")
for lvl, name in enumerate(names):
f = detect(lvl)
tp, fp, fn = len(f & real), len(f - real), len(real - f)
print("{:<31} {:>7} {:>5} {:>5} {:>5} {:>9.3f} {:>6.3f}".format(
name, len(f), tp, fp, fn, tp / len(f) if f else 0.0,
tp / len(real)))
missed = sorted(real - detect(3))
alias = len([i for i in missed if lost[i]])
print("L3 misses: {} with the conflicting statement under an unlisted"
.format(alias))
print(" alias, {} with a change hidden below the printed precision"
.format(len(missed) - alias))Single source and citation
Fan, Y. (2026). Cross-Volume Theorem-Conflict Detection in Large Bilingual Monograph Libraries, Anti-Decay of Code Reproducibility and Erasure-Coded Cold Storage (Version 1.0.0), Chapter 2: Normalization and Hash Indexing: False Alarms and Misses in Cross-Volume Conflict Detection. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-182 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026182,
author = {Fan, Yafu},
title = {Cross-Volume Theorem-Conflict Detection in Large Bilingual Monograph Libraries, Anti-Decay of Code Reproducibility and Erasure-Coded Cold Storage},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Library as a Graph: Volumes, Chapters, Terms and Numeric Claims as Nodes
- Chapter 3 · The Sampling Theory of Regression Audits: the Rule of Three and the Finite-Population Correction
- Chapter 4 · The Decay of Code Reproducibility: Hazard Rates of Dependency Drift and Byte-Exact Output Regression
- Chapter 5 · Reed-Solomon over GF(2^8): Erasure Coding and Recovery in the Standard Library
- Chapter 6 · The Reliability of Replicas and Erasure Codes: Binomial Windows, Markov Chains and Scrubbing for Bit Rot
- Chapter 7 · Classical Sources: Siku Collation, the Fate of the Seven Pavilions and Lachmann's Stemmatics
- Chapter 8 · Practice and Falsification: a Publisher's Checklists for Consistency, Reproducibility and Cold Storage