What does Most Conflicts Are Redundancy conclude, in numbers, about “Judging Synonyms: Two Errors of Text Similarity, and Canonical Keys”?

Answer: Chapter 6 shows text similarity recalling only 0.417 of true duplicates at the 0.85 cut-off and proposes canonical keys.

Scope of the numbers: The rule libraries and values are illustrative; every calculation is generated by the book's programs; the book offers general business method, not legal, compliance or business-decision advice, and adding or deleting a rule must be released by its owner. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.417, 0.85

72 sentences, 2556 pairs; rows truth, columns Jaccard band
truth dup>0.85 near below total
same105 0 147 252
number differs54 6 101 161
other meaning0 36 2107 2143
pairs merged at 0.85 that are not duplicates54 of 159
true duplicates missed at 0.85147 of 252 (recall 0.417)
tap 44 vs purchase-path tap 48: Jaccard 0.771
Show the full program
# Chapter 6. p61: text similarity merges conflicts and misses synonyms
# Model: 72 rule sentences written by different hands from 6 meanings
# (metric plus scope), 2 to 3 phrasings each; each writer keeps the
# canonical number with probability 0.7, else shifts it (illustrative,
# fixed seed). Similarity is the Jaccard index of character bigrams
# after lower-casing and removing spaces and punctuation, the method
# of the production pruner; its cut-offs are 0.85 (duplicate) and
# 0.60 (near-synonym). Truth labels: "same" = same meaning and
# number; "number differs" = same meaning, other number (a threshold
# variant to rule on, never to merge); "other meaning".

import random
import re
from collections import Counter
from itertools import combinations

MEANINGS = [
    ("tap", 44, 4, ["Touch targets must be at least {} px on the short "
                    "side.",
                    "Every tap area is no smaller than {} px on its "
                    "short edge.",
                    "Minimum touch target: {} px, short side."]),
    ("tap_buy", 48, 4, ["Purchase-path touch targets must be at least {} "
                        "px on the short side.",
                        "On the purchase path every tap area is no "
                        "smaller than {} px."]),
    ("followups", 2, 1, ["Follow up at most {} times after the customer "
                         "goes silent.",
                         "No more than {} follow-up messages once the "
                         "customer is silent.",
                         "After silence, send at most {} follow-ups."]),
    ("quiet", 21, 1, ["Do not send messages between {}:00 and 09:00.",
                      "Quiet hours run from {}:00 to 09:00; nothing is "
                      "sent."]),
    ("contrast", 45, 5, ["Body text contrast is at least {} to 1.",
                         "Text must reach a contrast ratio of {} to 1 "
                         "or more."]),
    ("pushes", 4, 1, ["At most {} pushes per week.",
                      "Push notifications are capped at {} per week."]),
]


def norm(s):
    return re.sub(r"[\s.,;:!?()\-]", "", s.lower())


def grams(s):
    return {s[i:i + 2] for i in range(len(s) - 1)}


def jac(a, b):
    ga, gb = grams(norm(a)), grams(norm(b))
    return len(ga & gb) / len(ga | gb)


def show(metric, v):
    if metric == "contrast":
        return "%d.%d" % divmod(v, 10)
    return str(v)


rng = random.Random(20261113)
rules = []
for _ in range(72):
    metric, c, step, forms = MEANINGS[rng.randrange(len(MEANINGS))]
    v = c if rng.random() < 0.7 else c + step * rng.choice([-1, 1])
    rules.append((metric, v, rng.choice(forms).format(show(metric, v))))
cells = Counter()
for (m1, v1, t1), (m2, v2, t2) in combinations(rules, 2):
    truth = "other meaning"
    if m1 == m2:
        truth = "same" if v1 == v2 else "number differs"
    j = jac(t1, t2)
    band = "dup>=0.85" if j >= 0.85 else (
        "near" if j >= 0.60 else "below")
    cells[(truth, band)] += 1
print("72 sentences, %d pairs; rows truth, columns Jaccard band"
      % (72 * 71 // 2))
print("truth            dup>=0.85    near   below   total")
for truth in ("same", "number differs", "other meaning"):
    row = [cells[(truth, b)] for b in ("dup>=0.85", "near", "below")]
    print("%-15s  %9d  %6d  %6d  %6d"
          % (truth, row[0], row[1], row[2], row[0] + row[1] + row[2]))
dup = cells[("same", "dup>=0.85")] + cells[("number differs",
                                              "dup>=0.85")] \
    + cells[("other meaning", "dup>=0.85")]
print("pairs merged at 0.85 that are not duplicates: %d of %d"
      % (dup - cells[("same", "dup>=0.85")], dup))
same = cells[("same", "dup>=0.85")] + cells[("same", "near")] \
    + cells[("same", "below")]
print("true duplicates missed at 0.85: %d of %d (recall %.3f)"
      % (same - cells[("same", "dup>=0.85")], same,
         cells[("same", "dup>=0.85")] / same))
a = "Touch targets must be at least 44 px on the short side."
b = "Purchase-path touch targets must be at least 48 px on the short " \
    "side."
print("tap 44 vs purchase-path tap 48: Jaccard %.3f" % jac(a, b))

Single source and citation

Fan, Y. (2026). Cross-Volume Rule Resolution and Axiom Reduction: Detecting Threshold Conflicts among a Thousand Machine Rules and Finding a Minimal Primitive Set (Version 1.0.0), Chapter 6: Judging Synonyms: Two Errors of Text Similarity, and Canonical Keys. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-258 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026258,
  author    = {Fan, Yafu},
  title     = {Cross-Volume Rule Resolution and Axiom Reduction: Detecting Threshold Conflicts among a Thousand Machine Rules and Finding a Minimal Primitive Set},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book