What does Anti-Spam Detection and Compliant Bridge Pages conclude, in numbers, about “The Fingerprint of Words: Shingles, the Jaccard Coefficient and Minimum Hashing”?

Answer: Chapter 3 uses shingles and minimum hashing: template posts resemble each other at about 0.522, and two of them are caught by banded detection with a chance of 0.915309.

Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs; it is general business analysis, not personalised business, legal or financial advice, and rules must be checked by a human reviewer before publication. Marked boxes separate published evidence, runnable Python calculations (20 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.522

wtemplate pairs: min Jmean J | original pairs: meanmax
10.8330.842 |0.060 0.188
20.6920.699 |0.002 0.026
30.5170.522 |0.000 0.000
40.3750.381 |0.000 0.000
50.2730.277 |0.000 0.000
w 3: original vs template mean J 0.002 over 64 pairs
w 3: containment of post 1 in each other template post 0.682
w 3: shingles per post, template 22 and original 18.9
Show the full program
# Chapter 3. p31: word shingles, resemblance and containment.
# Model: Broder (1997) turns a text into its set S of w-word shingles
# (contiguous word runs) and measures resemblance r = |S1 & S2| / |S1 | S2|
# (the Jaccard coefficient) and containment c = |S1 & S2| / |S1|.
# Corpus (synthetic): 8 posts filled from one template with a product
# slot and 8 posts written separately on the same products. Shingles
# are hashed to 64-bit integers with hashlib so that p32 can sketch
# them. We report the pairwise Jaccard for shingle widths w = 1 to 5.
import hashlib
import itertools
import math
import statistics

ITEMS = ["desk lamp", "air fryer", "phone stand", "rice cooker",
         "yoga mat", "kettle", "travel bag", "monitor arm"]
TEMPLATE = ("big price drop on the {} today, tap my link to grab the {}"
            " deal before stock runs out and share with friends")
ORIGINAL = [
    "we measured the desk lamp at three heights and the glare on a matte"
    " screen only stayed low at the middle setting",
    "after a month the air fryer basket coating shows wear where the"
    " shaker hits it so check the warranty terms first",
    "the phone stand folds flat but the hinge loosens if you adjust it"
    " daily so tighten the screw every few weeks",
    "rice cooker test: brown rice needed an extra quarter cup of water"
    " and the keep warm mode dried the bottom layer",
    "a thicker yoga mat helped on tile floors but slid on wood, a cotton"
    " towel underneath fixed the slipping",
    "we timed the kettle from cold tap water and it boiled one litre in"
    " just under four minutes at full power",
    "the travel bag fits under most airline seats we tried, though the"
    " side pocket is too shallow for a large bottle",
    "a monitor arm needs a desk edge at least two centimetres thick, our"
    " thinner desk flexed when we moved the screen",
]


def shingles(text, w=3):
    words = text.lower().replace(",", " ").replace(":", " ").split()
    out = set()
    for i in range(len(words) - w + 1):
        g = " ".join(words[i:i + w]).encode("ascii")
        out.add(int.from_bytes(hashlib.md5(g).digest()[:8], "big"))
    return out


def jac(a, b):
    return len(a & b) / len(a | b)


def corpus():
    return [TEMPLATE.format(x, x) for x in ITEMS], ORIGINAL


def pairs_j(sets):
    return [jac(a, b) for a, b in itertools.combinations(sets, 2)]


if __name__ == "__main__":
    tpl, org = corpus()
    print("w   template pairs: min J  mean J | original pairs: mean  max")
    for w in (1, 2, 3, 4, 5):
        T = [shingles(t, w) for t in tpl]
        O = [shingles(t, w) for t in org]
        jt, jo = pairs_j(T), pairs_j(O)
        print("{}  {:>21.3f} {:>7.3f} | {:>21.3f} {:>5.3f}".format(
            w, min(jt), statistics.fmean(jt), statistics.fmean(jo),
            max(jo)))
    T = [shingles(t) for t in tpl]
    O = [shingles(t) for t in org]
    cross = [jac(a, b) for a in O for b in T]
    cont = [len(T[0] & t) / len(T[0]) for t in T[1:]]
    print("w 3: original vs template mean J {:.3f} over {} pairs".format(
        statistics.fmean(cross), len(cross)))
    print("w 3: containment of post 1 in each other template post {:.3f}"
          .format(math.fsum(cont) / len(cont)))
    print("w 3: shingles per post, template {} and original {:.1f}"
          .format(len(T[0]), statistics.fmean(len(o) for o in O)))

Single source and citation

Fan, Y. (2026). Social-Graph Anti-Spam Detection Bounds, Compliant Affiliate Bridge-Page Funnels and the Kinetics of Recovery after Reach Throttling (Version 1.0.0), Chapter 3: The Fingerprint of Words: Shingles, the Jaccard Coefficient and Minimum Hashing. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-342 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026342,
  author    = {Fan, Yafu},
  title     = {Social-Graph Anti-Spam Detection Bounds, Compliant Affiliate Bridge-Page Funnels and the Kinetics of Recovery after Reach Throttling},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book