Normalised Compression Distance and Kolmogorov Complexity: Detecting Rewritten Copies and Clustering Families of Shared-Origin Sales Scripts: chapter answers
Operators who publish content often hear that replacing three words in ten defeats a duplicate checker, that compression distance is steadier, and that only deep paraphrase fools every tool, so it must be recognised by its numbers and the order of its argument. This book splits that claim into parts that programs can test and tests each on a pseudo-word corpus in which the substitution rate can be set exactly. Its corpora, thresholds and parameters are illustrative, every sample text is generated by program or written by the author, and all calculations are generated by its programs. Its purp…
DOI pending · monograph page goes live after publication · 5DL-B-293 · 中文版
- Chapter 1 · Sweeping the Substitution Rate: Where Literal Matching Fails
Chapter 1 finds that the Jaccard index of word 3-grams falls most steeply at the start of substitution and has lost 0.641 of its range by thirty per cent, with a fixed alarm bar of 0.5 crossed at a rate of 0.211; calibr… - Chapter 2 · Kolmogorov Complexity and Real Compressors: Upper Bounds, Normality and the Window
Chapter 2 shows that compression only bounds Kolmogorov complexity from above, and that once a text exceeds zlib's window the compression distance of a complete copy jumps to 0.9330. - Chapter 3 · Short-Text Bias and Threshold Sensitivity: One Line Cannot Rule Every Length
Chapter 3 measures short-text bias: the power to separate is still 0.9894, yet a fixed threshold gives false positives at a rate of 0.900 at twenty words and catches only 0.030 at 640 words. - Chapter 4 · N-grams, MinHash and Locality-Sensitive Hashing: a First Screen over Millions of Pages
Chapter 4 shows that locality-sensitive hashing designed for near copies misses rewrites, admitting only 0.5088 of thirty per cent rewrites at a knee of 0.4204, so the knee must come down to 0.1250. - Chapter 5 · Deep Paraphrase: Number Fingerprints and Argument-Sequence Fingerprints
Chapter 5 takes on deep paraphrase: the area under the curve of compression distance is only 0.0949, the ordered number fingerprint detects 0.923, and the argument-sequence fingerprint detects only 0.050 because of genr… - Chapter 6 · Hierarchical Clustering of Shared-Origin Script Families
Chapter 6 clusters scripts into families of shared origin by average linkage, with an adjusted Rand index of 0.9962 at a calibrated cut. - Chapter 7 · The Classics Examined: the Colophon of the Cheng House at Meishan and the Statute of Anne
Chapter 7 examines the colophon of the Cheng house at Meishan in the Song edition of the Dongdu shilüe and the Statute of Anne of 1710; under a half-life model, fourteen years cover nine tenths of sales only if the half… - Chapter 8 · Closing the Loop: Weekly Monitoring for Rewritten Copies, De-duplicating Direct-Answer Pages and a Pre-registered Test
Chapter 8 gives 5D Library's weekly monitoring, the de-duplication of direct-answer pages, the Rewrite and Shared-Origin Detector and a pre-registered test plan: library-wide calibration of the number fingerprint cuts f…
Illustrative model calculation, not a live quote