What does Chunking and Blending conclude, in numbers, about “Cook Ding and the Phaedrus: Two Ancient Arguments for Cutting at the Natural Joints”?

Answer: Chapter 7 examines Cook Ding carving the ox in the Zhuangzi and Plato's division at the natural joints in the Phaedrus: cutting by paragraph gives zero overlap and zero breakage, fixed windows of 64 at an overlap of one half still break 0.401 of the answers, and misread joints quickly erase this advantage.

Scope of the numbers: Implementing BM25 and a concept-level bag-of-words cosine in standard-library programs on fixed-seed synthetic corpora, the book corrects the blueprint in three places. Parameters are illustrative and all numbers come from the book's programs, which read and write no file and connect to no live knowledge base.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.401, 64

chunkermean-lenmax-lenindexbroken
joints (paragraph)64.11121.000.000
fixed w= 64 o=0.0064.0641.000.633
fixed w= 64 o=0.2564.0641.330.515
fixed w= 64 o=0.5064.0642.000.401
fixed w=112 o=0.00112.01121.000.341
fixed w=112 o=0.50112.01122.000.018
Show the full program
# Chapter 7. p71: cutting at the joints versus hacking at fixed windows.
# Model: a stream of 20000 paragraphs, lengths uniform on 16..112 tokens
# (seed 20261113); each paragraph is one answer. A paragraph is broken
# when no single chunk holds at least 80 % of it. Joint chunking cuts
# exactly at paragraph ends; fixed windows ignore them. Illustrative.
import math
import random

rng = random.Random(20261113)
lens = [rng.randint(16, 112) for _ in range(20000)]
starts, pos = [], 0
for ln in lens:
    starts.append(pos)
    pos += ln
total = pos


def broken(w, s):
    bad = 0
    for a, ln in zip(starts, lens):
        j0 = max(0, (a - w) // s)
        best = 0
        for j in range(j0, a // s + 2):
            lo = j * s
            best = max(best, min(lo + w, a + ln) - max(lo, a))
        bad += best < 0.8 * ln
    return bad / len(lens)


print("chunker            mean-len  max-len  index  broken")
print("joints (paragraph) %8.1f  %7d  %5.2f  %6.3f" % (
    math.fsum(lens) / len(lens), max(lens), 1.0, 0.0))
for w, o in ((64, 0.0), (64, 0.25), (64, 0.5), (112, 0.0), (112, 0.5)):
    s = int(round(w * (1 - o)))
    print("fixed w=%3d o=%.2f %8.1f  %7d  %5.2f  %6.3f" % (
        w, o, w, w, w / s, broken(w, s)))

Single source and citation

Fan, Y. (2026). Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval (Version 1.0.0), Chapter 7: Cook Ding and the Phaedrus: Two Ancient Arguments for Cutting at the Natural Joints. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-261 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026261,
  author    = {Fan, Yafu},
  title     = {Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book