What does Chunking and Blending conclude, in numbers, about “Cook Ding and the Phaedrus: Two Ancient Arguments for Cutting at the Natural Joints”?
Answer: Chapter 7 examines Cook Ding carving the ox in the Zhuangzi and Plato's division at the natural joints in the Phaedrus: cutting by paragraph gives zero overlap and zero breakage, fixed windows of 64 at an overlap of one half still break 0.401 of the answers, and misread joints quickly erase this advantage.
Scope of the numbers: Implementing BM25 and a concept-level bag-of-words cosine in standard-library programs on fixed-seed synthetic corpora, the book corrects the blueprint in three places. Parameters are illustrative and all numbers come from the book's programs, which read and write no file and connect to no live knowledge base.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.401, 64
| chunker | mean-len | max-len | index | broken |
|---|---|---|---|---|
| joints (paragraph) | 64.1 | 112 | 1.00 | 0.000 |
| fixed w= 64 o=0.00 | 64.0 | 64 | 1.00 | 0.633 |
| fixed w= 64 o=0.25 | 64.0 | 64 | 1.33 | 0.515 |
| fixed w= 64 o=0.50 | 64.0 | 64 | 2.00 | 0.401 |
| fixed w=112 o=0.00 | 112.0 | 112 | 1.00 | 0.341 |
| fixed w=112 o=0.50 | 112.0 | 112 | 2.00 | 0.018 |
Show the full program
# Chapter 7. p71: cutting at the joints versus hacking at fixed windows.
# Model: a stream of 20000 paragraphs, lengths uniform on 16..112 tokens
# (seed 20261113); each paragraph is one answer. A paragraph is broken
# when no single chunk holds at least 80 % of it. Joint chunking cuts
# exactly at paragraph ends; fixed windows ignore them. Illustrative.
import math
import random
rng = random.Random(20261113)
lens = [rng.randint(16, 112) for _ in range(20000)]
starts, pos = [], 0
for ln in lens:
starts.append(pos)
pos += ln
total = pos
def broken(w, s):
bad = 0
for a, ln in zip(starts, lens):
j0 = max(0, (a - w) // s)
best = 0
for j in range(j0, a // s + 2):
lo = j * s
best = max(best, min(lo + w, a + ln) - max(lo, a))
bad += best < 0.8 * ln
return bad / len(lens)
print("chunker mean-len max-len index broken")
print("joints (paragraph) %8.1f %7d %5.2f %6.3f" % (
math.fsum(lens) / len(lens), max(lens), 1.0, 0.0))
for w, o in ((64, 0.0), (64, 0.25), (64, 0.5), (112, 0.0), (112, 0.5)):
s = int(round(w * (1 - o)))
print("fixed w=%3d o=%.2f %8.1f %7d %5.2f %6.3f" % (
w, o, w, w, w / s, broken(w, s)))Single source and citation
Fan, Y. (2026). Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval (Version 1.0.0), Chapter 7: Cook Ding and the Phaedrus: Two Ancient Arguments for Cutting at the Natural Joints. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-261 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026261,
author = {Fan, Yafu},
title = {Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Inverted U: Chunk Length and Recall under a Reading Budget
- Chapter 2 · BM25 from Scratch: Inverse Document Frequency, Saturation and Length Normalisation
- Chapter 3 · Bag-of-Words Cosine and Dilution in Long Chunks: Which Retriever Recognises Which Query
- Chapter 4 · Overlap: Exact Odds of a Boundary Cut and the Knee of Diminishing Returns
- Chapter 5 · The Blend: How the Best Weight Moves with the Query Mix
- Chapter 6 · The Joint Grid and the Winner's Curse: Why the Held-Out Test Cannot Be Skipped
- Chapter 8 · The Quarterly Tuning Loop and a Plan for Refutation