Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval: chapter answers
Building a knowledge base means fixing three numbers: chunk length, overlap between neighbouring chunks, and the blend weight between vector and term-frequency retrieval. Against the folk wisdom that smaller is more precise and one blend suffices, the commissioning claim says recall traces an inverted U against chunk length, overlap has diminishing returns, and the best blend depends on the query type. Implementing BM25 and a concept-level bag-of-words cosine in standard-library programs on fixed-seed synthetic corpora, the book corrects the blueprint in three places. The cited 2009 survey by…
DOI pending · monograph page goes live after publication · 5DL-B-261 · 中文版
- Chapter 1 · The Inverted U: Chunk Length and Recall under a Reading Budget
With a span of 12 the best chunk length is 48, which is 4.00 times the span; with a span of 24 the best length is still 48, now 2.00 times the span; with a span of 48 it moves to 96, again 2.00 times. With a span of 48,… - Chapter 2 · BM25 from Scratch: Inverse Document Frequency, Saturation and Length Normalisation
Chapter 2 writes BM25 from scratch; its length normalisation idles under equal-length chunks. - Chapter 3 · Bag-of-Words Cosine and Dilution in Long Chunks: Which Retriever Recognises Which Query
The wrong chunks are more instructive. At a chunk length of 12, the strongest wrong chunk has a mean cosine of 0.584, above the 0.573 of the gold chunk, so the margin is negative. A gold chunk of 12 tokens holds only ha… - Chapter 4 · Overlap: Exact Odds of a Boundary Cut and the Knee of Diminishing Returns
With chunks of 32, recall rises from 0.584 without overlap to 0.970 at an overlap of one half, in line with the saturation point predicted by p41. At an overlap of three quarters the number of chunks grows from 2400 to… - Chapter 5 · The Blend: How the Best Weight Moves with the Query Mix
Chapter 5 sweeps the blend weight: the best value is 0.0 for exact-ID queries and 1.0 for paraphrase queries, and the illustrative 0.7 / 0.3 baseline regrets 0.098 on an all-paraphrase workload; routing by type at a cla… - Chapter 6 · The Joint Grid and the Winner's Curse: Why the Held-Out Test Cannot Be Skipped
Chapter 6 runs the joint grid and finds an optimism of 0.028 when the champion is picked from only 30 labelled queries. - Chapter 7 · Cook Ding and the Phaedrus: Two Ancient Arguments for Cutting at the Natural Joints
Chapter 7 examines Cook Ding carving the ox in the Zhuangzi and Plato's division at the natural joints in the Phaedrus: cutting by paragraph gives zero overlap and zero breakage, fixed windows of 64 at an overlap of one… - Chapter 8 · The Quarterly Tuning Loop and a Plan for Refutation
Chapter 8 gives a quarterly tuning routine for knowledge-base chunking, the distilled "Chunk-and-Blend Grid Optimizer", a pre-registered test with sample sizes from Connor's formula (the exact McNemar test reaches an ac…
Illustrative model calculation, not a live quote