What does Chunking and Blending conclude, in numbers, about “BM25 from Scratch: Inverse Document Frequency, Saturation and Length Normalisation”?
Answer: Chapter 2 writes BM25 from scratch; its length normalisation idles under equal-length chunks.
Scope of the numbers: Implementing BM25 and a concept-level bag-of-words cosine in standard-library programs on fixed-seed synthetic corpora, the book corrects the blueprint in three places. Parameters are illustrative and all numbers come from the book's programs, which read and write no file and connect to no live knowledge base.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 25
| avg length 8.25; idf refund 0.357, receipt 1.204 | |
| chunk 0 len | 8 BM25 1.714 |
| chunk 1 len | 9 BM25 0.000 |
| chunk 2 len | 6 BM25 0.634 |
| chunk 3 len 10 BM25 0.328 | |
| tf | 1 2 4 8 limit (b = 0, k1 varies) |
| k1 | 0.5 1.000 1.200 1.333 1.412 1.500 |
| k1 | 1.2 1.000 1.375 1.692 1.913 2.200 |
| k1 | 2.0 1.000 1.500 2.000 2.400 3.000 |
| L/avg | 0.5 1.0 2.0 4.0 (tf = 2, k1 = 1.2) |
| b | 0.00 1.375 1.375 1.375 1.375 |
| b | 0.75 1.600 1.375 1.073 0.746 |
| b | 1.00 1.692 1.375 1.000 0.647 |
Show the full program
# Chapter 2. p21: BM25 by hand: idf, term-frequency saturation, length.
# Model: Robertson-Sparck Jones idf with the +1 inside the log (never
# negative), term weight tf (k1 + 1) / (tf + k1 (1 - b + b L / avgL)).
# Four toy chunks; k1 = 1.2, b = 0.75 are the textbook defaults.
import math
CHUNKS = [
"refund window is thirty days refund needs receipt".split(),
"shipping takes five days shipping is free over fifty".split(),
"refund refund refund refund policy text".split(),
"store hours and refund desk location and parking and lifts".split(),
]
K1, B = 1.2, 0.75
def idf(n, df):
return math.log(1 + (n - df + 0.5) / (df + 0.5))
def term(tf, ln, avg, k1=K1, b=B):
return tf * (k1 + 1) / (tf + k1 * (1 - b + b * ln / avg))
n = len(CHUNKS)
avg = math.fsum(len(c) for c in CHUNKS) / n
q = ["refund", "receipt"]
print("avg length %.2f; idf refund %.3f, receipt %.3f" % (
avg, idf(n, 3), idf(n, 1)))
for i, c in enumerate(CHUNKS):
parts = [idf(n, 3 if t == "refund" else 1) * term(c.count(t),
len(c), avg) for t in q if c.count(t)]
print("chunk %d len %2d BM25 %.3f" % (i, len(c), math.fsum(parts)))
print("tf 1 2 4 8 limit (b = 0, k1 varies)")
for k1 in (0.5, 1.2, 2.0):
row = [term(t, 1, 1, k1, 0.0) for t in (1, 2, 4, 8)]
print("k1=%.1f " % k1 + " ".join("%6.3f" % x for x in row)
+ " %6.3f" % (k1 + 1))
print("L/avg 0.5 1.0 2.0 4.0 (tf = 2, k1 = 1.2)")
for b in (0.0, 0.75, 1.0):
row = [term(2, r, 1.0, K1, b) for r in (0.5, 1.0, 2.0, 4.0)]
print("b=%.2f " % b + " ".join("%6.3f" % x for x in row))Single source and citation
Fan, Y. (2026). Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval (Version 1.0.0), Chapter 2: BM25 from Scratch: Inverse Document Frequency, Saturation and Length Normalisation. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-261 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026261,
author = {Fan, Yafu},
title = {Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Inverted U: Chunk Length and Recall under a Reading Budget
- Chapter 3 · Bag-of-Words Cosine and Dilution in Long Chunks: Which Retriever Recognises Which Query
- Chapter 4 · Overlap: Exact Odds of a Boundary Cut and the Knee of Diminishing Returns
- Chapter 5 · The Blend: How the Best Weight Moves with the Query Mix
- Chapter 6 · The Joint Grid and the Winner's Curse: Why the Held-Out Test Cannot Be Skipped
- Chapter 7 · Cook Ding and the Phaedrus: Two Ancient Arguments for Cutting at the Natural Joints
- Chapter 8 · The Quarterly Tuning Loop and a Plan for Refutation