What does Chunking and Blending conclude, in numbers, about “BM25 from Scratch: Inverse Document Frequency, Saturation and Length Normalisation”?

Answer: Chapter 2 writes BM25 from scratch; its length normalisation idles under equal-length chunks.

Scope of the numbers: Implementing BM25 and a concept-level bag-of-words cosine in standard-library programs on fixed-seed synthetic corpora, the book corrects the blueprint in three places. Parameters are illustrative and all numbers come from the book's programs, which read and write no file and connect to no live knowledge base.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 25

avg length 8.25; idf refund 0.357, receipt 1.204
chunk 0 len8 BM25 1.714
chunk 1 len9 BM25 0.000
chunk 2 len6 BM25 0.634
chunk 3 len 10 BM25 0.328
tf1 2 4 8 limit (b = 0, k1 varies)
k10.5 1.000 1.200 1.333 1.412 1.500
k11.2 1.000 1.375 1.692 1.913 2.200
k12.0 1.000 1.500 2.000 2.400 3.000
L/avg0.5 1.0 2.0 4.0 (tf = 2, k1 = 1.2)
b0.00 1.375 1.375 1.375 1.375
b0.75 1.600 1.375 1.073 0.746
b1.00 1.692 1.375 1.000 0.647
Show the full program
# Chapter 2. p21: BM25 by hand: idf, term-frequency saturation, length.
# Model: Robertson-Sparck Jones idf with the +1 inside the log (never
# negative), term weight tf (k1 + 1) / (tf + k1 (1 - b + b L / avgL)).
# Four toy chunks; k1 = 1.2, b = 0.75 are the textbook defaults.
import math

CHUNKS = [
    "refund window is thirty days refund needs receipt".split(),
    "shipping takes five days shipping is free over fifty".split(),
    "refund refund refund refund policy text".split(),
    "store hours and refund desk location and parking and lifts".split(),
]
K1, B = 1.2, 0.75


def idf(n, df):
    return math.log(1 + (n - df + 0.5) / (df + 0.5))


def term(tf, ln, avg, k1=K1, b=B):
    return tf * (k1 + 1) / (tf + k1 * (1 - b + b * ln / avg))


n = len(CHUNKS)
avg = math.fsum(len(c) for c in CHUNKS) / n
q = ["refund", "receipt"]
print("avg length %.2f; idf refund %.3f, receipt %.3f" % (
    avg, idf(n, 3), idf(n, 1)))
for i, c in enumerate(CHUNKS):
    parts = [idf(n, 3 if t == "refund" else 1) * term(c.count(t),
             len(c), avg) for t in q if c.count(t)]
    print("chunk %d len %2d  BM25 %.3f" % (i, len(c), math.fsum(parts)))
print("tf      1      2      4      8     limit  (b = 0, k1 varies)")
for k1 in (0.5, 1.2, 2.0):
    row = [term(t, 1, 1, k1, 0.0) for t in (1, 2, 4, 8)]
    print("k1=%.1f " % k1 + " ".join("%6.3f" % x for x in row)
          + "  %6.3f" % (k1 + 1))
print("L/avg   0.5    1.0    2.0    4.0   (tf = 2, k1 = 1.2)")
for b in (0.0, 0.75, 1.0):
    row = [term(2, r, 1.0, K1, b) for r in (0.5, 1.0, 2.0, 4.0)]
    print("b=%.2f " % b + " ".join("%6.3f" % x for x in row))

Single source and citation

Fan, Y. (2026). Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval (Version 1.0.0), Chapter 2: BM25 from Scratch: Inverse Document Frequency, Saturation and Length Normalisation. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-261 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026261,
  author    = {Fan, Yafu},
  title     = {Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book