关于「从头写 BM25:逆文档频率、词频饱和与长度归一」,《分块与混合配比》给出的定量结论是什么?
结论:第二章从头写 BM25,说明长度归一在等长分块里几乎闲置。
数据口径:书中参数为示意;全部计算由书中程序生成;程序不读写任何文件,不连接任何线上知识库;本书是非个人化经营建议,任何线上知识库参数的改动须经离线评测并由主事者人工复核放行。 第五章扫描混合权重:编号型查询最优 0.0,改写型最优 1.0,示意基线 0.7 / 0.3 在全改写型时遗憾 0.098;按类型分流在判别出错率为 0.10 时留出召回 0.928,已低于调好的全局权重 0.938。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 25
| avg length 8.25; idf refund 0.357, receipt 1.204 | |
| chunk 0 len | 8 BM25 1.714 |
| chunk 1 len | 9 BM25 0.000 |
| chunk 2 len | 6 BM25 0.634 |
| chunk 3 len 10 BM25 0.328 | |
| tf | 1 2 4 8 limit (b = 0, k1 varies) |
| k1 | 0.5 1.000 1.200 1.333 1.412 1.500 |
| k1 | 1.2 1.000 1.375 1.692 1.913 2.200 |
| k1 | 2.0 1.000 1.500 2.000 2.400 3.000 |
| L/avg | 0.5 1.0 2.0 4.0 (tf = 2, k1 = 1.2) |
| b | 0.00 1.375 1.375 1.375 1.375 |
| b | 0.75 1.600 1.375 1.073 0.746 |
| b | 1.00 1.692 1.375 1.000 0.647 |
查看完整验算程序
# Chapter 2. p21: BM25 by hand: idf, term-frequency saturation, length.
# Model: Robertson-Sparck Jones idf with the +1 inside the log (never
# negative), term weight tf (k1 + 1) / (tf + k1 (1 - b + b L / avgL)).
# Four toy chunks; k1 = 1.2, b = 0.75 are the textbook defaults.
import math
CHUNKS = [
"refund window is thirty days refund needs receipt".split(),
"shipping takes five days shipping is free over fifty".split(),
"refund refund refund refund policy text".split(),
"store hours and refund desk location and parking and lifts".split(),
]
K1, B = 1.2, 0.75
def idf(n, df):
return math.log(1 + (n - df + 0.5) / (df + 0.5))
def term(tf, ln, avg, k1=K1, b=B):
return tf * (k1 + 1) / (tf + k1 * (1 - b + b * ln / avg))
n = len(CHUNKS)
avg = math.fsum(len(c) for c in CHUNKS) / n
q = ["refund", "receipt"]
print("avg length %.2f; idf refund %.3f, receipt %.3f" % (
avg, idf(n, 3), idf(n, 1)))
for i, c in enumerate(CHUNKS):
parts = [idf(n, 3 if t == "refund" else 1) * term(c.count(t),
len(c), avg) for t in q if c.count(t)]
print("chunk %d len %2d BM25 %.3f" % (i, len(c), math.fsum(parts)))
print("tf 1 2 4 8 limit (b = 0, k1 varies)")
for k1 in (0.5, 1.2, 2.0):
row = [term(t, 1, 1, k1, 0.0) for t in (1, 2, 4, 8)]
print("k1=%.1f " % k1 + " ".join("%6.3f" % x for x in row)
+ " %6.3f" % (k1 + 1))
print("L/avg 0.5 1.0 2.0 4.0 (tf = 2, k1 = 1.2)")
for b in (0.0, 0.75, 1.0):
row = [term(2, r, 1.0, K1, b) for r in (0.5, 1.0, 2.0, 4.0)]
print("b=%.2f " % b + " ".join("%6.3f" % x for x in row))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《知识库分块边界网格寻优、重叠率与向量词频混合检索配比母典》第 2 章「从头写 BM25:逆文档频率、词频饱和与长度归一」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-261 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026261,
author = {Fan, Yafu},
title = {Grid Search over Knowledge-Base Chunk Boundaries: Overlap Ratios and the Hybrid Weighting of Vector and Term-Frequency Retrieval},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}