关于「BM25 与长度归一化:堆砌、填充与直答句的词频账」,《知识库权威传播与直答片段》给出的定量结论是什么?
结论:第三章用标准库实现 BM25,说明堆砌很快到顶、填充按长度打折:十二个词的直答句填充到二百四十个词只剩原分数的 0.302。
数据口径:书中参数为示意;全部计算由书中程序生成;本书不是法律、理财或经营方面的个人化建议,也不对任何系统中的引用排名作出承诺。 第一章用幂迭代与随机冲浪者计算 PageRank 与 HITS:在阻尼系数 0.85 的示意知识库里,母典页拿到 0.3240,是均匀份额的 2.59 倍。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 25
| query: pagerank damping factor | avgdl 22.9 | ||||
| snippet | length | BM25 b=0.75 | rank | BM25 b=0 | rank |
| stuffed | 11 | 4.371 | 1 | 3.827 | 1 |
| def | 21 | 2.672 | 2 | 2.582 | 2 |
| faq | 16 | 2.154 | 3 | 1.889 | 3 |
| notes | 10 | 0.900 | 4 | 0.693 | 5 |
| long | 97 | 0.752 | 5 | 1.173 | 4 |
| bm25 | 8 | 0.000 | 6 | 0.000 | 6 |
| hits | 9 | 0.000 | 7 | 0.000 | 7 |
| rag | 11 | 0.000 | 8 | 0.000 | 8 |
| stuffed leads def by 1.700 at b=0.75 and 1.245 at b=0 |
查看完整验算程序
# Chapter 3. p31: BM25 on an illustrative eight-snippet corpus.
# Score = sum over query terms of idf * tf (k1 + 1) /
# (tf + k1 (1 - b + b dl / avgdl)),
# idf = ln(1 + (N - n + 0.5) / (n + 0.5)), the non-negative variant.
# k1 = 1.2 and b = 0.75 are common defaults discussed by Robertson and
# Zaragoza (2009). Snippets and query are invented for illustration.
import math
from collections import Counter
FILL = ("our team has written this complete guide so that every reader"
" can learn all about search and links and why they matter for"
" a growing brand in a crowded market").split()
DOCS = {
"def": "pagerank damping factor d is the chance that the surfer"
" follows a link and brin and page used d of 0.85",
"faq": "what is the damping factor short answer the chance of"
" following a link rather than jumping",
"long": " ".join(FILL * 3 + "pagerank matters pagerank helps"
" pagerank grows pagerank".split()),
"stuffed": "pagerank pagerank pagerank damping damping factor"
" factor pagerank best pagerank guide",
"bm25": "bm25 scores terms by frequency and document length",
"hits": "hits computes hubs and authorities for a query subgraph",
"rag": "retrieval augmented generation puts the top k passages"
" into the prompt",
"notes": "notes on link analysis pagerank hits and the random"
" surfer",
}
QUERY = "pagerank damping factor".split()
def bm25(docs, query, k1=1.2, b=0.75):
toks = {k: v.split() for k, v in docs.items()}
n_docs = len(toks)
avgdl = math.fsum(len(t) for t in toks.values()) / n_docs
df = Counter()
for t in toks.values():
df.update(set(t))
out = {}
for k, t in toks.items():
tf = Counter(t)
s = []
for q in query:
idf = math.log(1 + (n_docs - df[q] + 0.5) / (df[q] + 0.5))
norm = k1 * (1 - b + b * len(t) / avgdl)
s.append(idf * tf[q] * (k1 + 1) / (tf[q] + norm))
out[k] = math.fsum(s)
return out, avgdl
if __name__ == "__main__":
full, avgdl = bm25(DOCS, QUERY)
flat, _ = bm25(DOCS, QUERY, b=0.0)
print("query: " + " ".join(QUERY) + " avgdl {:.1f}".format(avgdl))
print("snippet length BM25 b=0.75 rank BM25 b=0 rank")
r1 = sorted(full, key=lambda k: (-full[k], k))
r0 = sorted(flat, key=lambda k: (-flat[k], k))
for k in r1:
print("{:<9}{:7d}{:13.3f}{:6d}{:10.3f}{:6d}".format(
k, len(DOCS[k].split()), full[k], r1.index(k) + 1, flat[k],
r0.index(k) + 1))
print("stuffed leads def by {:.3f} at b=0.75 and {:.3f} at b=0".format(
full["stuffed"] - full["def"], flat["stuffed"] - flat["def"]))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《机器可读知识库超链接权威传播、生成式问答检索截断博弈与高信噪比摘要提取》第 3 章「BM25 与长度归一化:堆砌、填充与直答句的词频账」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-176 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026176,
author = {Fan, Yafu},
title = {Hyperlink Authority Propagation in Machine-Readable Knowledge Bases, Retrieval-Truncation Games in Generative Answering and High-SNR Snippet Extraction},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}