关于「BM25 与长度归一化:堆砌、填充与直答句的词频账」,《知识库权威传播与直答片段》给出的定量结论是什么?

结论:第三章用标准库实现 BM25,说明堆砌很快到顶、填充按长度打折:十二个词的直答句填充到二百四十个词只剩原分数的 0.302。

数据口径:书中参数为示意;全部计算由书中程序生成;本书不是法律、理财或经营方面的个人化建议,也不对任何系统中的引用排名作出承诺。 第一章用幂迭代与随机冲浪者计算 PageRank 与 HITS:在阻尼系数 0.85 的示意知识库里,母典页拿到 0.3240,是均匀份额的 2.59 倍。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 25

query: pagerank damping factoravgdl 22.9
snippetlengthBM25 b=0.75rankBM25 b=0rank
stuffed114.37113.8271
def212.67222.5822
faq162.15431.8893
notes100.90040.6935
long970.75251.1734
bm2580.00060.0006
hits90.00070.0007
rag110.00080.0008
stuffed leads def by 1.700 at b=0.75 and 1.245 at b=0
查看完整验算程序
# Chapter 3. p31: BM25 on an illustrative eight-snippet corpus.
# Score = sum over query terms of idf * tf (k1 + 1) /
#         (tf + k1 (1 - b + b dl / avgdl)),
# idf = ln(1 + (N - n + 0.5) / (n + 0.5)), the non-negative variant.
# k1 = 1.2 and b = 0.75 are common defaults discussed by Robertson and
# Zaragoza (2009). Snippets and query are invented for illustration.
import math
from collections import Counter

FILL = ("our team has written this complete guide so that every reader"
        " can learn all about search and links and why they matter for"
        " a growing brand in a crowded market").split()
DOCS = {
    "def": "pagerank damping factor d is the chance that the surfer"
           " follows a link and brin and page used d of 0.85",
    "faq": "what is the damping factor short answer the chance of"
           " following a link rather than jumping",
    "long": " ".join(FILL * 3 + "pagerank matters pagerank helps"
                     " pagerank grows pagerank".split()),
    "stuffed": "pagerank pagerank pagerank damping damping factor"
               " factor pagerank best pagerank guide",
    "bm25": "bm25 scores terms by frequency and document length",
    "hits": "hits computes hubs and authorities for a query subgraph",
    "rag": "retrieval augmented generation puts the top k passages"
           " into the prompt",
    "notes": "notes on link analysis pagerank hits and the random"
             " surfer",
}
QUERY = "pagerank damping factor".split()


def bm25(docs, query, k1=1.2, b=0.75):
    toks = {k: v.split() for k, v in docs.items()}
    n_docs = len(toks)
    avgdl = math.fsum(len(t) for t in toks.values()) / n_docs
    df = Counter()
    for t in toks.values():
        df.update(set(t))
    out = {}
    for k, t in toks.items():
        tf = Counter(t)
        s = []
        for q in query:
            idf = math.log(1 + (n_docs - df[q] + 0.5) / (df[q] + 0.5))
            norm = k1 * (1 - b + b * len(t) / avgdl)
            s.append(idf * tf[q] * (k1 + 1) / (tf[q] + norm))
        out[k] = math.fsum(s)
    return out, avgdl


if __name__ == "__main__":
    full, avgdl = bm25(DOCS, QUERY)
    flat, _ = bm25(DOCS, QUERY, b=0.0)
    print("query: " + " ".join(QUERY) + "   avgdl {:.1f}".format(avgdl))
    print("snippet   length  BM25 b=0.75  rank  BM25 b=0  rank")
    r1 = sorted(full, key=lambda k: (-full[k], k))
    r0 = sorted(flat, key=lambda k: (-flat[k], k))
    for k in r1:
        print("{:<9}{:7d}{:13.3f}{:6d}{:10.3f}{:6d}".format(
            k, len(DOCS[k].split()), full[k], r1.index(k) + 1, flat[k],
            r0.index(k) + 1))
    print("stuffed leads def by {:.3f} at b=0.75 and {:.3f} at b=0".format(
        full["stuffed"] - full["def"], flat["stuffed"] - flat["def"]))

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《机器可读知识库超链接权威传播、生成式问答检索截断博弈与高信噪比摘要提取》第 3 章「BM25 与长度归一化:堆砌、填充与直答句的词频账」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-176 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026176,
  author    = {Fan, Yafu},
  title     = {Hyperlink Authority Propagation in Machine-Readable Knowledge Bases, Retrieval-Truncation Games in Generative Answering and High-SNR Snippet Extraction},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论