关于「混合打分几何:BM25 饱和、余弦方向与秩融合」,《生成式搜索引用权重》给出的定量结论是什么?

结论:第二章讲 BM25 饱和、余弦方向与秩融合:加入一个新的真实匹配词所得是重复一次旧词的近三倍。

数据口径:书中参数均为示意;全部计算由书中程序生成。 第五章讲可核查性,并用模拟说明从外部推断特征权重的脆弱:漏掉长度时统计数字的权重被高估 0.3252。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库

k1 = 1.2, b = 0.75, avgdl = 60, idf = 2.0
extradlw(t1)w(t2)totalmarginal
0602.00002.00004.0000-
1612.73721.98654.7236+0.7236
2623.12061.97315.0937+0.3700
3633.35561.95995.3155+0.2218
5653.62891.93415.5629+0.1237
8683.83721.89665.7338+0.0569
12723.97771.84875.8265+0.0232
20804.10671.76005.8667+0.0050
401004.21501.57145.7864-0.0040
ceiling of w(t1) as tf grows: idf (k1 + 1) = 4.4000
one new matching term t3 : total 5.9594 (+1.9594)
one more repetition of t1: total 4.7236 (+0.7236)

核心公式

\[w = \mathrm{idf} \cdot \frac{\mathrm{tf}\,(k_1+1)}{\mathrm{tf} + k_1\left(1-b+b\,\frac{dl}{avgdl}\right)}.\]
查看完整验算程序
# Chapter 2. p21: BM25 saturation, the arithmetic behind stuffing.
# Okapi BM25 term weight (Robertson and Zaragoza 2009):
#   w = idf * tf (k1 + 1) / (tf + k1 (1 - b + b dl / avgdl))
# A 60-token passage answers a two-term query (t1, t2) once each. We
# repeat t1 r more times (each repetition adds one token) and compare
# with adding one new matching term t3 of the same idf instead.
# k1 = 1.2, b = 0.75 are common defaults; idf = 2.0 is illustrative.
import math

K1, B, AVGDL, IDF = 1.2, 0.75, 60.0, 2.0


def w(tf, dl):
    if tf == 0:
        return 0.0
    return IDF * tf * (K1 + 1) / (tf + K1 * (1 - B + B * dl / AVGDL))


print("k1 = %.1f, b = %.2f, avgdl = %.0f, idf = %.1f" % (K1, B, AVGDL,
                                                        IDF))
print("%6s %4s %8s %8s %8s %9s" % ("extra", "dl", "w(t1)", "w(t2)",
                                   "total", "marginal"))
prev = None
for r in (0, 1, 2, 3, 5, 8, 12, 20, 40):
    dl = 60 + r
    total = math.fsum([w(1 + r, dl), w(1, dl)])
    mg = "-" if prev is None else "%+.4f" % (total - prev[1])
    if prev is not None and r - prev[0] > 1:
        mg = "%+.4f" % ((total - prev[1]) / (r - prev[0]))
    print("%6d %4d %8.4f %8.4f %8.4f %9s" % (r, dl, w(1 + r, dl),
                                             w(1, dl), total, mg))
    prev = (r, total)
cap = IDF * (K1 + 1)
print("ceiling of w(t1) as tf grows: idf (k1 + 1) = %.4f" % cap)
base = w(1, 60) * 2
new = w(1, 61) * 3
rep = math.fsum([w(2, 61), w(1, 61)])
print("one new matching term t3 : total %.4f (+%.4f)" % (new,
                                                        new - base))
print("one more repetition of t1: total %.4f (+%.4f)" % (rep,
                                                        rep - base))

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《生成式搜索引用权重几何学、问答引擎检索重排特征与高被引双语直答页段落拓扑母典》第 2 章「混合打分几何:BM25 饱和、余弦方向与秩融合」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-260 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026260,
  author    = {Fan, Yafu},
  title     = {The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论