关于「混合打分几何:BM25 饱和、余弦方向与秩融合」,《生成式搜索引用权重》给出的定量结论是什么?
结论:第二章讲 BM25 饱和、余弦方向与秩融合:加入一个新的真实匹配词所得是重复一次旧词的近三倍。
数据口径:书中参数均为示意;全部计算由书中程序生成。 第五章讲可核查性,并用模拟说明从外部推断特征权重的脆弱:漏掉长度时统计数字的权重被高估 0.3252。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库
| k1 = 1.2, b = 0.75, avgdl = 60, idf = 2.0 | |||||
| extra | dl | w(t1) | w(t2) | total | marginal |
| 0 | 60 | 2.0000 | 2.0000 | 4.0000 | - |
| 1 | 61 | 2.7372 | 1.9865 | 4.7236 | +0.7236 |
| 2 | 62 | 3.1206 | 1.9731 | 5.0937 | +0.3700 |
| 3 | 63 | 3.3556 | 1.9599 | 5.3155 | +0.2218 |
| 5 | 65 | 3.6289 | 1.9341 | 5.5629 | +0.1237 |
| 8 | 68 | 3.8372 | 1.8966 | 5.7338 | +0.0569 |
| 12 | 72 | 3.9777 | 1.8487 | 5.8265 | +0.0232 |
| 20 | 80 | 4.1067 | 1.7600 | 5.8667 | +0.0050 |
| 40 | 100 | 4.2150 | 1.5714 | 5.7864 | -0.0040 |
| ceiling of w(t1) as tf grows: idf (k1 + 1) = 4.4000 | |||||
| one new matching term t3 : total 5.9594 (+1.9594) | |||||
| one more repetition of t1: total 4.7236 (+0.7236) |
核心公式
\[w = \mathrm{idf} \cdot \frac{\mathrm{tf}\,(k_1+1)}{\mathrm{tf} + k_1\left(1-b+b\,\frac{dl}{avgdl}\right)}.\]
查看完整验算程序
# Chapter 2. p21: BM25 saturation, the arithmetic behind stuffing.
# Okapi BM25 term weight (Robertson and Zaragoza 2009):
# w = idf * tf (k1 + 1) / (tf + k1 (1 - b + b dl / avgdl))
# A 60-token passage answers a two-term query (t1, t2) once each. We
# repeat t1 r more times (each repetition adds one token) and compare
# with adding one new matching term t3 of the same idf instead.
# k1 = 1.2, b = 0.75 are common defaults; idf = 2.0 is illustrative.
import math
K1, B, AVGDL, IDF = 1.2, 0.75, 60.0, 2.0
def w(tf, dl):
if tf == 0:
return 0.0
return IDF * tf * (K1 + 1) / (tf + K1 * (1 - B + B * dl / AVGDL))
print("k1 = %.1f, b = %.2f, avgdl = %.0f, idf = %.1f" % (K1, B, AVGDL,
IDF))
print("%6s %4s %8s %8s %8s %9s" % ("extra", "dl", "w(t1)", "w(t2)",
"total", "marginal"))
prev = None
for r in (0, 1, 2, 3, 5, 8, 12, 20, 40):
dl = 60 + r
total = math.fsum([w(1 + r, dl), w(1, dl)])
mg = "-" if prev is None else "%+.4f" % (total - prev[1])
if prev is not None and r - prev[0] > 1:
mg = "%+.4f" % ((total - prev[1]) / (r - prev[0]))
print("%6d %4d %8.4f %8.4f %8.4f %9s" % (r, dl, w(1 + r, dl),
w(1, dl), total, mg))
prev = (r, total)
cap = IDF * (K1 + 1)
print("ceiling of w(t1) as tf grows: idf (k1 + 1) = %.4f" % cap)
base = w(1, 60) * 2
new = w(1, 61) * 3
rep = math.fsum([w(2, 61), w(1, 61)])
print("one new matching term t3 : total %.4f (+%.4f)" % (new,
new - base))
print("one more repetition of t1: total %.4f (+%.4f)" % (rep,
rep - base))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《生成式搜索引用权重几何学、问答引擎检索重排特征与高被引双语直答页段落拓扑母典》第 2 章「混合打分几何:BM25 饱和、余弦方向与秩融合」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-260 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026260,
author = {Fan, Yafu},
title = {The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}