5D Library › 检索增强生成几何、倒数秩融合与多智能体误差级联 · English

第二章 词法与稠密检索:各自的盲区与倒数秩融合的平滑常数

两种检索,两种失败

稀疏的词法检索(以 BM25 为代表)按查询词与文本的精确重合打分 (Robertson and Zaragoza 2009);稠密检索按语义向量的相似度打分 (Karpukhin et al. 2020)。前者认得字面、认不得意思,后者认得意思、认不得字面。在含有错误代码、零件编号、版本号或罕见术语的查询上,稠密检索有一个结构性的弱点:分词器把这些字符串切成常见的子词,不同编号的向量几乎相同,内积无法分辨它们。本章先用一个合成语料把两种失败并排展示,再看把两者合起来的最常用方法——倒数秩融合(RRF)——在什么条件下有效,以及它的平滑常数 kk 应当取多少。

盲区并排

推导

# Chapter 2. p04: Lexical (BM25) and dense retrieval fail on different
# queries.
# Synthetic corpus: 2,000 chunks in 50 topics. Each chunk has 20 topic
# words, 3 detail words of its own, and a unique error code such as
# E1043. Two query types, 200 each:
#   paraphrase  3 topic synonyms + 3 paraphrased detail words - no word
#               in common with the gold chunk
#   exact code  "what does code E1043 mean" - the code is the only clue
# BM25 (k1 = 1.2, b = 0.75) scores exact token overlap. The dense model
# (d = 64) maps a word and its synonym/paraphrase to nearby vectors, but
# like a subword tokenizer sees every code as almost the same token: a
# shared "code" vector plus only 0.15 of a code-specific part. A chunk
# or query vector is the normalised mean of its word vectors.
# Hybrid: reciprocal rank fusion, score = sum 1 / (60 + rank), done two
# ways: RRF-all fuses complete rankings (ties broken at random); RRF-100
# fuses each ranker's top-100 list only, and BM25 lists only chunks with
# a positive score - the usual implementation.
import math
import random

D, TOPICS, DOCS = 64, 50, 2000

def unit(rng):
    v = [rng.gauss(0, 1) for _ in range(D)]
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]

def mix(*parts):
    v = [0.0] * D
    for w, vec in parts:
        v = [a + w * b for a, b in zip(v, vec)]
    return v

def norm(v):
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]

class Corpus:
    def __init__(self, rng):
        self.rng = rng
        self.topic = [unit(rng) for _ in range(TOPICS)]
        self.code_dir = unit(rng)
        self.vec = {}
        self.docs = []
        for i in range(DOCS):
            t = i % TOPICS
            words = ["t{}w{}".format(t, rng.randrange(40)) for _ in range(20)]
            words += ["d{}x{}".format(i, k) for k in range(3)]
            words += ["E{}".format(1000 + i)]
            self.docs.append((t, words))
        self.df = {}
        for _, ws in self.docs:
            for w in set(ws):
                self.df[w] = self.df.get(w, 0) + 1
        self.avg = sum(len(ws) for _, ws in self.docs) / DOCS
        self.emb = [self.embed(ws) for _, ws in self.docs]

    def wvec(self, w):
        if w in self.vec:
            return self.vec[w]
        r = self.rng
        if w.startswith("t"):                       # topic word or synonym
            t = int(w[1:].replace("s", "w").split("w")[0])
            v = mix((1.0, self.topic[t]), (0.5, unit(r)))
        elif w.startswith("d"):                     # detail word/paraphrase
            base = w.replace("p", "x")
            if base not in self.vec:
                self.vec[base] = unit(r)
            v = mix((1.0, self.vec[base]), (0.3, unit(r)))
            if w == base:
                return self.vec[base]
        elif w.startswith("E"):                     # code: almost one token
            v = mix((1.0, self.code_dir), (0.15, unit(r)))
        else:
            v = unit(r)
        self.vec[w] = v
        return v

    def embed(self, words):
        return norm(mix(*[(1.0, self.wvec(w)) for w in words]))

    def bm25(self, q):
        out = []
        for i, (_, ws) in enumerate(self.docs):
            s = 0.0
            for w in set(q):
                f = ws.count(w)
                if f:
                    n = self.df[w]
                    idf = math.log(1 + (DOCS - n + 0.5) / (n + 0.5))
                    s += idf * f * 2.2 / (f + 1.2 * (0.25 + 0.75 * len(ws)
                                                     / self.avg))
            out.append(s)
        return out

    def dense(self, q):
        qv = self.embed(q)
        return [sum(a * b for a, b in zip(qv, e)) for e in self.emb]

def ranks(scores, rng):
    order = sorted(range(len(scores)),
                   key=lambda i: (-scores[i], rng.random()))
    r = [0] * len(scores)
    for pos, i in enumerate(order):
        r[i] = pos + 1
    return r

if __name__ == "__main__":
    rng = random.Random(61004)
    c = Corpus(rng)
    res = {}
    for qtype in ("paraphrase", "exact code"):
        hits = {"BM25": [0, 0], "dense": [0, 0], "RRF-all": [0, 0],
                "RRF-100": [0, 0]}
        for _ in range(200):
            g = rng.randrange(DOCS)
            t = c.docs[g][0]
            if qtype == "paraphrase":
                q = ["t{}s{}".format(t, rng.randrange(40)) for _ in range(3)]
                q += ["d{}p{}".format(g, k) for k in range(3)]
            else:
                q = ["what", "does", "code", "E{}".format(1000 + g), "mean"]
            sb = c.bm25(q)
            rb = ranks(sb, rng)
            rd = ranks(c.dense(q), rng)
            fused = [1 / (60 + a) + 1 / (60 + b) for a, b in zip(rb, rd)]
            rf = ranks(fused, rng)
            f100 = [(1 / (60 + a) if a <= 100 and s > 0 else 0.0)
                    + (1 / (60 + b) if b <= 100 else 0.0)
                    for a, b, s in zip(rb, rd, sb)]
            r100 = ranks(f100, rng)
            for name, r in (("BM25", rb), ("dense", rd), ("RRF-all", rf),
                            ("RRF-100", r100)):
                hits[name][0] += r[g] == 1
                hits[name][1] += r[g] <= 10
        res[qtype] = hits
    print("   query type     ranker   recall@1   recall@10")
    for qtype, hits in res.items():
        for name, (h1, h10) in hits.items():
            print("   {:<13}  {:<7}  {:7.3f}    {:7.3f}".format(
                qtype, name, h1 / 200, h10 / 200))
    print("\nReading: BM25 cannot find a chunk that shares no words with")
    print("the query; the dense model cannot tell one code from another,")
    print("because every code is nearly the same vector. Fusion is not")
    print("automatically the best of both: fusing full rankings lets the")
    print("blind ranker's noise outvote the one that knows. Fusing short")
    print("lists, with zero-score chunks left out, repairs paraphrase")
    print("queries completely but leaves exact-code queries a near coin")
    print("toss: the dense top hit ties with the true chunk.")
#    query type     ranker   recall@1   recall@10
#    paraphrase     BM25       0.005      0.005
#    paraphrase     dense      0.980      1.000
#    paraphrase     RRF-all    0.055      0.740
#    paraphrase     RRF-100    0.980      1.000
#    exact code     BM25       1.000      1.000
#    exact code     dense      0.000      0.005
#    exact code     RRF-all    0.055      0.750
#    exact code     RRF-100    0.475      1.000
#
# Reading: BM25 cannot find a chunk that shares no words with
# the query; the dense model cannot tell one code from another,
# because every code is nearly the same vector. Fusion is not
# automatically the best of both: fusing full rankings lets the
# blind ranker's noise outvote the one that knows. Fusing short
# lists, with zero-score chunks left out, repairs paraphrase
# queries completely but leaves exact-code queries a near coin
# toss: the dense top hit ties with the true chunk.

在改写式查询上(与正确块没有一个共同的词),BM25 的 recall@1 只有 0.005,稠密检索是 0.980。在精确代码查询上正好相反:BM25 是 1.000,稠密检索是 0.000——它把每个代码都看成几乎同一个向量,代码之间的差别只占向量的很小一部分,淹没在主题的差异里。这是对「纯向量内积在精确匹配上存在分辨率下界」的直接演示。

融合并不自动得到两者之长。把两份完整排名直接融合(RRF-all),两类查询的 recall@1 都只剩 0.055:对每一类查询,总有一个检索器在乱排,它的噪声与另一个检索器的正确信号获得同样的权重。按通常的做法只融合各自的前 100 名、并去掉 BM25 得分为零的块(RRF-100),改写式查询完全恢复(0.980),但精确代码查询只到 0.475:稠密检索排第一的那个错误块,与 BM25 排第一的正确块,在融合分数上打成平手,接近抛硬币。这个 0.475 来自等权重下的平局退化:正确块只靠 BM25 一路命中(单路第一名),稠密检索排第一的错误块也只靠一路命中,两者的 RRF 分数同为 1/(k+1)1/(k+1),谁排第一只取决于平局的打破方式。recall@10 都是 1.000,所以正确块总在前十名里——问题只在第一名。

注

这对实践意味着什么。 融合两个检索器之前,先按查询类型看各自的盲区。对已知含有标识符的查询(可以用简单的正则判断),给词法检索更高的权重或直接走词法通道,往往比对所有查询一视同仁的融合更好。

平滑常数 k

推导

倒数秩融合。 对每个块,融合分数为 RRF(d)=∑i1k+ri(d),\mathrm{RRF}(d) = \sum_{i} \frac{1}{k + r_i(d)} , ri(d)r_i(d) 是它在第 ii 个检索器中的名次 (Cormack et al. 2009)。kk 小时,第一名与第二名的分数差别很大,融合由各检索器的头部决定;kk 大时,分数随名次变化平缓,融合接近于名次的平均。Cormack 等人在原文中使用 k=60k = 60。

推导

# Chapter 2. p05: How the smoothing constant k in RRF moves head and tail.
# Two rankers score 1,000 chunks. Irrelevant chunks get standard-normal
# scores; the gold chunk gets mu plus noise. The two rankers' noise can
# be correlated (rho), as when both are fooled by the same chunk. Fused
# score = sum over rankers of 1 / (k + rank). Three scenarios, 2,000
# queries each:
#   S1 equal     mu = 2.5 and 2.5, rho = 0
#   S2 unequal   mu = 3.5 and 1.0, rho = 0   (one strong, one weak)
#   S3 shared    mu = 2.5 and 2.5, rho = 0.6
# For comparison, a convex blend of the raw scores, alpha * a + (1 -
# alpha) * b, with the best alpha on a 0.05 grid (it needs comparable
# scores, which real rankers do not give for free).
import math
import random

N, Q = 1000, 2000
KS = (1, 5, 20, 60, 200, 1000)

def scenario(rng, mu_a, mu_b, rho):
    out = []
    for _ in range(Q):
        a, b = [], []
        for _ in range(N):
            z = rng.gauss(0, 1)
            a.append(z)
            b.append(rho * z + math.sqrt(1 - rho * rho) * rng.gauss(0, 1))
        a[0] += mu_a
        b[0] += mu_b
        out.append((a, b))
    return out

def rank_of_gold(score):
    g = score[0]
    return 1 + sum(1 for s in score[1:] if s > g)

def rank_list(score):
    order = sorted(range(len(score)), key=lambda i: -score[i])
    r = [0] * len(score)
    for p, i in enumerate(order):
        r[i] = p + 1
    return r

def metrics(golds):
    n = len(golds)
    return (sum(r == 1 for r in golds) / n, sum(1 / r for r in golds) / n,
            sum(r <= 20 for r in golds) / n)

if __name__ == "__main__":
    rng = random.Random(61005)
    for name, ma, mb, rho in (("S1 equal", 2.5, 2.5, 0.0),
                              ("S2 unequal", 3.5, 1.0, 0.0),
                              ("S3 shared", 2.5, 2.5, 0.6)):
        data = scenario(rng, ma, mb, rho)
        ranked = [(rank_list(a), rank_list(b), a, b) for a, b in data]
        print("{}  (alone: A r@1 {:.3f}, B r@1 {:.3f})".format(
            name, metrics([ra[0] for ra, _, _, _ in ranked])[0],
            metrics([rb[0] for _, rb, _, _ in ranked])[0]))
        print("      k     recall@1    MRR     recall@20")
        for k in KS:
            g = [rank_of_gold([1 / (k + x) + 1 / (k + y)
                               for x, y in zip(ra, rb)])
                 for ra, rb, _, _ in ranked]
            r1, mrr, r20 = metrics(g)
            print("   {:5d}     {:.3f}     {:.3f}     {:.3f}".format(
                k, r1, mrr, r20))
        best = max((metrics([rank_of_gold([al * x + (1 - al) * y
                                           for x, y in zip(a, b)])
                             for _, _, a, b in ranked])[1], al)
                   for al in [i / 20 for i in range(21)])
        print("   best score blend: alpha {:.2f}, MRR {:.3f}\n".format(
            best[1], best[0]))
    print("Reading: with two equal rankers fusion more than doubles")
    print("recall@1, but k = 1 is clearly worse than k = 20 to 1000.")
    print("With one strong and one weak ranker, every k does worse at the")
    print("head than the strong ranker alone - small k loses least - and")
    print("from k = 200 the tail collapses too. With shared errors fusion")
    print("gains little whatever k is. In none of the three is k = 60 the")
    print("best for recall@1; where scores are comparable, a tuned blend")
    print("beats every k.")
# S1 equal  (alone: A r@1 0.229, B r@1 0.247)
#       k     recall@1    MRR     recall@20
#        1     0.425     0.516     0.847
#        5     0.507     0.588     0.855
#       20     0.586     0.679     0.881
#       60     0.560     0.667     0.917
#      200     0.547     0.650     0.911
#     1000     0.541     0.642     0.895
#    best score blend: alpha 0.50, MRR 0.713
#
# S2 unequal  (alone: A r@1 0.604, B r@1 0.017)
#       k     recall@1    MRR     recall@20
#        1     0.478     0.601     0.892
#        5     0.470     0.601     0.897
#       20     0.297     0.486     0.907
#       60     0.211     0.353     0.903
#      200     0.197     0.297     0.640
#     1000     0.191     0.279     0.567
#    best score blend: alpha 0.80, MRR 0.738
#
# S3 shared  (alone: A r@1 0.243, B r@1 0.229)
#       k     recall@1    MRR     recall@20
#        1     0.309     0.412     0.729
#        5     0.321     0.432     0.737
#       20     0.319     0.434     0.755
#       60     0.323     0.434     0.756
#      200     0.324     0.434     0.751
#     1000     0.324     0.433     0.748
#    best score blend: alpha 0.55, MRR 0.443
#
# Reading: with two equal rankers fusion more than doubles
# recall@1, but k = 1 is clearly worse than k = 20 to 1000.
# With one strong and one weak ranker, every k does worse at the
# head than the strong ranker alone - small k loses least - and
# from k = 200 the tail collapses too. With shared errors fusion
# gains little whatever k is. In none of the three is k = 60 the
# best for recall@1; where scores are comparable, a tuned blend
# beats every k.

三种情形给出三种不同的答案。两个检索器同样好(S1)时,融合让 recall@1 从单个检索器的 0.229 与 0.247 升到最高 0.586(k=20k = 20);k=1k = 1 明显更差(0.425),kk 在 20 到 1000 之间相差不大。一强一弱(S2)时,每一个 kk 在头部都不如强检索器单独使用(0.604):kk 越小损失越少(k=1k = 1 时 0.478),k=60k = 60 时只有 0.211,从 k=200k = 200 起连 recall@20 也崩塌(0.640)。两个检索器犯同样的错(S3)时,融合几乎没有收益,kk 取多少都差不多。

在三种情形里,k=60k = 60 都不是 recall@1 的最优值。若两个检索器的分数可以放到同一尺度上,调好权重的分数凸组合(最优 α\alpha)在 S1 与 S2 中都胜过任何 kk(MRR 0.713 与 0.738)。所以对提纲的修正是:不存在对所有分布都最优的 kk;它取决于检索器的相对强弱与错误的相关程度,需要在自己的查询上调。

倒数秩融合的 recall@1 随平滑常数 k 的变化,三种情形:两检索器同样好、一强一弱、错误相关。

假说 · 尚未验证

作者的假说(尚未验证): 在真实的混合检索系统中,按查询是否含有标识符分两类分别调 kk(或分别设定权重),比全局使用 k=60k = 60 在 recall@1 上至少高 3 个百分点。什么结果会证明它错:在预先登记的评测集上,分类调参与全局 k=60k = 60 的配对差异的 95% 置信区间上限低于 3 个百分点。

小结

词法检索认不出改写,稠密检索认不出编号;两者的盲区互补,但融合不会自动取长补短。融合完整排名时,盲的一方会用噪声抵消知道的一方;只融合短列表并去掉零分块,可以修复改写查询,却让编号查询停在抛硬币附近。RRF 的平滑常数 kk 没有普遍最优值:两检索器相当时 kk 从 20 起差别不大,一强一弱时任何 kk 都不如强者单用,错误相关时融合几乎无益。

参考文献

Cormack, Gordon V., Charles L. A. Clarke, and Stefan Büttcher. 2009. “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.” Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 758–59.
Karpukhin, Vladimir, Barlas Oğuz, Sewon Min, et al. 2020. “Dense Passage Retrieval for Open-Domain Question Answering.” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 6769–81.
Robertson, Stephen, and Hugo Zaragoza. 2009. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval 3 (4): 333–89.

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).