5D Library › 检索增强生成几何、倒数秩融合与多智能体误差级联 · English

第八章 五步核查与一项预注册的离线评测

搭一个检索增强或多智能体系统之前的五个问题

本章把全书收成五个问题,可以向任何 RAG 或智能体系统提出——包括作者自己搭的那一个。

一、嵌入的有效维度是多少? 名义维度不等于有效维度(第一章)。计算嵌入协方差的参与率,统计近邻出现次数的偏度,找出枢纽块并检查它们的内容。

二、查询里有多少是「字面」的? 含标识符、编号、版本号、罕见术语的查询,稠密检索看不清(第二章)。先按查询类型拆开评测,再决定融合方式与权重,不要把 k=60k = 60 当成定律。

三、上下文放几块? 召回率随块数上升,准确率不一定(第三章)。在自己的问答集上扫描块数,看准确率而不是召回率;把最强的块放在两端。

四、每一步有没有独立的检查? 无检查的十步链只剩 43.4% 能成功(第四章,p08)。能写成确定性检查的步骤就写成检查;只能由模型评判的步骤,测量评判的检出率与误报率,特别是在生成器最弱的地方(第五章)。

五、评测能不能看见你要找的差别? 三个百分点的改进需要上千条查询、配对检验;语言模型评判要交换顺序;调过参的配置要在只碰一次的测试集上报告(第六章)。

一项预注册的离线评测

假说 · 尚未验证

研究方案。 问题:在自己的语料上,倒数秩融合的混合检索(各取前 100 名、去掉 BM25 零分块、k=60k = 60)是否优于单独的稠密检索?查询集在评测前按事先规定的分层规则从日志中抽取(含标识符、编号或罕见术语的查询占固定比例),正确块由不看任何系统输出的标注者标注。主要估计量:配对的 recall@10 之差(混合减稠密)。最小关注效应(SESOI):3 个百分点。检验:对不一致查询作 McNemar 检验(双侧 5%),并作等价检验(90% 区间两端都在 ±3 个百分点之内)。判定规则见程序;所有分支事先命名。本评测只使用软件与内部日志中去标识化的查询文本,不涉及对任何人的干预。

推导

# Chapter 8. p16: A pre-registered offline test: hybrid versus dense.
# Question: on one's own corpus, does rank-fused hybrid retrieval (top-100
# lists, BM25 zero scores dropped, k = 60) beat dense retrieval alone?
# Query set: drawn before the test from logged queries by a stratified
# rule (a fixed share containing identifiers, codes or rare terms),
# gold chunks labelled blind to either system's output. Primary estimand:
# paired difference in recall@10, hybrid minus dense. Smallest effect of
# interest (SESOI): 3 points. Test: McNemar on discordant queries,
# two-sided 5 %, plus an equivalence check (both 90 % bounds within
# +-3 points). The model below assumes 15 % discordant queries.
# A: power at true differences of 0, 3 and 6 points. B: the decision
# rule applied to one simulated query set. 2,000 runs per cell.
import math
import random

def sample(rng, n, delta, disc=0.15):
    up = (disc + delta) / 2              # hybrid-only hits
    dn = (disc - delta) / 2              # dense-only hits
    ba = ab = 0
    for _ in range(n):
        u = rng.random()
        if u < up:
            ba += 1
        elif u < up + dn:
            ab += 1
    return ab, ba

def decide(n, ab, ba, sesoi=0.03):
    d = (ba - ab) / n
    se = math.sqrt(ab + ba - (ba - ab) ** 2 / n) / n if ab + ba else 0.0
    lo90, hi90 = d - 1.645 * se, d + 1.645 * se
    lo95 = d - 1.96 * se
    if lo95 > 0 and d >= sesoi:
        return "SUPPORTED: hybrid better by at least the SESOI"
    if lo95 > 0:
        return "BETTER, BELOW SESOI: real but smaller than 3 points"
    if lo90 > -sesoi and hi90 < sesoi:
        return "EQUIVALENT: no difference of practical size"
    if d + 1.96 * se < 0:
        return "REVERSED: dense alone is better"
    return "INCONCLUSIVE: interval too wide"

if __name__ == "__main__":
    rng = random.Random(61016)
    print("A. n queries   true +0   true +3   true +6   (share 'supported')")
    for n in (300, 1000, 2000, 3000):
        row = []
        for delta in (0.0, 0.03, 0.06):
            k = sum(decide(n, *sample(rng, n, delta)).startswith("SUPP")
                    for _ in range(2000))
            row.append(k / 2000)
        print("   {:7d}      {:.3f}     {:.3f}     {:.3f}".format(n, *row))
    ab, ba = sample(rng, 2000, 0.03)
    d = (ba - ab) / 2000
    print("\nB. one query set, n = 2000, true difference +3 points:")
    print("   dense-only hits {}, hybrid-only hits {}, difference {:+.4f}"
          .format(ab, ba, d))
    print("   " + decide(2000, ab, ba))
    tally = {}
    for _ in range(2000):
        v = decide(2000, *sample(rng, 2000, 0.03)).split(":")[0]
        tally[v] = tally.get(v, 0) + 1
    print("   verdicts over 2,000 such query sets:")
    for v in sorted(tally, key=lambda x: -tally[x]):
        print("     {:<20} {:.3f}".format(v, tally[v] / 2000))
    print("\nReading: a true 3-point gain is declared 'supported' only")
    print("about half the time even at 2,000 queries, because the rule")
    print("also demands that the estimate reach the SESOI; a 6-point gain")
    print("is caught reliably from about 1,000. The single query set in B")
    print("drew low and was called 'equivalent' - the kind of error the")
    print("equivalence check allows about one time in twenty when the")
    print("truth sits exactly at the SESOI. The rule names every outcome")
    print("in advance, including the wrong-looking ones.")
# A. n queries   true +0   true +3   true +6   (share 'supported')
#        300      0.026     0.275     0.782
#       1000      0.008     0.502     0.995
#       2000      0.000     0.508     0.999
#       3000      0.000     0.513     1.000
#
# B. one query set, n = 2000, true difference +3 points:
#    dense-only hits 136, hybrid-only hits 158, difference +0.0110
#    EQUIVALENT: no difference of practical size
#    verdicts over 2,000 such query sets:
#      SUPPORTED            0.517
#      BETTER, BELOW SESOI  0.426
#      EQUIVALENT           0.045
#      INCONCLUSIVE         0.011
#
# Reading: a true 3-point gain is declared 'supported' only
# about half the time even at 2,000 queries, because the rule
# also demands that the estimate reach the SESOI; a 6-point gain
# is caught reliably from about 1,000. The single query set in B
# drew low and was called 'equivalent' - the kind of error the
# equivalence check allows about one time in twenty when the
# truth sits exactly at the SESOI. The rule names every outcome
# in advance, including the wrong-looking ones.

真实差别为 6 个百分点时,1,000 条查询已有 0.995 的把握判为「支持」;为 3 个百分点、恰好等于最小关注效应时,即使 2,000 条也只有一半左右(0.508),因为规则同时要求估计值达到 3 个百分点。差别为零时,「支持」的比例不超过 0.026。

B 段在一份模拟的 2,000 条查询集上演示判定规则,而这一份恰好抽得偏低:混合检索只在 158 条查询上独中,稠密检索独中 136 条,估计差 +0.0110,规则判为「等价」——这是一个错误的结论,因为模拟中真实差别是 +3 个百分点。重复 2,000 次可以看到这种错误的频率:「等价」占 0.045,约二十次一次,正是等价检验在真实效应恰等于最小关注效应时所允许的错误率;「支持」占 0.517,「更好但低于 SESOI」占 0.426。本书如实展示这一次「坏运气」,因为它说明了单次评测的结论应当怎样读:带着它的错误率读。

注

结果意味着什么、不意味着什么。 「支持」说明在这一语料、这一查询分布上,混合检索的 recall@10 至少高 3 个百分点;它不说明问答准确率同样提高(第三章),也不说明换一个语料还成立。「等价」不说明两者完全相同,只说明差别小于 3 个百分点的把握足够大。

本书没有做的事

本书没有评测任何具体的嵌入模型、向量数据库或语言模型产品,没有使用任何真实用户的数据,也不推荐任何供应商。书中所有语料、查询、检索结果与智能体运行都是程序生成的合成数据;它们说明的是机制与量级,具体数字须在读者自己的系统上重新测量。

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).