5D Library › 检索增强生成几何、倒数秩融合与多智能体误差级联 · English

第六章 评测本身的统计:样本量、评判偏差与赢家诅咒

怎样知道改进是真的

前五章的每一个建议——混合检索、调 kk、减少块数、加验证器——最终都要在自己的数据上评测。评测本身也是一个统计问题,而且很容易出错:查询太少,看不见真实的改进;用语言模型当评判,评判有偏差;在开发集上反复调参,选出来的配置看上去比实际更好。本章算清这三件事。

需要多少条查询

推导

# Chapter 6. p12: How many queries does a retrieval comparison need?
# Systems A and B answer the same n queries; a query is a hit if the gold
# chunk is in the top 10. True recall@10: A 0.70, B 0.73 (3 points).
# Their outcomes are positively correlated (most queries are easy or
# hard for both): the two disagree on 15 % of queries. Tests at 5 %:
#   paired     McNemar, on the discordant queries only (normal approx.)
#   unpaired   two-proportion z-test, as if the query sets differed
# 2,000 simulated evaluations per n.
import math
import random

def outcomes(rng, n):
    # discordant 0.15: B-only hits 0.09, A-only 0.06 -> difference 0.03
    a = b = 0
    ab, ba = 0, 0
    for _ in range(n):
        u = rng.random()
        if u < 0.09:
            b += 1
            ba += 1
        elif u < 0.15:
            a += 1
            ab += 1
        elif u < 0.15 + 0.64:
            a += 1
            b += 1
    return a, b, ab, ba

if __name__ == "__main__":
    rng = random.Random(61012)
    print("     n    power paired   power unpaired")
    for n in (100, 300, 1000, 3000):
        pp = pu = 0
        for _ in range(2000):
            a, b, ab, ba = outcomes(rng, n)
            d = ab + ba
            if d and abs(ba - ab) / math.sqrt(d) > 1.96:
                pp += 1
            pa, pb = a / n, b / n
            pool = (a + b) / (2 * n)
            se = math.sqrt(2 * pool * (1 - pool) / n)
            if se and abs(pb - pa) / se > 1.96:
                pu += 1
        print("   {:5d}     {:.3f}          {:.3f}".format(
            n, pp / 2000, pu / 2000))
    print("\nReading: a 3-point recall gain is invisible on 100 queries")
    print("and uncertain on 300; on 1,000 the paired test finds it about")
    print("seven times in ten, and only at 3,000 reliably. Ignoring the")
    print("pairing - treating the two runs as independent samples - is")
    print("far weaker at every size, because it counts the many queries")
    print("both systems agree on as noise.")
#      n    power paired   power unpaired
#      100     0.112          0.009
#      300     0.248          0.025
#     1000     0.692          0.227
#     3000     0.989          0.846
#
# Reading: a 3-point recall gain is invisible on 100 queries
# and uncertain on 300; on 1,000 the paired test finds it about
# seven times in ten, and only at 3,000 reliably. Ignoring the
# pairing - treating the two runs as independent samples - is
# far weaker at every size, because it counts the many queries
# both systems agree on as noise.

系统 B 的 recall@10 比 A 高 3 个百分点(0.73 对 0.70),两者在 15% 的查询上结果不同。100 条查询时,配对检验只有 0.112 的把握度发现这个差别;300 条时 0.248;1,000 条时 0.692;要到 3,000 条才达到 0.989。若把两组结果当成独立样本来检验,把握度在每个规模上都低得多(1,000 条时只有 0.227),因为两个系统在大多数查询上结果相同,这部分信息被当成了噪声。McNemar 检验只看两者不同的查询,正是利用了这种配对 (McNemar 1947; Smucker et al. 2007)。

语言模型当评判

推导

# Chapter 6. p13: An LLM judge with a position bias.
# Pairwise comparisons of system A against B. The true share of queries
# on which A's answer is better is 0.55. The judge reads the answers in
# a fixed order and, independently of quality, prefers whichever came
# first with extra probability b; otherwise it agrees with the truth with
# accuracy 0.8. Two protocols:
#   fixed order   A always shown first
#   swap          each query judged twice, in both orders; ties count
#                 half
# 400 simulated evaluations of 400 queries each: mean estimated win
# rate for A and the share of evaluations concluding "A better" (95 %
# interval above 0.5).
import math
import random

def judge(rng, a_better, a_first, b):
    if rng.random() < b:
        return a_first
    right = rng.random() < 0.8
    return a_better if right else not a_better

def evaluate(rng, b, swap, n=400):
    wins = 0.0
    for _ in range(n):
        a_better = rng.random() < 0.55
        if swap:
            wins += 0.5 * (judge(rng, a_better, True, b)
                           + judge(rng, a_better, False, b))
        else:
            wins += judge(rng, a_better, True, b)
    return wins / n

if __name__ == "__main__":
    rng = random.Random(61013)
    print("   bias b   protocol      mean win rate   'A better' called")
    for b in (0.0, 0.1, 0.25):
        for swap in (False, True):
            est = [evaluate(rng, b, swap) for _ in range(400)]
            m = sum(est) / len(est)
            called = sum(e - 1.96 * math.sqrt(e * (1 - e) / 400) > 0.5
                         for e in est) / len(est)
            print("    {:4.2f}    {:<12}     {:.3f}           {:.3f}".format(
                b, "swap" if swap else "fixed order", m, called))
    print("\nReading: an unbiased judge that is right 80 % of the time")
    print("already shrinks a 0.55 truth towards 0.5. A first-position")
    print("bias inflates the system shown first; at b = 0.25 a fixed")
    print("order 'proves' A better in every evaluation. Judging both")
    print("orders removes the bias, but the biased verdicts still dilute")
    print("the signal: the more biased the judge, the less often the real")
    print("difference is detected.")
#    bias b   protocol      mean win rate   'A better' called
#     0.00    fixed order      0.530           0.203
#     0.00    swap             0.529           0.150
#     0.10    fixed order      0.577           0.895
#     0.10    swap             0.526           0.130
#     0.25    fixed order      0.646           1.000
#     0.25    swap             0.523           0.070
#
# Reading: an unbiased judge that is right 80 % of the time
# already shrinks a 0.55 truth towards 0.5. A first-position
# bias inflates the system shown first; at b = 0.25 a fixed
# order 'proves' A better in every evaluation. Judging both
# orders removes the bias, but the biased verdicts still dilute
# the signal: the more biased the judge, the less often the real
# difference is detected.

在真实情况下,系统 A 的回答在 55% 的查询上更好。一个没有位置偏差、但只有 80% 准确率的评判,把这个 0.55 压缩到 0.530。若评判还偏爱先出现的回答(偏差 bb),固定顺序的评测就会系统性地抬高排在前面的系统:b=0.25b = 0.25 时估计胜率 0.646,所有评测都「证明」A 更好。交换顺序各评一次可以去掉偏差(估计回到 0.523),但有偏差的评判结果仍然稀释信号:偏差越大,真实差别被检出的比例越低(0.150、0.130、0.070)(Zheng et al. 2023)。

赢家诅咒

推导

# Chapter 6. p14: Tuning on the dev set: the winner's curse.
# m pipeline configurations (chunk size x top-k x reranker ...) that are
# in truth equally good: recall@10 = 0.70 each. Each is scored on the
# same 300 dev queries; outcomes are correlated across configurations
# (a query that is hard for one is often hard for all: each config's
# hit = shared query difficulty + own noise). The best dev score is
# picked and the winner rescored on 300 fresh test queries. 2,000
# repetitions. Expected inflation ~ sd x sqrt(2 ln m) for independent
# configurations; correlation reduces it.
import math
import random
from statistics import NormalDist

PHI = NormalDist()
Z70 = PHI.inv_cdf(0.70)

def run(rng, m, rho, n=300):
    q = [rng.gauss(0, 1) for _ in range(n)]
    best = -1.0
    for _ in range(m):
        hits = sum(1 for z in q
                   if math.sqrt(rho) * z + math.sqrt(1 - rho)
                   * rng.gauss(0, 1) < Z70)
        best = max(best, hits / n)
    test = sum(rng.random() < 0.70 for _ in range(n)) / n
    return best, test

if __name__ == "__main__":
    rng = random.Random(61014)
    sd = math.sqrt(0.7 * 0.3 / 300)
    print("   configs m   rho   best dev   fresh test   inflation"
          "   sd*sqrt(2 ln m)")
    for rho in (0.0, 0.6):
        for m in (1, 4, 16, 64, 256):
            res = [run(rng, m, rho) for _ in range(400 if m >= 64 else 2000)]
            dev = sum(r[0] for r in res) / len(res)
            test = sum(r[1] for r in res) / len(res)
            ref = sd * math.sqrt(2 * math.log(m)) if m > 1 else 0.0
            print("   {:8d}   {:3.1f}    {:.4f}     {:.4f}      {:+.4f}"
                  "      {:.4f}".format(m, rho, dev, test, dev - test, ref))
    print("\nReading: with 256 equally good configurations the dev winner")
    print("looks more than seven points better than it is, and the gap")
    print("vanishes on fresh queries. Correlated queries soften the curse")
    print("but do not remove it. A held-out test set, touched once, is")
    print("the only honest score of a tuned pipeline.")
#    configs m   rho   best dev   fresh test   inflation   sd*sqrt(2 ln m)
#           1   0.0    0.6999     0.6999      +0.0000      0.0000
#           4   0.0    0.7272     0.7003      +0.0268      0.0441
#          16   0.0    0.7462     0.7000      +0.0462      0.0623
#          64   0.0    0.7612     0.7004      +0.0609      0.0763
#         256   0.0    0.7741     0.7000      +0.0741      0.0881
#           1   0.6    0.7012     0.6996      +0.0016      0.0000
#           4   0.6    0.7217     0.7000      +0.0217      0.0441
#          16   0.6    0.7363     0.6999      +0.0364      0.0623
#          64   0.6    0.7484     0.6990      +0.0494      0.0763
#         256   0.6    0.7573     0.6999      +0.0574      0.0881
#
# Reading: with 256 equally good configurations the dev winner
# looks more than seven points better than it is, and the gap
# vanishes on fresh queries. Correlated queries soften the curse
# but do not remove it. A held-out test set, touched once, is
# the only honest score of a tuned pipeline.

mm 个真实效果完全相同(recall@10 都是 0.70)的配置在同一组 300 条开发查询上评分,选出最好的一个。16 个配置时,选出的配置在开发集上平均 0.7462;256 个时 0.7741——看起来比真实值高出七个多百分点。在新的 300 条测试查询上,它们都回到 0.70。查询难度在配置之间相关(ρ=0.6\rho = 0.6)时,膨胀小一些(256 个时 +0.0574),但并不消失。膨胀的量级接近 σ2ln⁡m\sigma\sqrt{2\ln m}。

这就是提纲中「切块塞进向量库就越准」之类结论在实践中被「证实」的常见方式:在同一个开发集上试了几十种组合,报告最好的那一个。只碰一次的留出测试集,是调过参的流水线唯一可信的分数 (Dwork et al. 2015)。

赢家诅咒:同样好的配置越多,开发集上选出的最好分数越高,而在新数据上都回到真实值。

假说 · 尚未验证

作者的假说(尚未验证): 公开发表的 RAG 配置比较中,相当一部分报告的提高小于本章 p14 给出的、与其所试配置数相应的赢家诅咒膨胀量。什么结果会证明它错:按事先登记的抽样规则检查一批论文与技术报告,报告的提高普遍大于相应的膨胀量的两倍。

小结

三个百分点的检索改进,在 100 条查询上看不见,在 1,000 条上有七成把握,在 3,000 条上才可靠;配对检验的把握度远高于把两组结果当独立样本。语言模型评判压缩真实差别,位置偏差会制造假胜者;交换顺序去掉偏差,但不能恢复被稀释的信号。在同一开发集上从 256 个同样好的配置中挑最好的,看上去会好七个多百分点;只碰一次的测试集才是诚实的分数。

参考文献

Dwork, Cynthia, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. 2015. “The Reusable Holdout: Preserving Validity in Adaptive Data Analysis.” Science 349 (6248): 636–38.
McNemar, Quinn. 1947. “Note on the Sampling Error of the Difference between Correlated Proportions or Percentages.” Psychometrika 12 (2): 153–57.
Smucker, Mark D., James Allan, and Ben Carterette. 2007. “A Comparison of Statistical Significance Tests for Information Retrieval Evaluation.” Proceedings of the 16th ACM Conference on Information and Knowledge Management (CIKM), 623–32.
Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, et al. 2023. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” Advances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track.

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).