关于「评测本身的统计:样本量、评判偏差与赢家诅咒」,《检索增强与多智能体的可靠性》给出的定量结论是什么?

结论:第六章说明三个百分点的改进需要约三千条查询才能可靠检出(3,000 条时把握度 0.989),从 256 个同样好的配置中挑最好的会虚高七个多百分点。

数据口径:全部数据为程序生成的合成数据,数字说明机制与量级,不是对任何产品的测量。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.989, 3000

     n    power paired   power unpaired
     100     0.112          0.009
     300     0.248          0.025
    1000     0.692          0.227
    3000     0.989          0.846
Reading: a 3-point recall gain is invisible on 100 queries
and uncertain on 300; on 1,000 the paired test finds it about
seven times in ten, and only at 3,000 reliably. Ignoring the
pairing - treating the two runs as independent samples - is
far weaker at every size, because it counts the many queries
both systems agree on as noise.
查看完整验算程序
# Chapter 6. p12: How many queries does a retrieval comparison need?
# Systems A and B answer the same n queries; a query is a hit if the gold
# chunk is in the top 10. True recall@10: A 0.70, B 0.73 (3 points).
# Their outcomes are positively correlated (most queries are easy or
# hard for both): the two disagree on 15 % of queries. Tests at 5 %:
#   paired     McNemar, on the discordant queries only (normal approx.)
#   unpaired   two-proportion z-test, as if the query sets differed
# 2,000 simulated evaluations per n.
import math
import random

def outcomes(rng, n):
    # discordant 0.15: B-only hits 0.09, A-only 0.06 -> difference 0.03
    a = b = 0
    ab, ba = 0, 0
    for _ in range(n):
        u = rng.random()
        if u < 0.09:
            b += 1
            ba += 1
        elif u < 0.15:
            a += 1
            ab += 1
        elif u < 0.15 + 0.64:
            a += 1
            b += 1
    return a, b, ab, ba

if __name__ == "__main__":
    rng = random.Random(61012)
    print("     n    power paired   power unpaired")
    for n in (100, 300, 1000, 3000):
        pp = pu = 0
        for _ in range(2000):
            a, b, ab, ba = outcomes(rng, n)
            d = ab + ba
            if d and abs(ba - ab) / math.sqrt(d) > 1.96:
                pp += 1
            pa, pb = a / n, b / n
            pool = (a + b) / (2 * n)
            se = math.sqrt(2 * pool * (1 - pool) / n)
            if se and abs(pb - pa) / se > 1.96:
                pu += 1
        print("   {:5d}     {:.3f}          {:.3f}".format(
            n, pp / 2000, pu / 2000))
    print("\nReading: a 3-point recall gain is invisible on 100 queries")
    print("and uncertain on 300; on 1,000 the paired test finds it about")
    print("seven times in ten, and only at 3,000 reliably. Ignoring the")
    print("pairing - treating the two runs as independent samples - is")
    print("far weaker at every size, because it counts the many queries")
    print("both systems agree on as noise.")

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《检索增强生成几何、倒数秩融合与多智能体误差级联》第 6 章「评测本身的统计:样本量、评判偏差与赢家诅咒」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · 阅读本章所在专著 · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

本书其他章节结论