关于「评测本身的统计:样本量、评判偏差与赢家诅咒」,《检索增强与多智能体的可靠性》给出的定量结论是什么?
结论:第六章说明三个百分点的改进需要约三千条查询才能可靠检出(3,000 条时把握度 0.989),从 256 个同样好的配置中挑最好的会虚高七个多百分点。
数据口径:全部数据为程序生成的合成数据,数字说明机制与量级,不是对任何产品的测量。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.989, 3000
n power paired power unpaired
100 0.112 0.009
300 0.248 0.025
1000 0.692 0.227
3000 0.989 0.846
Reading: a 3-point recall gain is invisible on 100 queries
and uncertain on 300; on 1,000 the paired test finds it about
seven times in ten, and only at 3,000 reliably. Ignoring the
pairing - treating the two runs as independent samples - is
far weaker at every size, because it counts the many queries
both systems agree on as noise.查看完整验算程序
# Chapter 6. p12: How many queries does a retrieval comparison need?
# Systems A and B answer the same n queries; a query is a hit if the gold
# chunk is in the top 10. True recall@10: A 0.70, B 0.73 (3 points).
# Their outcomes are positively correlated (most queries are easy or
# hard for both): the two disagree on 15 % of queries. Tests at 5 %:
# paired McNemar, on the discordant queries only (normal approx.)
# unpaired two-proportion z-test, as if the query sets differed
# 2,000 simulated evaluations per n.
import math
import random
def outcomes(rng, n):
# discordant 0.15: B-only hits 0.09, A-only 0.06 -> difference 0.03
a = b = 0
ab, ba = 0, 0
for _ in range(n):
u = rng.random()
if u < 0.09:
b += 1
ba += 1
elif u < 0.15:
a += 1
ab += 1
elif u < 0.15 + 0.64:
a += 1
b += 1
return a, b, ab, ba
if __name__ == "__main__":
rng = random.Random(61012)
print(" n power paired power unpaired")
for n in (100, 300, 1000, 3000):
pp = pu = 0
for _ in range(2000):
a, b, ab, ba = outcomes(rng, n)
d = ab + ba
if d and abs(ba - ab) / math.sqrt(d) > 1.96:
pp += 1
pa, pb = a / n, b / n
pool = (a + b) / (2 * n)
se = math.sqrt(2 * pool * (1 - pool) / n)
if se and abs(pb - pa) / se > 1.96:
pu += 1
print(" {:5d} {:.3f} {:.3f}".format(
n, pp / 2000, pu / 2000))
print("\nReading: a 3-point recall gain is invisible on 100 queries")
print("and uncertain on 300; on 1,000 the paired test finds it about")
print("seven times in ten, and only at 3,000 reliably. Ignoring the")
print("pairing - treating the two runs as independent samples - is")
print("far weaker at every size, because it counts the many queries")
print("both systems agree on as noise.")唯一数据源与引用
范雅福(Fan, Y.). (2026). 《检索增强生成几何、倒数秩融合与多智能体误差级联》第 6 章「评测本身的统计:样本量、评判偏差与赢家诅咒」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170
DOI: 10.5281/zenodo.24090170 · 阅读本章所在专著 · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026061,
author = {Fan, Yafu},
title = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090170},
url = {https://doi.org/10.5281/zenodo.24090170}
}