5D Library › Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades · 中文
Chapter 6. The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner’s Curse
How to know an improvement is real
Every recommendation of the first five chapters — hybrid retrieval, tuning , fewer chunks, a verifier — must in the end be evaluated on one’s own data. Evaluation is itself a statistical problem, and an easy one to get wrong: too few queries and a real improvement is invisible; a language model as judge brings its own bias; tune repeatedly on a dev set and the chosen configuration looks better than it is. This chapter works out all three.
How many queries
Calculation
# Chapter 6. p12: How many queries does a retrieval comparison need?
# Systems A and B answer the same n queries; a query is a hit if the gold
# chunk is in the top 10. True recall@10: A 0.70, B 0.73 (3 points).
# Their outcomes are positively correlated (most queries are easy or
# hard for both): the two disagree on 15 % of queries. Tests at 5 %:
# paired McNemar, on the discordant queries only (normal approx.)
# unpaired two-proportion z-test, as if the query sets differed
# 2,000 simulated evaluations per n.
import math
import random
def outcomes(rng, n):
# discordant 0.15: B-only hits 0.09, A-only 0.06 -> difference 0.03
a = b = 0
ab, ba = 0, 0
for _ in range(n):
u = rng.random()
if u < 0.09:
b += 1
ba += 1
elif u < 0.15:
a += 1
ab += 1
elif u < 0.15 + 0.64:
a += 1
b += 1
return a, b, ab, ba
if __name__ == "__main__":
rng = random.Random(61012)
print(" n power paired power unpaired")
for n in (100, 300, 1000, 3000):
pp = pu = 0
for _ in range(2000):
a, b, ab, ba = outcomes(rng, n)
d = ab + ba
if d and abs(ba - ab) / math.sqrt(d) > 1.96:
pp += 1
pa, pb = a / n, b / n
pool = (a + b) / (2 * n)
se = math.sqrt(2 * pool * (1 - pool) / n)
if se and abs(pb - pa) / se > 1.96:
pu += 1
print(" {:5d} {:.3f} {:.3f}".format(
n, pp / 2000, pu / 2000))
print("\nReading: a 3-point recall gain is invisible on 100 queries")
print("and uncertain on 300; on 1,000 the paired test finds it about")
print("seven times in ten, and only at 3,000 reliably. Ignoring the")
print("pairing - treating the two runs as independent samples - is")
print("far weaker at every size, because it counts the many queries")
print("both systems agree on as noise.")
# n power paired power unpaired
# 100 0.112 0.009
# 300 0.248 0.025
# 1000 0.692 0.227
# 3000 0.989 0.846
#
# Reading: a 3-point recall gain is invisible on 100 queries
# and uncertain on 300; on 1,000 the paired test finds it about
# seven times in ten, and only at 3,000 reliably. Ignoring the
# pairing - treating the two runs as independent samples - is
# far weaker at every size, because it counts the many queries
# both systems agree on as noise.System B’s recall@10 is 3 points higher than A’s (0.73 against 0.70), and the two differ on 15 % of queries. With 100 queries a paired test has a power of only 0.112 to detect the difference; with 300, 0.248; with 1,000, 0.692; only at 3,000 does it reach 0.989. Treating the two sets of results as independent samples gives much lower power at every size (only 0.227 at 1,000), because the two systems agree on most queries and that information is counted as noise. McNemar’s test looks only at the queries where the systems differ, which is exactly how it exploits the pairing (McNemar 1947; Smucker et al. 2007).
A language model as judge
Calculation
# Chapter 6. p13: An LLM judge with a position bias.
# Pairwise comparisons of system A against B. The true share of queries
# on which A's answer is better is 0.55. The judge reads the answers in
# a fixed order and, independently of quality, prefers whichever came
# first with extra probability b; otherwise it agrees with the truth with
# accuracy 0.8. Two protocols:
# fixed order A always shown first
# swap each query judged twice, in both orders; ties count
# half
# 400 simulated evaluations of 400 queries each: mean estimated win
# rate for A and the share of evaluations concluding "A better" (95 %
# interval above 0.5).
import math
import random
def judge(rng, a_better, a_first, b):
if rng.random() < b:
return a_first
right = rng.random() < 0.8
return a_better if right else not a_better
def evaluate(rng, b, swap, n=400):
wins = 0.0
for _ in range(n):
a_better = rng.random() < 0.55
if swap:
wins += 0.5 * (judge(rng, a_better, True, b)
+ judge(rng, a_better, False, b))
else:
wins += judge(rng, a_better, True, b)
return wins / n
if __name__ == "__main__":
rng = random.Random(61013)
print(" bias b protocol mean win rate 'A better' called")
for b in (0.0, 0.1, 0.25):
for swap in (False, True):
est = [evaluate(rng, b, swap) for _ in range(400)]
m = sum(est) / len(est)
called = sum(e - 1.96 * math.sqrt(e * (1 - e) / 400) > 0.5
for e in est) / len(est)
print(" {:4.2f} {:<12} {:.3f} {:.3f}".format(
b, "swap" if swap else "fixed order", m, called))
print("\nReading: an unbiased judge that is right 80 % of the time")
print("already shrinks a 0.55 truth towards 0.5. A first-position")
print("bias inflates the system shown first; at b = 0.25 a fixed")
print("order 'proves' A better in every evaluation. Judging both")
print("orders removes the bias, but the biased verdicts still dilute")
print("the signal: the more biased the judge, the less often the real")
print("difference is detected.")
# bias b protocol mean win rate 'A better' called
# 0.00 fixed order 0.530 0.203
# 0.00 swap 0.529 0.150
# 0.10 fixed order 0.577 0.895
# 0.10 swap 0.526 0.130
# 0.25 fixed order 0.646 1.000
# 0.25 swap 0.523 0.070
#
# Reading: an unbiased judge that is right 80 % of the time
# already shrinks a 0.55 truth towards 0.5. A first-position
# bias inflates the system shown first; at b = 0.25 a fixed
# order 'proves' A better in every evaluation. Judging both
# orders removes the bias, but the biased verdicts still dilute
# the signal: the more biased the judge, the less often the real
# difference is detected.In truth, system A’s answer is better on 55 % of queries. A judge with no position bias but only 80 % accuracy squeezes that 0.55 to 0.530. If the judge also favours whichever answer comes first (bias ), a fixed-order evaluation systematically inflates the system shown first: at the estimated win rate is 0.646, and every evaluation “proves” A better. Judging both orders removes the bias (the estimate returns to 0.523), but the biased verdicts still dilute the signal: the more biased the judge, the less often the real difference is detected (0.150, 0.130, 0.070) (Zheng et al. 2023).
The winner’s curse
Calculation
# Chapter 6. p14: Tuning on the dev set: the winner's curse.
# m pipeline configurations (chunk size x top-k x reranker ...) that are
# in truth equally good: recall@10 = 0.70 each. Each is scored on the
# same 300 dev queries; outcomes are correlated across configurations
# (a query that is hard for one is often hard for all: each config's
# hit = shared query difficulty + own noise). The best dev score is
# picked and the winner rescored on 300 fresh test queries. 2,000
# repetitions. Expected inflation ~ sd x sqrt(2 ln m) for independent
# configurations; correlation reduces it.
import math
import random
from statistics import NormalDist
PHI = NormalDist()
Z70 = PHI.inv_cdf(0.70)
def run(rng, m, rho, n=300):
q = [rng.gauss(0, 1) for _ in range(n)]
best = -1.0
for _ in range(m):
hits = sum(1 for z in q
if math.sqrt(rho) * z + math.sqrt(1 - rho)
* rng.gauss(0, 1) < Z70)
best = max(best, hits / n)
test = sum(rng.random() < 0.70 for _ in range(n)) / n
return best, test
if __name__ == "__main__":
rng = random.Random(61014)
sd = math.sqrt(0.7 * 0.3 / 300)
print(" configs m rho best dev fresh test inflation"
" sd*sqrt(2 ln m)")
for rho in (0.0, 0.6):
for m in (1, 4, 16, 64, 256):
res = [run(rng, m, rho) for _ in range(400 if m >= 64 else 2000)]
dev = sum(r[0] for r in res) / len(res)
test = sum(r[1] for r in res) / len(res)
ref = sd * math.sqrt(2 * math.log(m)) if m > 1 else 0.0
print(" {:8d} {:3.1f} {:.4f} {:.4f} {:+.4f}"
" {:.4f}".format(m, rho, dev, test, dev - test, ref))
print("\nReading: with 256 equally good configurations the dev winner")
print("looks more than seven points better than it is, and the gap")
print("vanishes on fresh queries. Correlated queries soften the curse")
print("but do not remove it. A held-out test set, touched once, is")
print("the only honest score of a tuned pipeline.")
# configs m rho best dev fresh test inflation sd*sqrt(2 ln m)
# 1 0.0 0.6999 0.6999 +0.0000 0.0000
# 4 0.0 0.7272 0.7003 +0.0268 0.0441
# 16 0.0 0.7462 0.7000 +0.0462 0.0623
# 64 0.0 0.7612 0.7004 +0.0609 0.0763
# 256 0.0 0.7741 0.7000 +0.0741 0.0881
# 1 0.6 0.7012 0.6996 +0.0016 0.0000
# 4 0.6 0.7217 0.7000 +0.0217 0.0441
# 16 0.6 0.7363 0.6999 +0.0364 0.0623
# 64 0.6 0.7484 0.6990 +0.0494 0.0763
# 256 0.6 0.7573 0.6999 +0.0574 0.0881
#
# Reading: with 256 equally good configurations the dev winner
# looks more than seven points better than it is, and the gap
# vanishes on fresh queries. Correlated queries soften the curse
# but do not remove it. A held-out test set, touched once, is
# the only honest score of a tuned pipeline.configurations that are in truth equally good (recall@10 of 0.70 each) are scored on the same 300 dev queries and the best is chosen. With 16 configurations the winner averages 0.7462 on the dev set; with 256, 0.7741 — apparently more than seven points above the truth. On 300 fresh test queries they all fall back to 0.70. When query difficulty is correlated across configurations (), the inflation is smaller (+0.0574 with 256) but does not vanish. Its size is close to .
This is how conclusions like the outline’s “packing more chunks into the vector store makes it more accurate” commonly get “confirmed” in practice: dozens of combinations are tried on one dev set and the best is reported. A held-out test set, touched once, is the only trustworthy score of a tuned pipeline (Dwork et al. 2015).
Hypothesis · not established
The author’s hypothesis (not yet tested): among published comparisons of RAG configurations, a sizeable share report improvements smaller than the winner’s-curse inflation that this chapter’s p14 gives for the number of configurations they tried. What would show it wrong: checking a batch of papers and technical reports under a sampling rule registered in advance, the reported improvements are generally more than twice the corresponding inflation.
Summary
A 3-point retrieval improvement is invisible on 100 queries, detected seven times in ten on 1,000 and reliably only on 3,000; a paired test is far more powerful than treating the two sets of results as independent. A language-model judge squeezes real differences, and position bias manufactures false winners; judging both orders removes the bias but cannot restore the diluted signal. Picking the best of 256 equally good configurations on one dev set looks more than seven points better; a test set touched once is the honest score.
References
Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).