What does RAG and Multi-Agent Reliability conclude, in numbers, about “The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner's Curse”?
Answer: Chapter 6 shows that a 3-point improvement needs about three thousand queries to be detected reliably (power 0.989 at 3,000) and that picking the best of 256 equally good configurations inflates it by more than seven points.
Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.989, 3000
n power paired power unpaired
100 0.112 0.009
300 0.248 0.025
1000 0.692 0.227
3000 0.989 0.846
Reading: a 3-point recall gain is invisible on 100 queries
and uncertain on 300; on 1,000 the paired test finds it about
seven times in ten, and only at 3,000 reliably. Ignoring the
pairing - treating the two runs as independent samples - is
far weaker at every size, because it counts the many queries
both systems agree on as noise.Show the full program
# Chapter 6. p12: How many queries does a retrieval comparison need?
# Systems A and B answer the same n queries; a query is a hit if the gold
# chunk is in the top 10. True recall@10: A 0.70, B 0.73 (3 points).
# Their outcomes are positively correlated (most queries are easy or
# hard for both): the two disagree on 15 % of queries. Tests at 5 %:
# paired McNemar, on the discordant queries only (normal approx.)
# unpaired two-proportion z-test, as if the query sets differed
# 2,000 simulated evaluations per n.
import math
import random
def outcomes(rng, n):
# discordant 0.15: B-only hits 0.09, A-only 0.06 -> difference 0.03
a = b = 0
ab, ba = 0, 0
for _ in range(n):
u = rng.random()
if u < 0.09:
b += 1
ba += 1
elif u < 0.15:
a += 1
ab += 1
elif u < 0.15 + 0.64:
a += 1
b += 1
return a, b, ab, ba
if __name__ == "__main__":
rng = random.Random(61012)
print(" n power paired power unpaired")
for n in (100, 300, 1000, 3000):
pp = pu = 0
for _ in range(2000):
a, b, ab, ba = outcomes(rng, n)
d = ab + ba
if d and abs(ba - ab) / math.sqrt(d) > 1.96:
pp += 1
pa, pb = a / n, b / n
pool = (a + b) / (2 * n)
se = math.sqrt(2 * pool * (1 - pool) / n)
if se and abs(pb - pa) / se > 1.96:
pu += 1
print(" {:5d} {:.3f} {:.3f}".format(
n, pp / 2000, pu / 2000))
print("\nReading: a 3-point recall gain is invisible on 100 queries")
print("and uncertain on 300; on 1,000 the paired test finds it about")
print("seven times in ten, and only at 3,000 reliably. Ignoring the")
print("pairing - treating the two runs as independent samples - is")
print("far weaker at every size, because it counts the many queries")
print("both systems agree on as noise.")Single source and citation
Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 6: The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner's Curse. 5D Library. https://doi.org/10.5281/zenodo.24090170
DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026061,
author = {Fan, Yafu},
title = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090170},
url = {https://doi.org/10.5281/zenodo.24090170}
}Other chapter answers in this book
- Chapter 1 · High-Dimensional Geometry: Concentration of Measure, Effective Dimension and Hub Chunks
- Chapter 2 · Lexical and Dense Retrieval: Their Blind Spots and the Smoothing Constant of Reciprocal Rank Fusion
- Chapter 3 · The Context Window: Position Decay, Evidence Dilution and the Best Number of Chunks
- Chapter 4 · Multi-Agent Pipelines: Error Cascades and the Verifier-Gated Absorbing Markov Chain
- Chapter 5 · Verifier Independence and the Cost of Retries
- Chapter 7 · The Routing of the *Qilue* and Han Fei's *Canyan*: Two Ancient Methods of Retrieval and Verification
- Chapter 8 · A Five-Step Check and a Pre-Registered Offline Evaluation