What does RAG and Multi-Agent Reliability conclude, in numbers, about “The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner's Curse”?

Answer: Chapter 6 shows that a 3-point improvement needs about three thousand queries to be detected reliably (power 0.989 at 3,000) and that picking the best of 256 equally good configurations inflates it by more than seven points.

Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.989, 3000

     n    power paired   power unpaired
     100     0.112          0.009
     300     0.248          0.025
    1000     0.692          0.227
    3000     0.989          0.846
Reading: a 3-point recall gain is invisible on 100 queries
and uncertain on 300; on 1,000 the paired test finds it about
seven times in ten, and only at 3,000 reliably. Ignoring the
pairing - treating the two runs as independent samples - is
far weaker at every size, because it counts the many queries
both systems agree on as noise.
Show the full program
# Chapter 6. p12: How many queries does a retrieval comparison need?
# Systems A and B answer the same n queries; a query is a hit if the gold
# chunk is in the top 10. True recall@10: A 0.70, B 0.73 (3 points).
# Their outcomes are positively correlated (most queries are easy or
# hard for both): the two disagree on 15 % of queries. Tests at 5 %:
#   paired     McNemar, on the discordant queries only (normal approx.)
#   unpaired   two-proportion z-test, as if the query sets differed
# 2,000 simulated evaluations per n.
import math
import random

def outcomes(rng, n):
    # discordant 0.15: B-only hits 0.09, A-only 0.06 -> difference 0.03
    a = b = 0
    ab, ba = 0, 0
    for _ in range(n):
        u = rng.random()
        if u < 0.09:
            b += 1
            ba += 1
        elif u < 0.15:
            a += 1
            ab += 1
        elif u < 0.15 + 0.64:
            a += 1
            b += 1
    return a, b, ab, ba

if __name__ == "__main__":
    rng = random.Random(61012)
    print("     n    power paired   power unpaired")
    for n in (100, 300, 1000, 3000):
        pp = pu = 0
        for _ in range(2000):
            a, b, ab, ba = outcomes(rng, n)
            d = ab + ba
            if d and abs(ba - ab) / math.sqrt(d) > 1.96:
                pp += 1
            pa, pb = a / n, b / n
            pool = (a + b) / (2 * n)
            se = math.sqrt(2 * pool * (1 - pool) / n)
            if se and abs(pb - pa) / se > 1.96:
                pu += 1
        print("   {:5d}     {:.3f}          {:.3f}".format(
            n, pp / 2000, pu / 2000))
    print("\nReading: a 3-point recall gain is invisible on 100 queries")
    print("and uncertain on 300; on 1,000 the paired test finds it about")
    print("seven times in ten, and only at 3,000 reliably. Ignoring the")
    print("pairing - treating the two runs as independent samples - is")
    print("far weaker at every size, because it counts the many queries")
    print("both systems agree on as noise.")

Single source and citation

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 6: The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner's Curse. 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

Other chapter answers in this book