5D Library › Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades · 中文

Chapter 6. The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner’s Curse

How to know an improvement is real

Every recommendation of the first five chapters — hybrid retrieval, tuning kk, fewer chunks, a verifier — must in the end be evaluated on one’s own data. Evaluation is itself a statistical problem, and an easy one to get wrong: too few queries and a real improvement is invisible; a language model as judge brings its own bias; tune repeatedly on a dev set and the chosen configuration looks better than it is. This chapter works out all three.

How many queries

Calculation

# Chapter 6. p12: How many queries does a retrieval comparison need?
# Systems A and B answer the same n queries; a query is a hit if the gold
# chunk is in the top 10. True recall@10: A 0.70, B 0.73 (3 points).
# Their outcomes are positively correlated (most queries are easy or
# hard for both): the two disagree on 15 % of queries. Tests at 5 %:
#   paired     McNemar, on the discordant queries only (normal approx.)
#   unpaired   two-proportion z-test, as if the query sets differed
# 2,000 simulated evaluations per n.
import math
import random

def outcomes(rng, n):
    # discordant 0.15: B-only hits 0.09, A-only 0.06 -> difference 0.03
    a = b = 0
    ab, ba = 0, 0
    for _ in range(n):
        u = rng.random()
        if u < 0.09:
            b += 1
            ba += 1
        elif u < 0.15:
            a += 1
            ab += 1
        elif u < 0.15 + 0.64:
            a += 1
            b += 1
    return a, b, ab, ba

if __name__ == "__main__":
    rng = random.Random(61012)
    print("     n    power paired   power unpaired")
    for n in (100, 300, 1000, 3000):
        pp = pu = 0
        for _ in range(2000):
            a, b, ab, ba = outcomes(rng, n)
            d = ab + ba
            if d and abs(ba - ab) / math.sqrt(d) > 1.96:
                pp += 1
            pa, pb = a / n, b / n
            pool = (a + b) / (2 * n)
            se = math.sqrt(2 * pool * (1 - pool) / n)
            if se and abs(pb - pa) / se > 1.96:
                pu += 1
        print("   {:5d}     {:.3f}          {:.3f}".format(
            n, pp / 2000, pu / 2000))
    print("\nReading: a 3-point recall gain is invisible on 100 queries")
    print("and uncertain on 300; on 1,000 the paired test finds it about")
    print("seven times in ten, and only at 3,000 reliably. Ignoring the")
    print("pairing - treating the two runs as independent samples - is")
    print("far weaker at every size, because it counts the many queries")
    print("both systems agree on as noise.")
#      n    power paired   power unpaired
#      100     0.112          0.009
#      300     0.248          0.025
#     1000     0.692          0.227
#     3000     0.989          0.846
#
# Reading: a 3-point recall gain is invisible on 100 queries
# and uncertain on 300; on 1,000 the paired test finds it about
# seven times in ten, and only at 3,000 reliably. Ignoring the
# pairing - treating the two runs as independent samples - is
# far weaker at every size, because it counts the many queries
# both systems agree on as noise.

System B’s recall@10 is 3 points higher than A’s (0.73 against 0.70), and the two differ on 15 % of queries. With 100 queries a paired test has a power of only 0.112 to detect the difference; with 300, 0.248; with 1,000, 0.692; only at 3,000 does it reach 0.989. Treating the two sets of results as independent samples gives much lower power at every size (only 0.227 at 1,000), because the two systems agree on most queries and that information is counted as noise. McNemar’s test looks only at the queries where the systems differ, which is exactly how it exploits the pairing (McNemar 1947; Smucker et al. 2007).

A language model as judge

Calculation

# Chapter 6. p13: An LLM judge with a position bias.
# Pairwise comparisons of system A against B. The true share of queries
# on which A's answer is better is 0.55. The judge reads the answers in
# a fixed order and, independently of quality, prefers whichever came
# first with extra probability b; otherwise it agrees with the truth with
# accuracy 0.8. Two protocols:
#   fixed order   A always shown first
#   swap          each query judged twice, in both orders; ties count
#                 half
# 400 simulated evaluations of 400 queries each: mean estimated win
# rate for A and the share of evaluations concluding "A better" (95 %
# interval above 0.5).
import math
import random

def judge(rng, a_better, a_first, b):
    if rng.random() < b:
        return a_first
    right = rng.random() < 0.8
    return a_better if right else not a_better

def evaluate(rng, b, swap, n=400):
    wins = 0.0
    for _ in range(n):
        a_better = rng.random() < 0.55
        if swap:
            wins += 0.5 * (judge(rng, a_better, True, b)
                           + judge(rng, a_better, False, b))
        else:
            wins += judge(rng, a_better, True, b)
    return wins / n

if __name__ == "__main__":
    rng = random.Random(61013)
    print("   bias b   protocol      mean win rate   'A better' called")
    for b in (0.0, 0.1, 0.25):
        for swap in (False, True):
            est = [evaluate(rng, b, swap) for _ in range(400)]
            m = sum(est) / len(est)
            called = sum(e - 1.96 * math.sqrt(e * (1 - e) / 400) > 0.5
                         for e in est) / len(est)
            print("    {:4.2f}    {:<12}     {:.3f}           {:.3f}".format(
                b, "swap" if swap else "fixed order", m, called))
    print("\nReading: an unbiased judge that is right 80 % of the time")
    print("already shrinks a 0.55 truth towards 0.5. A first-position")
    print("bias inflates the system shown first; at b = 0.25 a fixed")
    print("order 'proves' A better in every evaluation. Judging both")
    print("orders removes the bias, but the biased verdicts still dilute")
    print("the signal: the more biased the judge, the less often the real")
    print("difference is detected.")
#    bias b   protocol      mean win rate   'A better' called
#     0.00    fixed order      0.530           0.203
#     0.00    swap             0.529           0.150
#     0.10    fixed order      0.577           0.895
#     0.10    swap             0.526           0.130
#     0.25    fixed order      0.646           1.000
#     0.25    swap             0.523           0.070
#
# Reading: an unbiased judge that is right 80 % of the time
# already shrinks a 0.55 truth towards 0.5. A first-position
# bias inflates the system shown first; at b = 0.25 a fixed
# order 'proves' A better in every evaluation. Judging both
# orders removes the bias, but the biased verdicts still dilute
# the signal: the more biased the judge, the less often the real
# difference is detected.

In truth, system A’s answer is better on 55 % of queries. A judge with no position bias but only 80 % accuracy squeezes that 0.55 to 0.530. If the judge also favours whichever answer comes first (bias bb), a fixed-order evaluation systematically inflates the system shown first: at b=0.25b = 0.25 the estimated win rate is 0.646, and every evaluation “proves” A better. Judging both orders removes the bias (the estimate returns to 0.523), but the biased verdicts still dilute the signal: the more biased the judge, the less often the real difference is detected (0.150, 0.130, 0.070) (Zheng et al. 2023).

The winner’s curse

Calculation

# Chapter 6. p14: Tuning on the dev set: the winner's curse.
# m pipeline configurations (chunk size x top-k x reranker ...) that are
# in truth equally good: recall@10 = 0.70 each. Each is scored on the
# same 300 dev queries; outcomes are correlated across configurations
# (a query that is hard for one is often hard for all: each config's
# hit = shared query difficulty + own noise). The best dev score is
# picked and the winner rescored on 300 fresh test queries. 2,000
# repetitions. Expected inflation ~ sd x sqrt(2 ln m) for independent
# configurations; correlation reduces it.
import math
import random
from statistics import NormalDist

PHI = NormalDist()
Z70 = PHI.inv_cdf(0.70)

def run(rng, m, rho, n=300):
    q = [rng.gauss(0, 1) for _ in range(n)]
    best = -1.0
    for _ in range(m):
        hits = sum(1 for z in q
                   if math.sqrt(rho) * z + math.sqrt(1 - rho)
                   * rng.gauss(0, 1) < Z70)
        best = max(best, hits / n)
    test = sum(rng.random() < 0.70 for _ in range(n)) / n
    return best, test

if __name__ == "__main__":
    rng = random.Random(61014)
    sd = math.sqrt(0.7 * 0.3 / 300)
    print("   configs m   rho   best dev   fresh test   inflation"
          "   sd*sqrt(2 ln m)")
    for rho in (0.0, 0.6):
        for m in (1, 4, 16, 64, 256):
            res = [run(rng, m, rho) for _ in range(400 if m >= 64 else 2000)]
            dev = sum(r[0] for r in res) / len(res)
            test = sum(r[1] for r in res) / len(res)
            ref = sd * math.sqrt(2 * math.log(m)) if m > 1 else 0.0
            print("   {:8d}   {:3.1f}    {:.4f}     {:.4f}      {:+.4f}"
                  "      {:.4f}".format(m, rho, dev, test, dev - test, ref))
    print("\nReading: with 256 equally good configurations the dev winner")
    print("looks more than seven points better than it is, and the gap")
    print("vanishes on fresh queries. Correlated queries soften the curse")
    print("but do not remove it. A held-out test set, touched once, is")
    print("the only honest score of a tuned pipeline.")
#    configs m   rho   best dev   fresh test   inflation   sd*sqrt(2 ln m)
#           1   0.0    0.6999     0.6999      +0.0000      0.0000
#           4   0.0    0.7272     0.7003      +0.0268      0.0441
#          16   0.0    0.7462     0.7000      +0.0462      0.0623
#          64   0.0    0.7612     0.7004      +0.0609      0.0763
#         256   0.0    0.7741     0.7000      +0.0741      0.0881
#           1   0.6    0.7012     0.6996      +0.0016      0.0000
#           4   0.6    0.7217     0.7000      +0.0217      0.0441
#          16   0.6    0.7363     0.6999      +0.0364      0.0623
#          64   0.6    0.7484     0.6990      +0.0494      0.0763
#         256   0.6    0.7573     0.6999      +0.0574      0.0881
#
# Reading: with 256 equally good configurations the dev winner
# looks more than seven points better than it is, and the gap
# vanishes on fresh queries. Correlated queries soften the curse
# but do not remove it. A held-out test set, touched once, is
# the only honest score of a tuned pipeline.

mm configurations that are in truth equally good (recall@10 of 0.70 each) are scored on the same 300 dev queries and the best is chosen. With 16 configurations the winner averages 0.7462 on the dev set; with 256, 0.7741 — apparently more than seven points above the truth. On 300 fresh test queries they all fall back to 0.70. When query difficulty is correlated across configurations (ρ=0.6\rho = 0.6), the inflation is smaller (+0.0574 with 256) but does not vanish. Its size is close to σ2ln⁡m\sigma\sqrt{2\ln m}.

This is how conclusions like the outline’s “packing more chunks into the vector store makes it more accurate” commonly get “confirmed” in practice: dozens of combinations are tried on one dev set and the best is reported. A held-out test set, touched once, is the only trustworthy score of a tuned pipeline (Dwork et al. 2015).

The winner’s curse: the more equally good configurations, the higher the best dev score, while all fall back to the truth on fresh data.

Hypothesis · not established

The author’s hypothesis (not yet tested): among published comparisons of RAG configurations, a sizeable share report improvements smaller than the winner’s-curse inflation that this chapter’s p14 gives for the number of configurations they tried. What would show it wrong: checking a batch of papers and technical reports under a sampling rule registered in advance, the reported improvements are generally more than twice the corresponding inflation.

Summary

A 3-point retrieval improvement is invisible on 100 queries, detected seven times in ten on 1,000 and reliably only on 3,000; a paired test is far more powerful than treating the two sets of results as independent. A language-model judge squeezes real differences, and position bias manufactures false winners; judging both orders removes the bias but cannot restore the diluted signal. Picking the best of 256 equally good configurations on one dev set looks more than seven points better; a test set touched once is the honest score.

References

Dwork, Cynthia, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. 2015. “The Reusable Holdout: Preserving Validity in Adaptive Data Analysis.” Science 349 (6248): 636–38.
McNemar, Quinn. 1947. “Note on the Sampling Error of the Difference between Correlated Proportions or Percentages.” Psychometrika 12 (2): 153–57.
Smucker, Mark D., James Allan, and Ben Carterette. 2007. “A Comparison of Statistical Significance Tests for Information Retrieval Evaluation.” Proceedings of the 16th ACM Conference on Information and Knowledge Management (CIKM), 623–32.
Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, et al. 2023. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” Advances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track.

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).