5D Library › Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades · 中文

Chapter 8. A Five-Step Check and a Pre-Registered Offline Evaluation

Five questions before building a retrieval-augmented or multi-agent system

This chapter gathers the book into five questions that can be put to any RAG or agent system — including one the author builds himself.

1. What is the embedding’s effective dimension? Nominal dimension is not effective dimension (Chapter 1). Compute the participation ratio of the embedding covariance and the skewness of neighbour counts; find the hub chunks and check their content.

2. How many queries are “literal”? Queries with identifiers, codes, version numbers or rare terms are where dense retrieval cannot see (Chapter 2). Evaluate by query type first, then choose the fusion method and weights; do not treat k=60k = 60 as a law.

3. How many chunks in the context? Recall rises with the number of chunks; accuracy need not (Chapter 3). Sweep the number of chunks on one’s own question set and watch accuracy, not recall; put the strongest chunks at both ends.

4. Does every step have an independent check? An unchecked ten-step chain succeeds only 43.4 % of the time (Chapter 4, p08). Write steps as deterministic checks wherever possible; for steps only a model can judge, measure the judge’s detection and false-alarm rates, especially where the generator is weakest (Chapter 5).

5. Can the evaluation see the difference you are looking for? A 3-point improvement needs a thousand queries or more and a paired test; a language-model judge needs both orders; a tuned configuration must be reported on a test set touched once (Chapter 6).

A pre-registered offline evaluation

Hypothesis · not established

Protocol. Question: on one’s own corpus, does hybrid retrieval by reciprocal rank fusion (top 100 from each, BM25 zero-score chunks dropped, k=60k = 60) beat dense retrieval alone? The query set is drawn from logs before the evaluation by a stratification rule fixed in advance (a fixed share of queries with identifiers, codes or rare terms), and the gold chunks are labelled by annotators who see neither system’s output. Primary estimand: the paired difference in recall@10, hybrid minus dense. Smallest effect of interest (SESOI): 3 percentage points. Tests: McNemar on discordant queries (two-sided 5 %), plus an equivalence check (both ends of the 90 % interval within ±3 points). The decision rule is in the program; every branch is named in advance. The evaluation uses only software and de-identified query text from internal logs, and involves no intervention on anyone.

Calculation

# Chapter 8. p16: A pre-registered offline test: hybrid versus dense.
# Question: on one's own corpus, does rank-fused hybrid retrieval (top-100
# lists, BM25 zero scores dropped, k = 60) beat dense retrieval alone?
# Query set: drawn before the test from logged queries by a stratified
# rule (a fixed share containing identifiers, codes or rare terms),
# gold chunks labelled blind to either system's output. Primary estimand:
# paired difference in recall@10, hybrid minus dense. Smallest effect of
# interest (SESOI): 3 points. Test: McNemar on discordant queries,
# two-sided 5 %, plus an equivalence check (both 90 % bounds within
# +-3 points). The model below assumes 15 % discordant queries.
# A: power at true differences of 0, 3 and 6 points. B: the decision
# rule applied to one simulated query set. 2,000 runs per cell.
import math
import random

def sample(rng, n, delta, disc=0.15):
    up = (disc + delta) / 2              # hybrid-only hits
    dn = (disc - delta) / 2              # dense-only hits
    ba = ab = 0
    for _ in range(n):
        u = rng.random()
        if u < up:
            ba += 1
        elif u < up + dn:
            ab += 1
    return ab, ba

def decide(n, ab, ba, sesoi=0.03):
    d = (ba - ab) / n
    se = math.sqrt(ab + ba - (ba - ab) ** 2 / n) / n if ab + ba else 0.0
    lo90, hi90 = d - 1.645 * se, d + 1.645 * se
    lo95 = d - 1.96 * se
    if lo95 > 0 and d >= sesoi:
        return "SUPPORTED: hybrid better by at least the SESOI"
    if lo95 > 0:
        return "BETTER, BELOW SESOI: real but smaller than 3 points"
    if lo90 > -sesoi and hi90 < sesoi:
        return "EQUIVALENT: no difference of practical size"
    if d + 1.96 * se < 0:
        return "REVERSED: dense alone is better"
    return "INCONCLUSIVE: interval too wide"

if __name__ == "__main__":
    rng = random.Random(61016)
    print("A. n queries   true +0   true +3   true +6   (share 'supported')")
    for n in (300, 1000, 2000, 3000):
        row = []
        for delta in (0.0, 0.03, 0.06):
            k = sum(decide(n, *sample(rng, n, delta)).startswith("SUPP")
                    for _ in range(2000))
            row.append(k / 2000)
        print("   {:7d}      {:.3f}     {:.3f}     {:.3f}".format(n, *row))
    ab, ba = sample(rng, 2000, 0.03)
    d = (ba - ab) / 2000
    print("\nB. one query set, n = 2000, true difference +3 points:")
    print("   dense-only hits {}, hybrid-only hits {}, difference {:+.4f}"
          .format(ab, ba, d))
    print("   " + decide(2000, ab, ba))
    tally = {}
    for _ in range(2000):
        v = decide(2000, *sample(rng, 2000, 0.03)).split(":")[0]
        tally[v] = tally.get(v, 0) + 1
    print("   verdicts over 2,000 such query sets:")
    for v in sorted(tally, key=lambda x: -tally[x]):
        print("     {:<20} {:.3f}".format(v, tally[v] / 2000))
    print("\nReading: a true 3-point gain is declared 'supported' only")
    print("about half the time even at 2,000 queries, because the rule")
    print("also demands that the estimate reach the SESOI; a 6-point gain")
    print("is caught reliably from about 1,000. The single query set in B")
    print("drew low and was called 'equivalent' - the kind of error the")
    print("equivalence check allows about one time in twenty when the")
    print("truth sits exactly at the SESOI. The rule names every outcome")
    print("in advance, including the wrong-looking ones.")
# A. n queries   true +0   true +3   true +6   (share 'supported')
#        300      0.026     0.275     0.782
#       1000      0.008     0.502     0.995
#       2000      0.000     0.508     0.999
#       3000      0.000     0.513     1.000
#
# B. one query set, n = 2000, true difference +3 points:
#    dense-only hits 136, hybrid-only hits 158, difference +0.0110
#    EQUIVALENT: no difference of practical size
#    verdicts over 2,000 such query sets:
#      SUPPORTED            0.517
#      BETTER, BELOW SESOI  0.426
#      EQUIVALENT           0.045
#      INCONCLUSIVE         0.011
#
# Reading: a true 3-point gain is declared 'supported' only
# about half the time even at 2,000 queries, because the rule
# also demands that the estimate reach the SESOI; a 6-point gain
# is caught reliably from about 1,000. The single query set in B
# drew low and was called 'equivalent' - the kind of error the
# equivalence check allows about one time in twenty when the
# truth sits exactly at the SESOI. The rule names every outcome
# in advance, including the wrong-looking ones.

With a true difference of 6 points, 1,000 queries already give 0.995 power to call it “supported”; at 3 points — exactly the SESOI — even 2,000 queries give only about half (0.508), because the rule also requires the estimate to reach 3 points. With no true difference, “supported” occurs at most 0.026 of the time.

Part B runs the rule on one simulated set of 2,000 queries, and this one happened to draw low: hybrid alone hits on 158 queries, dense alone on 136, an estimated difference of +0.0110, and the rule says “equivalent” — a wrong conclusion, since in the simulation the true difference is +3 points. Repeating the draw 2,000 times shows how often such errors happen: “equivalent” 0.045, about one time in twenty, exactly the error rate the equivalence check allows when the true effect sits at the SESOI; “supported” 0.517; “better, below SESOI” 0.426. The book shows this piece of “bad luck” as it fell, because it shows how the conclusion of a single evaluation should be read: together with its error rate.

Note

What a result would mean and what it would not. “Supported” says that on this corpus and query distribution, hybrid recall@10 is at least 3 points higher; it does not say question-answering accuracy improves as much (Chapter 3), nor that it holds on another corpus. “Equivalent” does not say the two are the same, only that a difference below 3 points is established with enough confidence.

What this book does not do

This book evaluates no particular embedding model, vector database or language-model product, uses no real user’s data, and recommends no vendor. All corpora, queries, retrieval results and agent runs in it are synthetic data generated by its programs; they show mechanisms and orders of magnitude, and the specific numbers must be measured again on the reader’s own system.

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).