What does RAG and Multi-Agent Reliability conclude, in numbers, about “A Five-Step Check and a Pre-Registered Offline Evaluation”?

Answer: Chapter 8 gives a five-step check and a pre-registered offline evaluation, and shows as it fell a wrong conclusion reached through bad luck, with its error rate (0.045).

Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.045

A. n queries   true +0   true +3   true +6   (share 'supported')
       300      0.026     0.275     0.782
      1000      0.008     0.502     0.995
      2000      0.000     0.508     0.999
      3000      0.000     0.513     1.000
B. one query set, n = 2000, true difference +3 points:
   dense-only hits 136, hybrid-only hits 158, difference +0.0110
   EQUIVALENT: no difference of practical size
   verdicts over 2,000 such query sets:
     SUPPORTED            0.517
     BETTER, BELOW SESOI  0.426
     EQUIVALENT           0.045
     INCONCLUSIVE         0.011
Reading: a true 3-point gain is declared 'supported' only
about half the time even at 2,000 queries, because the rule
also demands that the estimate reach the SESOI; a 6-point gain
is caught reliably from about 1,000. The single query set in B
drew low and was called 'equivalent' - the kind of error the
equivalence check allows about one time in twenty when the
truth sits exactly at the SESOI. The rule names every outcome
in advance, including the wrong-looking ones.
Show the full program
# Chapter 8. p16: A pre-registered offline test: hybrid versus dense.
# Question: on one's own corpus, does rank-fused hybrid retrieval (top-100
# lists, BM25 zero scores dropped, k = 60) beat dense retrieval alone?
# Query set: drawn before the test from logged queries by a stratified
# rule (a fixed share containing identifiers, codes or rare terms),
# gold chunks labelled blind to either system's output. Primary estimand:
# paired difference in recall@10, hybrid minus dense. Smallest effect of
# interest (SESOI): 3 points. Test: McNemar on discordant queries,
# two-sided 5 %, plus an equivalence check (both 90 % bounds within
# +-3 points). The model below assumes 15 % discordant queries.
# A: power at true differences of 0, 3 and 6 points. B: the decision
# rule applied to one simulated query set. 2,000 runs per cell.
import math
import random

def sample(rng, n, delta, disc=0.15):
    up = (disc + delta) / 2              # hybrid-only hits
    dn = (disc - delta) / 2              # dense-only hits
    ba = ab = 0
    for _ in range(n):
        u = rng.random()
        if u < up:
            ba += 1
        elif u < up + dn:
            ab += 1
    return ab, ba

def decide(n, ab, ba, sesoi=0.03):
    d = (ba - ab) / n
    se = math.sqrt(ab + ba - (ba - ab) ** 2 / n) / n if ab + ba else 0.0
    lo90, hi90 = d - 1.645 * se, d + 1.645 * se
    lo95 = d - 1.96 * se
    if lo95 > 0 and d >= sesoi:
        return "SUPPORTED: hybrid better by at least the SESOI"
    if lo95 > 0:
        return "BETTER, BELOW SESOI: real but smaller than 3 points"
    if lo90 > -sesoi and hi90 < sesoi:
        return "EQUIVALENT: no difference of practical size"
    if d + 1.96 * se < 0:
        return "REVERSED: dense alone is better"
    return "INCONCLUSIVE: interval too wide"

if __name__ == "__main__":
    rng = random.Random(61016)
    print("A. n queries   true +0   true +3   true +6   (share 'supported')")
    for n in (300, 1000, 2000, 3000):
        row = []
        for delta in (0.0, 0.03, 0.06):
            k = sum(decide(n, *sample(rng, n, delta)).startswith("SUPP")
                    for _ in range(2000))
            row.append(k / 2000)
        print("   {:7d}      {:.3f}     {:.3f}     {:.3f}".format(n, *row))
    ab, ba = sample(rng, 2000, 0.03)
    d = (ba - ab) / 2000
    print("\nB. one query set, n = 2000, true difference +3 points:")
    print("   dense-only hits {}, hybrid-only hits {}, difference {:+.4f}"
          .format(ab, ba, d))
    print("   " + decide(2000, ab, ba))
    tally = {}
    for _ in range(2000):
        v = decide(2000, *sample(rng, 2000, 0.03)).split(":")[0]
        tally[v] = tally.get(v, 0) + 1
    print("   verdicts over 2,000 such query sets:")
    for v in sorted(tally, key=lambda x: -tally[x]):
        print("     {:<20} {:.3f}".format(v, tally[v] / 2000))
    print("\nReading: a true 3-point gain is declared 'supported' only")
    print("about half the time even at 2,000 queries, because the rule")
    print("also demands that the estimate reach the SESOI; a 6-point gain")
    print("is caught reliably from about 1,000. The single query set in B")
    print("drew low and was called 'equivalent' - the kind of error the")
    print("equivalence check allows about one time in twenty when the")
    print("truth sits exactly at the SESOI. The rule names every outcome")
    print("in advance, including the wrong-looking ones.")

Single source and citation

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 8: A Five-Step Check and a Pre-Registered Offline Evaluation. 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

Other chapter answers in this book