What does RAG and Multi-Agent Reliability conclude, in numbers, about “A Five-Step Check and a Pre-Registered Offline Evaluation”?
Answer: Chapter 8 gives a five-step check and a pre-registered offline evaluation, and shows as it fell a wrong conclusion reached through bad luck, with its error rate (0.045).
Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.045
A. n queries true +0 true +3 true +6 (share 'supported')
300 0.026 0.275 0.782
1000 0.008 0.502 0.995
2000 0.000 0.508 0.999
3000 0.000 0.513 1.000
B. one query set, n = 2000, true difference +3 points:
dense-only hits 136, hybrid-only hits 158, difference +0.0110
EQUIVALENT: no difference of practical size
verdicts over 2,000 such query sets:
SUPPORTED 0.517
BETTER, BELOW SESOI 0.426
EQUIVALENT 0.045
INCONCLUSIVE 0.011
Reading: a true 3-point gain is declared 'supported' only
about half the time even at 2,000 queries, because the rule
also demands that the estimate reach the SESOI; a 6-point gain
is caught reliably from about 1,000. The single query set in B
drew low and was called 'equivalent' - the kind of error the
equivalence check allows about one time in twenty when the
truth sits exactly at the SESOI. The rule names every outcome
in advance, including the wrong-looking ones.Show the full program
# Chapter 8. p16: A pre-registered offline test: hybrid versus dense.
# Question: on one's own corpus, does rank-fused hybrid retrieval (top-100
# lists, BM25 zero scores dropped, k = 60) beat dense retrieval alone?
# Query set: drawn before the test from logged queries by a stratified
# rule (a fixed share containing identifiers, codes or rare terms),
# gold chunks labelled blind to either system's output. Primary estimand:
# paired difference in recall@10, hybrid minus dense. Smallest effect of
# interest (SESOI): 3 points. Test: McNemar on discordant queries,
# two-sided 5 %, plus an equivalence check (both 90 % bounds within
# +-3 points). The model below assumes 15 % discordant queries.
# A: power at true differences of 0, 3 and 6 points. B: the decision
# rule applied to one simulated query set. 2,000 runs per cell.
import math
import random
def sample(rng, n, delta, disc=0.15):
up = (disc + delta) / 2 # hybrid-only hits
dn = (disc - delta) / 2 # dense-only hits
ba = ab = 0
for _ in range(n):
u = rng.random()
if u < up:
ba += 1
elif u < up + dn:
ab += 1
return ab, ba
def decide(n, ab, ba, sesoi=0.03):
d = (ba - ab) / n
se = math.sqrt(ab + ba - (ba - ab) ** 2 / n) / n if ab + ba else 0.0
lo90, hi90 = d - 1.645 * se, d + 1.645 * se
lo95 = d - 1.96 * se
if lo95 > 0 and d >= sesoi:
return "SUPPORTED: hybrid better by at least the SESOI"
if lo95 > 0:
return "BETTER, BELOW SESOI: real but smaller than 3 points"
if lo90 > -sesoi and hi90 < sesoi:
return "EQUIVALENT: no difference of practical size"
if d + 1.96 * se < 0:
return "REVERSED: dense alone is better"
return "INCONCLUSIVE: interval too wide"
if __name__ == "__main__":
rng = random.Random(61016)
print("A. n queries true +0 true +3 true +6 (share 'supported')")
for n in (300, 1000, 2000, 3000):
row = []
for delta in (0.0, 0.03, 0.06):
k = sum(decide(n, *sample(rng, n, delta)).startswith("SUPP")
for _ in range(2000))
row.append(k / 2000)
print(" {:7d} {:.3f} {:.3f} {:.3f}".format(n, *row))
ab, ba = sample(rng, 2000, 0.03)
d = (ba - ab) / 2000
print("\nB. one query set, n = 2000, true difference +3 points:")
print(" dense-only hits {}, hybrid-only hits {}, difference {:+.4f}"
.format(ab, ba, d))
print(" " + decide(2000, ab, ba))
tally = {}
for _ in range(2000):
v = decide(2000, *sample(rng, 2000, 0.03)).split(":")[0]
tally[v] = tally.get(v, 0) + 1
print(" verdicts over 2,000 such query sets:")
for v in sorted(tally, key=lambda x: -tally[x]):
print(" {:<20} {:.3f}".format(v, tally[v] / 2000))
print("\nReading: a true 3-point gain is declared 'supported' only")
print("about half the time even at 2,000 queries, because the rule")
print("also demands that the estimate reach the SESOI; a 6-point gain")
print("is caught reliably from about 1,000. The single query set in B")
print("drew low and was called 'equivalent' - the kind of error the")
print("equivalence check allows about one time in twenty when the")
print("truth sits exactly at the SESOI. The rule names every outcome")
print("in advance, including the wrong-looking ones.")Single source and citation
Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 8: A Five-Step Check and a Pre-Registered Offline Evaluation. 5D Library. https://doi.org/10.5281/zenodo.24090170
DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026061,
author = {Fan, Yafu},
title = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090170},
url = {https://doi.org/10.5281/zenodo.24090170}
}Other chapter answers in this book
- Chapter 1 · High-Dimensional Geometry: Concentration of Measure, Effective Dimension and Hub Chunks
- Chapter 2 · Lexical and Dense Retrieval: Their Blind Spots and the Smoothing Constant of Reciprocal Rank Fusion
- Chapter 3 · The Context Window: Position Decay, Evidence Dilution and the Best Number of Chunks
- Chapter 4 · Multi-Agent Pipelines: Error Cascades and the Verifier-Gated Absorbing Markov Chain
- Chapter 5 · Verifier Independence and the Cost of Retries
- Chapter 6 · The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner's Curse
- Chapter 7 · The Routing of the *Qilue* and Han Fei's *Canyan*: Two Ancient Methods of Retrieval and Verification