What does RAG and Multi-Agent Reliability conclude, in numbers, about “Verifier Independence and the Cost of Retries”?
Answer: Chapter 5 shows that a verifier's value lies in independence: if hard steps stay hard even a deterministic check reaches only 0.886, and a judge sharing the generator's blind spot only 0.6950; reliability is paid for in tail latency.
Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.6950, 0.886
share of hard steps h = 0.0545; one attempt right with 0.9200 verifier catches on hard steps success escalated code check 1.00 0.8860 0.1140 LLM judge 0.95 0.8342 0.1153 LLM judge 0.60 0.7656 0.0379 LLM judge 0.30 0.7200 0.0129 LLM judge 0.10 0.6950 0.0083 Reading: retries assume each attempt is a fresh draw. Here a hard step stays hard, so even an exact check reaches only 0.886 (p09's independent model gave 0.9949): what it cannot fix it escalates instead of letting through. A judge that is as good on hard steps as on easy ones comes close to it. One that shares the generator's blind spot lets most hard-step errors pass - fewer escalations, more silent failures. A verifier is worth most where it is independent of the generator's weakness.
Show the full program
# Chapter 5. p10: When the verifier shares the generator's blind spots.
# Ten-step chain, up to 3 attempts per step. Each step is either easy or
# hard, and a retry faces the same step, so a hard step stays hard:
# easy step generator right with 0.95
# hard step generator right with 0.40
# The share of hard steps h is chosen so a single attempt is right with
# 0.92 overall: h = 0.03 / 0.55. Two verifiers:
# code check exact on every step (s = 1, f = 0)
# LLM judge on easy steps catches 0.95 of errors, false alarms 0.05;
# on hard steps it shares the generator's blind spot and
# catches only c_hard of errors
# 100,000 simulated chains for each setting.
import random
H = 0.03 / 0.55
def run(rng, judge, c_hard, chains=100000):
ok_all = esc = 0
for _ in range(chains):
ok = True
stopped = False
for _ in range(10):
hard = rng.random() < H
p = 0.40 if hard else 0.95
done = False
for _ in range(3):
right = rng.random() < p
if judge == "code":
passed = right
else:
s = c_hard if hard else 0.95
passed = (rng.random() >= 0.05) if right else \
(rng.random() >= s)
if passed:
ok = ok and right
done = True
break
if not done:
stopped = True
break
esc += stopped
ok_all += ok and not stopped
return ok_all / chains, esc / chains
if __name__ == "__main__":
rng = random.Random(61010)
print("share of hard steps h = {:.4f}; one attempt right with {:.4f}"
.format(H, H * 0.40 + (1 - H) * 0.95))
print("\n verifier catches on hard steps success escalated")
s, e = run(rng, "code", 1.0)
print(" code check 1.00 {:.4f} {:.4f}"
.format(s, e))
for c in (0.95, 0.6, 0.3, 0.1):
s, e = run(rng, "judge", c)
print(" LLM judge {:.2f} {:.4f} {:.4f}"
.format(c, s, e))
print("\nReading: retries assume each attempt is a fresh draw. Here a")
print("hard step stays hard, so even an exact check reaches only")
print("0.886 (p09's independent model gave 0.9949): what it cannot")
print("fix it escalates instead of letting through. A judge that is as")
print("good on hard steps as on easy ones comes close to it. One that")
print("shares the generator's blind spot lets most hard-step errors")
print("pass - fewer escalations, more silent failures. A verifier is")
print("worth most where it is independent of the generator's")
print("weakness.")Single source and citation
Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 5: Verifier Independence and the Cost of Retries. 5D Library. https://doi.org/10.5281/zenodo.24090170
DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026061,
author = {Fan, Yafu},
title = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090170},
url = {https://doi.org/10.5281/zenodo.24090170}
}Other chapter answers in this book
- Chapter 1 · High-Dimensional Geometry: Concentration of Measure, Effective Dimension and Hub Chunks
- Chapter 2 · Lexical and Dense Retrieval: Their Blind Spots and the Smoothing Constant of Reciprocal Rank Fusion
- Chapter 3 · The Context Window: Position Decay, Evidence Dilution and the Best Number of Chunks
- Chapter 4 · Multi-Agent Pipelines: Error Cascades and the Verifier-Gated Absorbing Markov Chain
- Chapter 6 · The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner's Curse
- Chapter 7 · The Routing of the *Qilue* and Han Fei's *Canyan*: Two Ancient Methods of Retrieval and Verification
- Chapter 8 · A Five-Step Check and a Pre-Registered Offline Evaluation