What does RAG and Multi-Agent Reliability conclude, in numbers, about “Verifier Independence and the Cost of Retries”?

Answer: Chapter 5 shows that a verifier's value lies in independence: if hard steps stay hard even a deterministic check reaches only 0.886, and a judge sharing the generator's blind spot only 0.6950; reliability is paid for in tail latency.

Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.6950, 0.886

share of hard steps h = 0.0545; one attempt right with 0.9200
   verifier     catches on hard steps   success   escalated
   code check            1.00              0.8860    0.1140
   LLM judge             0.95              0.8342    0.1153
   LLM judge             0.60              0.7656    0.0379
   LLM judge             0.30              0.7200    0.0129
   LLM judge             0.10              0.6950    0.0083
Reading: retries assume each attempt is a fresh draw. Here a
hard step stays hard, so even an exact check reaches only
0.886 (p09's independent model gave 0.9949): what it cannot
fix it escalates instead of letting through. A judge that is as
good on hard steps as on easy ones comes close to it. One that
shares the generator's blind spot lets most hard-step errors
pass - fewer escalations, more silent failures. A verifier is
worth most where it is independent of the generator's
weakness.
Show the full program
# Chapter 5. p10: When the verifier shares the generator's blind spots.
# Ten-step chain, up to 3 attempts per step. Each step is either easy or
# hard, and a retry faces the same step, so a hard step stays hard:
#   easy step  generator right with 0.95
#   hard step  generator right with 0.40
# The share of hard steps h is chosen so a single attempt is right with
# 0.92 overall: h = 0.03 / 0.55. Two verifiers:
#   code check  exact on every step (s = 1, f = 0)
#   LLM judge   on easy steps catches 0.95 of errors, false alarms 0.05;
#               on hard steps it shares the generator's blind spot and
#               catches only c_hard of errors
# 100,000 simulated chains for each setting.
import random

H = 0.03 / 0.55

def run(rng, judge, c_hard, chains=100000):
    ok_all = esc = 0
    for _ in range(chains):
        ok = True
        stopped = False
        for _ in range(10):
            hard = rng.random() < H
            p = 0.40 if hard else 0.95
            done = False
            for _ in range(3):
                right = rng.random() < p
                if judge == "code":
                    passed = right
                else:
                    s = c_hard if hard else 0.95
                    passed = (rng.random() >= 0.05) if right else \
                        (rng.random() >= s)
                if passed:
                    ok = ok and right
                    done = True
                    break
            if not done:
                stopped = True
                break
        esc += stopped
        ok_all += ok and not stopped
    return ok_all / chains, esc / chains

if __name__ == "__main__":
    rng = random.Random(61010)
    print("share of hard steps h = {:.4f}; one attempt right with {:.4f}"
          .format(H, H * 0.40 + (1 - H) * 0.95))
    print("\n   verifier     catches on hard steps   success   escalated")
    s, e = run(rng, "code", 1.0)
    print("   code check            1.00              {:.4f}    {:.4f}"
          .format(s, e))
    for c in (0.95, 0.6, 0.3, 0.1):
        s, e = run(rng, "judge", c)
        print("   LLM judge             {:.2f}              {:.4f}    {:.4f}"
              .format(c, s, e))
    print("\nReading: retries assume each attempt is a fresh draw. Here a")
    print("hard step stays hard, so even an exact check reaches only")
    print("0.886 (p09's independent model gave 0.9949): what it cannot")
    print("fix it escalates instead of letting through. A judge that is as")
    print("good on hard steps as on easy ones comes close to it. One that")
    print("shares the generator's blind spot lets most hard-step errors")
    print("pass - fewer escalations, more silent failures. A verifier is")
    print("worth most where it is independent of the generator's")
    print("weakness.")

Single source and citation

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 5: Verifier Independence and the Cost of Retries. 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

Other chapter answers in this book