5D Library › Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades · 中文
Chapter 5. Verifier Independence and the Cost of Retries
Hard steps stay hard
Calculation
# Chapter 5. p10: When the verifier shares the generator's blind spots.
# Ten-step chain, up to 3 attempts per step. Each step is either easy or
# hard, and a retry faces the same step, so a hard step stays hard:
# easy step generator right with 0.95
# hard step generator right with 0.40
# The share of hard steps h is chosen so a single attempt is right with
# 0.92 overall: h = 0.03 / 0.55. Two verifiers:
# code check exact on every step (s = 1, f = 0)
# LLM judge on easy steps catches 0.95 of errors, false alarms 0.05;
# on hard steps it shares the generator's blind spot and
# catches only c_hard of errors
# 100,000 simulated chains for each setting.
import random
H = 0.03 / 0.55
def run(rng, judge, c_hard, chains=100000):
ok_all = esc = 0
for _ in range(chains):
ok = True
stopped = False
for _ in range(10):
hard = rng.random() < H
p = 0.40 if hard else 0.95
done = False
for _ in range(3):
right = rng.random() < p
if judge == "code":
passed = right
else:
s = c_hard if hard else 0.95
passed = (rng.random() >= 0.05) if right else \
(rng.random() >= s)
if passed:
ok = ok and right
done = True
break
if not done:
stopped = True
break
esc += stopped
ok_all += ok and not stopped
return ok_all / chains, esc / chains
if __name__ == "__main__":
rng = random.Random(61010)
print("share of hard steps h = {:.4f}; one attempt right with {:.4f}"
.format(H, H * 0.40 + (1 - H) * 0.95))
print("\n verifier catches on hard steps success escalated")
s, e = run(rng, "code", 1.0)
print(" code check 1.00 {:.4f} {:.4f}"
.format(s, e))
for c in (0.95, 0.6, 0.3, 0.1):
s, e = run(rng, "judge", c)
print(" LLM judge {:.2f} {:.4f} {:.4f}"
.format(c, s, e))
print("\nReading: retries assume each attempt is a fresh draw. Here a")
print("hard step stays hard, so even an exact check reaches only")
print("0.886 (p09's independent model gave 0.9949): what it cannot")
print("fix it escalates instead of letting through. A judge that is as")
print("good on hard steps as on easy ones comes close to it. One that")
print("shares the generator's blind spot lets most hard-step errors")
print("pass - fewer escalations, more silent failures. A verifier is")
print("worth most where it is independent of the generator's")
print("weakness.")
# share of hard steps h = 0.0545; one attempt right with 0.9200
#
# verifier catches on hard steps success escalated
# code check 1.00 0.8860 0.1140
# LLM judge 0.95 0.8342 0.1153
# LLM judge 0.60 0.7656 0.0379
# LLM judge 0.30 0.7200 0.0129
# LLM judge 0.10 0.6950 0.0083
#
# Reading: retries assume each attempt is a fresh draw. Here a
# hard step stays hard, so even an exact check reaches only
# 0.886 (p09's independent model gave 0.9949): what it cannot
# fix it escalates instead of letting through. A judge that is as
# good on hard steps as on easy ones comes close to it. One that
# shares the generator's blind spot lets most hard-step errors
# pass - fewer escalations, more silent failures. A verifier is
# worth most where it is independent of the generator's
# weakness.Here every step is either easy (right 0.95 of the time per attempt) or hard (right 0.40), with hard steps making up 0.0545 of the total, so that a single attempt is still right 0.9200 of the time, as in Chapter 4. The difference: a retry faces the same step, and a hard step stays hard.
Even with a deterministic code check, success is only 0.8860, against 0.9949 from Chapter 4’s independent model: the chance of failing a hard step three times running is not small, and the code check escalates those runs (0.1140) rather than letting them through. A language-model judge as sharp on hard steps as on easy ones (0.95) reaches 0.8342, close to the code check. But if it shares the generator’s blind spot — catching only a tenth of errors on hard steps — success falls to 0.6950: fewer escalations (0.0083), more silent failures.
This is the second correction to the outline: a verifier’s value lies not in how accurate it is on average but in whether it is independent where the generator is weakest. Using one model both to generate and to judge is exactly where such shared blind spots are most likely (Zheng et al. 2023). Nor is a retry a free independent draw: for truly hard steps, changing the method (another model, another prompt, a person) does more than repeating the same one.
Retry budget and tail latency
Calculation
# Chapter 5. p11: Retry budget, cost and tail latency.
# Ten-step chain with an exact checker (as in p09), single attempts right
# with p. Up to n attempts per step; each attempt costs one unit (tokens
# or seconds). Exact distribution of total attempts by convolution of
# the per-step truncated geometric law; the run escalates if any step
# uses up its n attempts. Reported: success, mean attempts, and the 95th
# and 99th percentiles of attempts among successful runs.
def step_law(p, n):
# P(step passes on attempt j), j = 1..n
return [p * (1 - p) ** (j - 1) for j in range(1, n + 1)]
def chain_law(p, n, steps=10):
dist = {0: 1.0}
law = step_law(p, n)
for _ in range(steps):
nxt = {}
for tot, pr in dist.items():
for j, q in enumerate(law, 1):
nxt[tot + j] = nxt.get(tot + j, 0.0) + pr * q
dist = nxt
return dist
def pct(dist, q):
tot = sum(dist.values())
acc = 0.0
for k in sorted(dist):
acc += dist[k] / tot
if acc >= q:
return k
return max(dist)
if __name__ == "__main__":
print(" p n success mean attempts p95 p99")
for p in (0.92, 0.80, 0.60):
for n in (1, 2, 3, 5):
d = chain_law(p, n)
succ = sum(d.values())
mean = sum(k * v for k, v in d.items()) / succ
print(" {:.2f} {:2d} {:.4f} {:6.2f} {:3d}"
" {:3d}".format(p, n, succ, mean, pct(d, 0.95),
pct(d, 0.99)))
print("\nReading: with a strong generator the second and third")
print("attempts buy almost all the reliability for little extra cost")
print("or tail. With a weak one, each extra allowed attempt still")
print("raises success substantially, but the 99th-percentile run")
print("grows by several attempts - the price of reliability is paid")
print("in the tail, and a latency budget should be set on it.")
# p n success mean attempts p95 p99
# 0.92 1 0.4344 10.00 10 10
# 0.92 2 0.9378 10.74 12 13
# 0.92 3 0.9949 10.85 13 14
# 0.92 5 1.0000 10.87 13 14
# 0.80 1 0.1074 10.00 10 10
# 0.80 2 0.6648 11.67 14 15
# 0.80 3 0.9228 12.26 15 16
# 0.80 5 0.9968 12.48 16 17
# 0.60 1 0.0060 10.00 10 10
# 0.60 2 0.1749 12.86 15 16
# 0.60 3 0.5161 14.62 18 20
# 0.60 5 0.9022 16.15 21 24
#
# Reading: with a strong generator the second and third
# attempts buy almost all the reliability for little extra cost
# or tail. With a weak one, each extra allowed attempt still
# raises success substantially, but the 99th-percentile run
# grows by several attempts - the price of reliability is paid
# in the tail, and a latency budget should be set on it.With a deterministic check, the trade between reliability and cost depends on how strong the generator is. At 0.92 a step, allowing two attempts per step lifts success from 0.4344 to 0.9378, and three to 0.9949; mean attempts barely move (10.00 to 10.85) and the 99th percentile goes from 10 to 14. At only 0.60 a step, each extra allowed attempt still raises success a great deal (two: 0.1749; three: 0.5161; five: 0.9022), but mean attempts rise from 10.00 to 16.15 and the 99th percentile to 24.
The price of reliability is paid mostly in the tail: most runs finish quickly, a few retry again and again. For systems with response-time guarantees, the retry cap should be set by tail latency (the 99th percentile, say), not by mean cost.
Note
Where these numbers apply. This chapter’s easy/hard split and fixed per-step accuracies are simplifications. In real tasks difficulty is continuous and depends on the input; when measuring one’s own system, record first-attempt accuracy, accuracy after retries and the verifier’s detection rate by step and input type, not as a single average.
Hypothesis · not established
The author’s hypothesis (not yet tested): in agent systems that use one language model both to generate and to judge, the judge’s detection rate on “steps where the generator failed on its first attempt” is clearly lower than its detection rate on randomly injected errors. What would show it wrong: on a pre-registered error set, the 95 % confidence interval for the difference between the two detection rates contains zero or is negative.
Summary
Retries assume each attempt is a fresh draw, but hard steps stay hard: even with an exact check a ten-step chain reaches only 0.886, with escalation as the backstop. A judge that shares the generator’s blind spot lets the hardest errors through — fewer escalations, more silent failures. Reliability is paid for in the tail: the weaker the generator, the more retries the same success needs and the longer the 99th-percentile latency.
References
Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).