5D Library › Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades · 中文

Chapter 5. Verifier Independence and the Cost of Retries

Two hidden assumptions in Chapter 4

Chapter 4’s handsome result — a ten-step chain lifted from 43 % to above 99 % — rests on two unstated assumptions. First, a retry is a fresh draw: a failed attempt does not change the next one’s chance of success. Second, the verifier’s errors are unrelated to the generator’s. This chapter relaxes both and counts one more cost of reliability: tail latency.

Hard steps stay hard

Calculation

# Chapter 5. p10: When the verifier shares the generator's blind spots.
# Ten-step chain, up to 3 attempts per step. Each step is either easy or
# hard, and a retry faces the same step, so a hard step stays hard:
#   easy step  generator right with 0.95
#   hard step  generator right with 0.40
# The share of hard steps h is chosen so a single attempt is right with
# 0.92 overall: h = 0.03 / 0.55. Two verifiers:
#   code check  exact on every step (s = 1, f = 0)
#   LLM judge   on easy steps catches 0.95 of errors, false alarms 0.05;
#               on hard steps it shares the generator's blind spot and
#               catches only c_hard of errors
# 100,000 simulated chains for each setting.
import random

H = 0.03 / 0.55

def run(rng, judge, c_hard, chains=100000):
    ok_all = esc = 0
    for _ in range(chains):
        ok = True
        stopped = False
        for _ in range(10):
            hard = rng.random() < H
            p = 0.40 if hard else 0.95
            done = False
            for _ in range(3):
                right = rng.random() < p
                if judge == "code":
                    passed = right
                else:
                    s = c_hard if hard else 0.95
                    passed = (rng.random() >= 0.05) if right else \
                        (rng.random() >= s)
                if passed:
                    ok = ok and right
                    done = True
                    break
            if not done:
                stopped = True
                break
        esc += stopped
        ok_all += ok and not stopped
    return ok_all / chains, esc / chains

if __name__ == "__main__":
    rng = random.Random(61010)
    print("share of hard steps h = {:.4f}; one attempt right with {:.4f}"
          .format(H, H * 0.40 + (1 - H) * 0.95))
    print("\n   verifier     catches on hard steps   success   escalated")
    s, e = run(rng, "code", 1.0)
    print("   code check            1.00              {:.4f}    {:.4f}"
          .format(s, e))
    for c in (0.95, 0.6, 0.3, 0.1):
        s, e = run(rng, "judge", c)
        print("   LLM judge             {:.2f}              {:.4f}    {:.4f}"
              .format(c, s, e))
    print("\nReading: retries assume each attempt is a fresh draw. Here a")
    print("hard step stays hard, so even an exact check reaches only")
    print("0.886 (p09's independent model gave 0.9949): what it cannot")
    print("fix it escalates instead of letting through. A judge that is as")
    print("good on hard steps as on easy ones comes close to it. One that")
    print("shares the generator's blind spot lets most hard-step errors")
    print("pass - fewer escalations, more silent failures. A verifier is")
    print("worth most where it is independent of the generator's")
    print("weakness.")
# share of hard steps h = 0.0545; one attempt right with 0.9200
#
#    verifier     catches on hard steps   success   escalated
#    code check            1.00              0.8860    0.1140
#    LLM judge             0.95              0.8342    0.1153
#    LLM judge             0.60              0.7656    0.0379
#    LLM judge             0.30              0.7200    0.0129
#    LLM judge             0.10              0.6950    0.0083
#
# Reading: retries assume each attempt is a fresh draw. Here a
# hard step stays hard, so even an exact check reaches only
# 0.886 (p09's independent model gave 0.9949): what it cannot
# fix it escalates instead of letting through. A judge that is as
# good on hard steps as on easy ones comes close to it. One that
# shares the generator's blind spot lets most hard-step errors
# pass - fewer escalations, more silent failures. A verifier is
# worth most where it is independent of the generator's
# weakness.

Here every step is either easy (right 0.95 of the time per attempt) or hard (right 0.40), with hard steps making up 0.0545 of the total, so that a single attempt is still right 0.9200 of the time, as in Chapter 4. The difference: a retry faces the same step, and a hard step stays hard.

Even with a deterministic code check, success is only 0.8860, against 0.9949 from Chapter 4’s independent model: the chance of failing a hard step three times running is not small, and the code check escalates those runs (0.1140) rather than letting them through. A language-model judge as sharp on hard steps as on easy ones (0.95) reaches 0.8342, close to the code check. But if it shares the generator’s blind spot — catching only a tenth of errors on hard steps — success falls to 0.6950: fewer escalations (0.0083), more silent failures.

This is the second correction to the outline: a verifier’s value lies not in how accurate it is on average but in whether it is independent where the generator is weakest. Using one model both to generate and to judge is exactly where such shared blind spots are most likely (Zheng et al. 2023). Nor is a retry a free independent draw: for truly hard steps, changing the method (another model, another prompt, a person) does more than repeating the same one.

Retry budget and tail latency

Calculation

# Chapter 5. p11: Retry budget, cost and tail latency.
# Ten-step chain with an exact checker (as in p09), single attempts right
# with p. Up to n attempts per step; each attempt costs one unit (tokens
# or seconds). Exact distribution of total attempts by convolution of
# the per-step truncated geometric law; the run escalates if any step
# uses up its n attempts. Reported: success, mean attempts, and the 95th
# and 99th percentiles of attempts among successful runs.
def step_law(p, n):
    # P(step passes on attempt j), j = 1..n
    return [p * (1 - p) ** (j - 1) for j in range(1, n + 1)]

def chain_law(p, n, steps=10):
    dist = {0: 1.0}
    law = step_law(p, n)
    for _ in range(steps):
        nxt = {}
        for tot, pr in dist.items():
            for j, q in enumerate(law, 1):
                nxt[tot + j] = nxt.get(tot + j, 0.0) + pr * q
        dist = nxt
    return dist

def pct(dist, q):
    tot = sum(dist.values())
    acc = 0.0
    for k in sorted(dist):
        acc += dist[k] / tot
        if acc >= q:
            return k
    return max(dist)

if __name__ == "__main__":
    print("     p    n   success   mean attempts   p95   p99")
    for p in (0.92, 0.80, 0.60):
        for n in (1, 2, 3, 5):
            d = chain_law(p, n)
            succ = sum(d.values())
            mean = sum(k * v for k, v in d.items()) / succ
            print("   {:.2f}  {:2d}   {:.4f}      {:6.2f}         {:3d}"
                  "   {:3d}".format(p, n, succ, mean, pct(d, 0.95),
                                    pct(d, 0.99)))
    print("\nReading: with a strong generator the second and third")
    print("attempts buy almost all the reliability for little extra cost")
    print("or tail. With a weak one, each extra allowed attempt still")
    print("raises success substantially, but the 99th-percentile run")
    print("grows by several attempts - the price of reliability is paid")
    print("in the tail, and a latency budget should be set on it.")
#      p    n   success   mean attempts   p95   p99
#    0.92   1   0.4344       10.00          10    10
#    0.92   2   0.9378       10.74          12    13
#    0.92   3   0.9949       10.85          13    14
#    0.92   5   1.0000       10.87          13    14
#    0.80   1   0.1074       10.00          10    10
#    0.80   2   0.6648       11.67          14    15
#    0.80   3   0.9228       12.26          15    16
#    0.80   5   0.9968       12.48          16    17
#    0.60   1   0.0060       10.00          10    10
#    0.60   2   0.1749       12.86          15    16
#    0.60   3   0.5161       14.62          18    20
#    0.60   5   0.9022       16.15          21    24
#
# Reading: with a strong generator the second and third
# attempts buy almost all the reliability for little extra cost
# or tail. With a weak one, each extra allowed attempt still
# raises success substantially, but the 99th-percentile run
# grows by several attempts - the price of reliability is paid
# in the tail, and a latency budget should be set on it.

With a deterministic check, the trade between reliability and cost depends on how strong the generator is. At 0.92 a step, allowing two attempts per step lifts success from 0.4344 to 0.9378, and three to 0.9949; mean attempts barely move (10.00 to 10.85) and the 99th percentile goes from 10 to 14. At only 0.60 a step, each extra allowed attempt still raises success a great deal (two: 0.1749; three: 0.5161; five: 0.9022), but mean attempts rise from 10.00 to 16.15 and the 99th percentile to 24.

The price of reliability is paid mostly in the tail: most runs finish quickly, a few retry again and again. For systems with response-time guarantees, the retry cap should be set by tail latency (the 99th percentile, say), not by mean cost.

Success of a ten-step chain and the 99th-percentile number of attempts when each step allows up to n attempts, for three per-step accuracies.

Note

Where these numbers apply. This chapter’s easy/hard split and fixed per-step accuracies are simplifications. In real tasks difficulty is continuous and depends on the input; when measuring one’s own system, record first-attempt accuracy, accuracy after retries and the verifier’s detection rate by step and input type, not as a single average.

Hypothesis · not established

The author’s hypothesis (not yet tested): in agent systems that use one language model both to generate and to judge, the judge’s detection rate on “steps where the generator failed on its first attempt” is clearly lower than its detection rate on randomly injected errors. What would show it wrong: on a pre-registered error set, the 95 % confidence interval for the difference between the two detection rates contains zero or is negative.

Summary

Retries assume each attempt is a fresh draw, but hard steps stay hard: even with an exact check a ten-step chain reaches only 0.886, with escalation as the backstop. A judge that shares the generator’s blind spot lets the hardest errors through — fewer escalations, more silent failures. Reliability is paid for in the tail: the weaker the generator, the more retries the same success needs and the longer the 99th-percentile latency.

References

Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, et al. 2023. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” Advances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track.

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).