5D Library › 检索增强生成几何、倒数秩融合与多智能体误差级联 · English
第五章 验证器的独立性与重试的代价
第四章的两个隐含假设
第四章的漂亮结果——十步链从 43% 升到 99% 以上——依赖两个没有明说的假设。第一,重试是一次全新的抽样:上一次失败不影响下一次成功的机会。第二,验证器的错误与生成器的错误无关。本章放松这两个假设,并计算可靠性的另一项代价:尾部延迟。
难的步骤一直难
推导
# Chapter 5. p10: When the verifier shares the generator's blind spots.
# Ten-step chain, up to 3 attempts per step. Each step is either easy or
# hard, and a retry faces the same step, so a hard step stays hard:
# easy step generator right with 0.95
# hard step generator right with 0.40
# The share of hard steps h is chosen so a single attempt is right with
# 0.92 overall: h = 0.03 / 0.55. Two verifiers:
# code check exact on every step (s = 1, f = 0)
# LLM judge on easy steps catches 0.95 of errors, false alarms 0.05;
# on hard steps it shares the generator's blind spot and
# catches only c_hard of errors
# 100,000 simulated chains for each setting.
import random
H = 0.03 / 0.55
def run(rng, judge, c_hard, chains=100000):
ok_all = esc = 0
for _ in range(chains):
ok = True
stopped = False
for _ in range(10):
hard = rng.random() < H
p = 0.40 if hard else 0.95
done = False
for _ in range(3):
right = rng.random() < p
if judge == "code":
passed = right
else:
s = c_hard if hard else 0.95
passed = (rng.random() >= 0.05) if right else \
(rng.random() >= s)
if passed:
ok = ok and right
done = True
break
if not done:
stopped = True
break
esc += stopped
ok_all += ok and not stopped
return ok_all / chains, esc / chains
if __name__ == "__main__":
rng = random.Random(61010)
print("share of hard steps h = {:.4f}; one attempt right with {:.4f}"
.format(H, H * 0.40 + (1 - H) * 0.95))
print("\n verifier catches on hard steps success escalated")
s, e = run(rng, "code", 1.0)
print(" code check 1.00 {:.4f} {:.4f}"
.format(s, e))
for c in (0.95, 0.6, 0.3, 0.1):
s, e = run(rng, "judge", c)
print(" LLM judge {:.2f} {:.4f} {:.4f}"
.format(c, s, e))
print("\nReading: retries assume each attempt is a fresh draw. Here a")
print("hard step stays hard, so even an exact check reaches only")
print("0.886 (p09's independent model gave 0.9949): what it cannot")
print("fix it escalates instead of letting through. A judge that is as")
print("good on hard steps as on easy ones comes close to it. One that")
print("shares the generator's blind spot lets most hard-step errors")
print("pass - fewer escalations, more silent failures. A verifier is")
print("worth most where it is independent of the generator's")
print("weakness.")
# share of hard steps h = 0.0545; one attempt right with 0.9200
#
# verifier catches on hard steps success escalated
# code check 1.00 0.8860 0.1140
# LLM judge 0.95 0.8342 0.1153
# LLM judge 0.60 0.7656 0.0379
# LLM judge 0.30 0.7200 0.0129
# LLM judge 0.10 0.6950 0.0083
#
# Reading: retries assume each attempt is a fresh draw. Here a
# hard step stays hard, so even an exact check reaches only
# 0.886 (p09's independent model gave 0.9949): what it cannot
# fix it escalates instead of letting through. A judge that is as
# good on hard steps as on easy ones comes close to it. One that
# shares the generator's blind spot lets most hard-step errors
# pass - fewer escalations, more silent failures. A verifier is
# worth most where it is independent of the generator's
# weakness.这里每一步要么容易(单次正确 0.95),要么困难(单次正确 0.40),困难步骤的比例 0.0545 使得单次尝试的总正确率仍是 0.9200,与第四章相同。区别在于:重试面对的是同一个步骤,困难的步骤一直困难。
即使是确定性的代码检查,成功率也只有 0.8860,而第四章独立模型给出的是 0.9949:困难步骤三次都做不对的概率不小,代码检查把这些情形升级(0.1140),而不是放过。语言模型评判在困难步骤上若与容易步骤一样敏锐(0.95),成功率 0.8342,接近代码检查。但若它与生成器共享盲区——在困难步骤上只能抓住一成的错误——成功率降到 0.6950:升级少了(0.0083),静默失败多了。
这是第二处对提纲的修正:验证器的价值不在于它平均有多准,而在于它在生成器最弱的地方是否独立。用同一个模型既生成又评判,最容易出现的正是这种共享盲区 (Zheng et al. 2023)。重试也不是免费的独立抽样:对真正困难的步骤,更换方法(换模型、换提示、交给人)比重复同一方法更有效。
重试预算与尾部延迟
推导
# Chapter 5. p11: Retry budget, cost and tail latency.
# Ten-step chain with an exact checker (as in p09), single attempts right
# with p. Up to n attempts per step; each attempt costs one unit (tokens
# or seconds). Exact distribution of total attempts by convolution of
# the per-step truncated geometric law; the run escalates if any step
# uses up its n attempts. Reported: success, mean attempts, and the 95th
# and 99th percentiles of attempts among successful runs.
def step_law(p, n):
# P(step passes on attempt j), j = 1..n
return [p * (1 - p) ** (j - 1) for j in range(1, n + 1)]
def chain_law(p, n, steps=10):
dist = {0: 1.0}
law = step_law(p, n)
for _ in range(steps):
nxt = {}
for tot, pr in dist.items():
for j, q in enumerate(law, 1):
nxt[tot + j] = nxt.get(tot + j, 0.0) + pr * q
dist = nxt
return dist
def pct(dist, q):
tot = sum(dist.values())
acc = 0.0
for k in sorted(dist):
acc += dist[k] / tot
if acc >= q:
return k
return max(dist)
if __name__ == "__main__":
print(" p n success mean attempts p95 p99")
for p in (0.92, 0.80, 0.60):
for n in (1, 2, 3, 5):
d = chain_law(p, n)
succ = sum(d.values())
mean = sum(k * v for k, v in d.items()) / succ
print(" {:.2f} {:2d} {:.4f} {:6.2f} {:3d}"
" {:3d}".format(p, n, succ, mean, pct(d, 0.95),
pct(d, 0.99)))
print("\nReading: with a strong generator the second and third")
print("attempts buy almost all the reliability for little extra cost")
print("or tail. With a weak one, each extra allowed attempt still")
print("raises success substantially, but the 99th-percentile run")
print("grows by several attempts - the price of reliability is paid")
print("in the tail, and a latency budget should be set on it.")
# p n success mean attempts p95 p99
# 0.92 1 0.4344 10.00 10 10
# 0.92 2 0.9378 10.74 12 13
# 0.92 3 0.9949 10.85 13 14
# 0.92 5 1.0000 10.87 13 14
# 0.80 1 0.1074 10.00 10 10
# 0.80 2 0.6648 11.67 14 15
# 0.80 3 0.9228 12.26 15 16
# 0.80 5 0.9968 12.48 16 17
# 0.60 1 0.0060 10.00 10 10
# 0.60 2 0.1749 12.86 15 16
# 0.60 3 0.5161 14.62 18 20
# 0.60 5 0.9022 16.15 21 24
#
# Reading: with a strong generator the second and third
# attempts buy almost all the reliability for little extra cost
# or tail. With a weak one, each extra allowed attempt still
# raises success substantially, but the 99th-percentile run
# grows by several attempts - the price of reliability is paid
# in the tail, and a latency budget should be set on it.有确定性检查时,可靠性与成本的关系取决于生成器的强弱。单步 0.92 时,每步允许两次就把成功率从 0.4344 提到 0.9378,三次到 0.9949;平均尝试次数几乎不变(10.00 到 10.85),第 99 百分位从 10 次增加到 14 次。单步只有 0.60 时,每多允许一次仍大幅提高成功率(两次 0.1749,三次 0.5161,五次 0.9022),但平均尝试从 10.00 涨到 16.15,第 99 百分位涨到 24 次。
可靠性的代价主要付在尾部:多数运行很快,少数运行要反复重试。对需要响应时间保证的系统,重试上限应当按尾部延迟(例如第 99 百分位)来定,而不是按平均成本。
注
这些数字的适用范围。 本章的「容易/困难」二分与固定的单步正确率都是简化。真实任务中,难度是连续的,且与输入有关;测量自己的系统时,应当按步骤与输入类型分别记录首次正确率、重试后正确率与验证器的检出率,而不是只记一个平均数。
假说 · 尚未验证
作者的假说(尚未验证): 在用同一个语言模型既生成又评判的智能体系统中,评判在「生成器首次失败的步骤」上的检出率,显著低于它在随机注入错误上的检出率。什么结果会证明它错:在预先登记的错误集上,两种检出率之差的 95% 置信区间包含零或为负。
小结
重试默认每次都是新的抽样,但困难的步骤一直困难:即使有确定的检查,十步链也只能达到 0.886,余下的由升级兜底。与生成器共享盲区的评判放过最难的错误,升级变少、静默失败变多。可靠性的代价付在尾部:生成器越弱,换取同样成功率所需的重试越多,第 99 百分位的延迟越长。
参考文献
Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).