关于「验证器的独立性与重试的代价」,《检索增强与多智能体的可靠性》给出的定量结论是什么?
结论:第五章说明验证器的价值在于独立:困难步骤一直困难时,即使确定性检查也只到 0.886,与生成器共享盲区的评判只到 0.6950;可靠性的代价付在尾部延迟上。
数据口径:全部数据为程序生成的合成数据,数字说明机制与量级,不是对任何产品的测量。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.6950, 0.886
share of hard steps h = 0.0545; one attempt right with 0.9200 verifier catches on hard steps success escalated code check 1.00 0.8860 0.1140 LLM judge 0.95 0.8342 0.1153 LLM judge 0.60 0.7656 0.0379 LLM judge 0.30 0.7200 0.0129 LLM judge 0.10 0.6950 0.0083 Reading: retries assume each attempt is a fresh draw. Here a hard step stays hard, so even an exact check reaches only 0.886 (p09's independent model gave 0.9949): what it cannot fix it escalates instead of letting through. A judge that is as good on hard steps as on easy ones comes close to it. One that shares the generator's blind spot lets most hard-step errors pass - fewer escalations, more silent failures. A verifier is worth most where it is independent of the generator's weakness.
查看完整验算程序
# Chapter 5. p10: When the verifier shares the generator's blind spots.
# Ten-step chain, up to 3 attempts per step. Each step is either easy or
# hard, and a retry faces the same step, so a hard step stays hard:
# easy step generator right with 0.95
# hard step generator right with 0.40
# The share of hard steps h is chosen so a single attempt is right with
# 0.92 overall: h = 0.03 / 0.55. Two verifiers:
# code check exact on every step (s = 1, f = 0)
# LLM judge on easy steps catches 0.95 of errors, false alarms 0.05;
# on hard steps it shares the generator's blind spot and
# catches only c_hard of errors
# 100,000 simulated chains for each setting.
import random
H = 0.03 / 0.55
def run(rng, judge, c_hard, chains=100000):
ok_all = esc = 0
for _ in range(chains):
ok = True
stopped = False
for _ in range(10):
hard = rng.random() < H
p = 0.40 if hard else 0.95
done = False
for _ in range(3):
right = rng.random() < p
if judge == "code":
passed = right
else:
s = c_hard if hard else 0.95
passed = (rng.random() >= 0.05) if right else \
(rng.random() >= s)
if passed:
ok = ok and right
done = True
break
if not done:
stopped = True
break
esc += stopped
ok_all += ok and not stopped
return ok_all / chains, esc / chains
if __name__ == "__main__":
rng = random.Random(61010)
print("share of hard steps h = {:.4f}; one attempt right with {:.4f}"
.format(H, H * 0.40 + (1 - H) * 0.95))
print("\n verifier catches on hard steps success escalated")
s, e = run(rng, "code", 1.0)
print(" code check 1.00 {:.4f} {:.4f}"
.format(s, e))
for c in (0.95, 0.6, 0.3, 0.1):
s, e = run(rng, "judge", c)
print(" LLM judge {:.2f} {:.4f} {:.4f}"
.format(c, s, e))
print("\nReading: retries assume each attempt is a fresh draw. Here a")
print("hard step stays hard, so even an exact check reaches only")
print("0.886 (p09's independent model gave 0.9949): what it cannot")
print("fix it escalates instead of letting through. A judge that is as")
print("good on hard steps as on easy ones comes close to it. One that")
print("shares the generator's blind spot lets most hard-step errors")
print("pass - fewer escalations, more silent failures. A verifier is")
print("worth most where it is independent of the generator's")
print("weakness.")唯一数据源与引用
范雅福(Fan, Y.). (2026). 《检索增强生成几何、倒数秩融合与多智能体误差级联》第 5 章「验证器的独立性与重试的代价」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170
DOI: 10.5281/zenodo.24090170 · 阅读本章所在专著 · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026061,
author = {Fan, Yafu},
title = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090170},
url = {https://doi.org/10.5281/zenodo.24090170}
}