关于「验证器的独立性与重试的代价」,《检索增强与多智能体的可靠性》给出的定量结论是什么?

结论:第五章说明验证器的价值在于独立:困难步骤一直困难时,即使确定性检查也只到 0.886,与生成器共享盲区的评判只到 0.6950;可靠性的代价付在尾部延迟上。

数据口径:全部数据为程序生成的合成数据,数字说明机制与量级,不是对任何产品的测量。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.6950, 0.886

share of hard steps h = 0.0545; one attempt right with 0.9200
   verifier     catches on hard steps   success   escalated
   code check            1.00              0.8860    0.1140
   LLM judge             0.95              0.8342    0.1153
   LLM judge             0.60              0.7656    0.0379
   LLM judge             0.30              0.7200    0.0129
   LLM judge             0.10              0.6950    0.0083
Reading: retries assume each attempt is a fresh draw. Here a
hard step stays hard, so even an exact check reaches only
0.886 (p09's independent model gave 0.9949): what it cannot
fix it escalates instead of letting through. A judge that is as
good on hard steps as on easy ones comes close to it. One that
shares the generator's blind spot lets most hard-step errors
pass - fewer escalations, more silent failures. A verifier is
worth most where it is independent of the generator's
weakness.
查看完整验算程序
# Chapter 5. p10: When the verifier shares the generator's blind spots.
# Ten-step chain, up to 3 attempts per step. Each step is either easy or
# hard, and a retry faces the same step, so a hard step stays hard:
#   easy step  generator right with 0.95
#   hard step  generator right with 0.40
# The share of hard steps h is chosen so a single attempt is right with
# 0.92 overall: h = 0.03 / 0.55. Two verifiers:
#   code check  exact on every step (s = 1, f = 0)
#   LLM judge   on easy steps catches 0.95 of errors, false alarms 0.05;
#               on hard steps it shares the generator's blind spot and
#               catches only c_hard of errors
# 100,000 simulated chains for each setting.
import random

H = 0.03 / 0.55

def run(rng, judge, c_hard, chains=100000):
    ok_all = esc = 0
    for _ in range(chains):
        ok = True
        stopped = False
        for _ in range(10):
            hard = rng.random() < H
            p = 0.40 if hard else 0.95
            done = False
            for _ in range(3):
                right = rng.random() < p
                if judge == "code":
                    passed = right
                else:
                    s = c_hard if hard else 0.95
                    passed = (rng.random() >= 0.05) if right else \
                        (rng.random() >= s)
                if passed:
                    ok = ok and right
                    done = True
                    break
            if not done:
                stopped = True
                break
        esc += stopped
        ok_all += ok and not stopped
    return ok_all / chains, esc / chains

if __name__ == "__main__":
    rng = random.Random(61010)
    print("share of hard steps h = {:.4f}; one attempt right with {:.4f}"
          .format(H, H * 0.40 + (1 - H) * 0.95))
    print("\n   verifier     catches on hard steps   success   escalated")
    s, e = run(rng, "code", 1.0)
    print("   code check            1.00              {:.4f}    {:.4f}"
          .format(s, e))
    for c in (0.95, 0.6, 0.3, 0.1):
        s, e = run(rng, "judge", c)
        print("   LLM judge             {:.2f}              {:.4f}    {:.4f}"
              .format(c, s, e))
    print("\nReading: retries assume each attempt is a fresh draw. Here a")
    print("hard step stays hard, so even an exact check reaches only")
    print("0.886 (p09's independent model gave 0.9949): what it cannot")
    print("fix it escalates instead of letting through. A judge that is as")
    print("good on hard steps as on easy ones comes close to it. One that")
    print("shares the generator's blind spot lets most hard-step errors")
    print("pass - fewer escalations, more silent failures. A verifier is")
    print("worth most where it is independent of the generator's")
    print("weakness.")

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《检索增强生成几何、倒数秩融合与多智能体误差级联》第 5 章「验证器的独立性与重试的代价」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · 阅读本章所在专著 · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

本书其他章节结论