关于「五步核查与一项预注册的离线评测」,《检索增强与多智能体的可靠性》给出的定量结论是什么?

结论:第八章给出五步核查与一项预注册的离线评测,并如实展示一次「坏运气」得出的错误结论及其错误率(0.045)。

数据口径:全部数据为程序生成的合成数据,数字说明机制与量级,不是对任何产品的测量。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.045

A. n queries   true +0   true +3   true +6   (share 'supported')
       300      0.026     0.275     0.782
      1000      0.008     0.502     0.995
      2000      0.000     0.508     0.999
      3000      0.000     0.513     1.000
B. one query set, n = 2000, true difference +3 points:
   dense-only hits 136, hybrid-only hits 158, difference +0.0110
   EQUIVALENT: no difference of practical size
   verdicts over 2,000 such query sets:
     SUPPORTED            0.517
     BETTER, BELOW SESOI  0.426
     EQUIVALENT           0.045
     INCONCLUSIVE         0.011
Reading: a true 3-point gain is declared 'supported' only
about half the time even at 2,000 queries, because the rule
also demands that the estimate reach the SESOI; a 6-point gain
is caught reliably from about 1,000. The single query set in B
drew low and was called 'equivalent' - the kind of error the
equivalence check allows about one time in twenty when the
truth sits exactly at the SESOI. The rule names every outcome
in advance, including the wrong-looking ones.
查看完整验算程序
# Chapter 8. p16: A pre-registered offline test: hybrid versus dense.
# Question: on one's own corpus, does rank-fused hybrid retrieval (top-100
# lists, BM25 zero scores dropped, k = 60) beat dense retrieval alone?
# Query set: drawn before the test from logged queries by a stratified
# rule (a fixed share containing identifiers, codes or rare terms),
# gold chunks labelled blind to either system's output. Primary estimand:
# paired difference in recall@10, hybrid minus dense. Smallest effect of
# interest (SESOI): 3 points. Test: McNemar on discordant queries,
# two-sided 5 %, plus an equivalence check (both 90 % bounds within
# +-3 points). The model below assumes 15 % discordant queries.
# A: power at true differences of 0, 3 and 6 points. B: the decision
# rule applied to one simulated query set. 2,000 runs per cell.
import math
import random

def sample(rng, n, delta, disc=0.15):
    up = (disc + delta) / 2              # hybrid-only hits
    dn = (disc - delta) / 2              # dense-only hits
    ba = ab = 0
    for _ in range(n):
        u = rng.random()
        if u < up:
            ba += 1
        elif u < up + dn:
            ab += 1
    return ab, ba

def decide(n, ab, ba, sesoi=0.03):
    d = (ba - ab) / n
    se = math.sqrt(ab + ba - (ba - ab) ** 2 / n) / n if ab + ba else 0.0
    lo90, hi90 = d - 1.645 * se, d + 1.645 * se
    lo95 = d - 1.96 * se
    if lo95 > 0 and d >= sesoi:
        return "SUPPORTED: hybrid better by at least the SESOI"
    if lo95 > 0:
        return "BETTER, BELOW SESOI: real but smaller than 3 points"
    if lo90 > -sesoi and hi90 < sesoi:
        return "EQUIVALENT: no difference of practical size"
    if d + 1.96 * se < 0:
        return "REVERSED: dense alone is better"
    return "INCONCLUSIVE: interval too wide"

if __name__ == "__main__":
    rng = random.Random(61016)
    print("A. n queries   true +0   true +3   true +6   (share 'supported')")
    for n in (300, 1000, 2000, 3000):
        row = []
        for delta in (0.0, 0.03, 0.06):
            k = sum(decide(n, *sample(rng, n, delta)).startswith("SUPP")
                    for _ in range(2000))
            row.append(k / 2000)
        print("   {:7d}      {:.3f}     {:.3f}     {:.3f}".format(n, *row))
    ab, ba = sample(rng, 2000, 0.03)
    d = (ba - ab) / 2000
    print("\nB. one query set, n = 2000, true difference +3 points:")
    print("   dense-only hits {}, hybrid-only hits {}, difference {:+.4f}"
          .format(ab, ba, d))
    print("   " + decide(2000, ab, ba))
    tally = {}
    for _ in range(2000):
        v = decide(2000, *sample(rng, 2000, 0.03)).split(":")[0]
        tally[v] = tally.get(v, 0) + 1
    print("   verdicts over 2,000 such query sets:")
    for v in sorted(tally, key=lambda x: -tally[x]):
        print("     {:<20} {:.3f}".format(v, tally[v] / 2000))
    print("\nReading: a true 3-point gain is declared 'supported' only")
    print("about half the time even at 2,000 queries, because the rule")
    print("also demands that the estimate reach the SESOI; a 6-point gain")
    print("is caught reliably from about 1,000. The single query set in B")
    print("drew low and was called 'equivalent' - the kind of error the")
    print("equivalence check allows about one time in twenty when the")
    print("truth sits exactly at the SESOI. The rule names every outcome")
    print("in advance, including the wrong-looking ones.")

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《检索增强生成几何、倒数秩融合与多智能体误差级联》第 8 章「五步核查与一项预注册的离线评测」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · 阅读本章所在专著 · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

本书其他章节结论