关于「五步核查与一项预注册的离线评测」,《检索增强与多智能体的可靠性》给出的定量结论是什么?
结论:第八章给出五步核查与一项预注册的离线评测,并如实展示一次「坏运气」得出的错误结论及其错误率(0.045)。
数据口径:全部数据为程序生成的合成数据,数字说明机制与量级,不是对任何产品的测量。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.045
A. n queries true +0 true +3 true +6 (share 'supported')
300 0.026 0.275 0.782
1000 0.008 0.502 0.995
2000 0.000 0.508 0.999
3000 0.000 0.513 1.000
B. one query set, n = 2000, true difference +3 points:
dense-only hits 136, hybrid-only hits 158, difference +0.0110
EQUIVALENT: no difference of practical size
verdicts over 2,000 such query sets:
SUPPORTED 0.517
BETTER, BELOW SESOI 0.426
EQUIVALENT 0.045
INCONCLUSIVE 0.011
Reading: a true 3-point gain is declared 'supported' only
about half the time even at 2,000 queries, because the rule
also demands that the estimate reach the SESOI; a 6-point gain
is caught reliably from about 1,000. The single query set in B
drew low and was called 'equivalent' - the kind of error the
equivalence check allows about one time in twenty when the
truth sits exactly at the SESOI. The rule names every outcome
in advance, including the wrong-looking ones.查看完整验算程序
# Chapter 8. p16: A pre-registered offline test: hybrid versus dense.
# Question: on one's own corpus, does rank-fused hybrid retrieval (top-100
# lists, BM25 zero scores dropped, k = 60) beat dense retrieval alone?
# Query set: drawn before the test from logged queries by a stratified
# rule (a fixed share containing identifiers, codes or rare terms),
# gold chunks labelled blind to either system's output. Primary estimand:
# paired difference in recall@10, hybrid minus dense. Smallest effect of
# interest (SESOI): 3 points. Test: McNemar on discordant queries,
# two-sided 5 %, plus an equivalence check (both 90 % bounds within
# +-3 points). The model below assumes 15 % discordant queries.
# A: power at true differences of 0, 3 and 6 points. B: the decision
# rule applied to one simulated query set. 2,000 runs per cell.
import math
import random
def sample(rng, n, delta, disc=0.15):
up = (disc + delta) / 2 # hybrid-only hits
dn = (disc - delta) / 2 # dense-only hits
ba = ab = 0
for _ in range(n):
u = rng.random()
if u < up:
ba += 1
elif u < up + dn:
ab += 1
return ab, ba
def decide(n, ab, ba, sesoi=0.03):
d = (ba - ab) / n
se = math.sqrt(ab + ba - (ba - ab) ** 2 / n) / n if ab + ba else 0.0
lo90, hi90 = d - 1.645 * se, d + 1.645 * se
lo95 = d - 1.96 * se
if lo95 > 0 and d >= sesoi:
return "SUPPORTED: hybrid better by at least the SESOI"
if lo95 > 0:
return "BETTER, BELOW SESOI: real but smaller than 3 points"
if lo90 > -sesoi and hi90 < sesoi:
return "EQUIVALENT: no difference of practical size"
if d + 1.96 * se < 0:
return "REVERSED: dense alone is better"
return "INCONCLUSIVE: interval too wide"
if __name__ == "__main__":
rng = random.Random(61016)
print("A. n queries true +0 true +3 true +6 (share 'supported')")
for n in (300, 1000, 2000, 3000):
row = []
for delta in (0.0, 0.03, 0.06):
k = sum(decide(n, *sample(rng, n, delta)).startswith("SUPP")
for _ in range(2000))
row.append(k / 2000)
print(" {:7d} {:.3f} {:.3f} {:.3f}".format(n, *row))
ab, ba = sample(rng, 2000, 0.03)
d = (ba - ab) / 2000
print("\nB. one query set, n = 2000, true difference +3 points:")
print(" dense-only hits {}, hybrid-only hits {}, difference {:+.4f}"
.format(ab, ba, d))
print(" " + decide(2000, ab, ba))
tally = {}
for _ in range(2000):
v = decide(2000, *sample(rng, 2000, 0.03)).split(":")[0]
tally[v] = tally.get(v, 0) + 1
print(" verdicts over 2,000 such query sets:")
for v in sorted(tally, key=lambda x: -tally[x]):
print(" {:<20} {:.3f}".format(v, tally[v] / 2000))
print("\nReading: a true 3-point gain is declared 'supported' only")
print("about half the time even at 2,000 queries, because the rule")
print("also demands that the estimate reach the SESOI; a 6-point gain")
print("is caught reliably from about 1,000. The single query set in B")
print("drew low and was called 'equivalent' - the kind of error the")
print("equivalence check allows about one time in twenty when the")
print("truth sits exactly at the SESOI. The rule names every outcome")
print("in advance, including the wrong-looking ones.")唯一数据源与引用
范雅福(Fan, Y.). (2026). 《检索增强生成几何、倒数秩融合与多智能体误差级联》第 8 章「五步核查与一项预注册的离线评测」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170
DOI: 10.5281/zenodo.24090170 · 阅读本章所在专著 · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026061,
author = {Fan, Yafu},
title = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090170},
url = {https://doi.org/10.5281/zenodo.24090170}
}