5D Library › 五维灵商量化架构、先验条件化生成与检索增强认知体 · English
第八章 七项审计与一项盲比实验
七个问题
本章把全书归结为七个可以由脚本检查的问题,用于任何一个「个人档案 + 书库检索」的智能体,包括作者自己的。它们不是心理测评或经营建议。
- 五个维度的相关是否高于 0.6,以致五个分数实际上是一个?(第一章)
- 是否有维度的 低于 0.80?(第二章)
- 报告是否从未做过与陌生人报告对照的盲选测验?(第二章)
- 档案的输入是否未经核对,错误率高于 0.25?(第三章)
- 档案是否占了上下文预算的四分之一以上?(第四章)
- 跨书问题是否只用一次混合检索作答?(第五章)
- 书库中是否有与其余各卷没有关键词联系的孤岛?(第六章)
审计程序
程序把七个问题写成七个自动标志,用于两套设计:一套「朴素」智能体(维度相关 0.8,每维 2 题,没有盲选测验,档案错误率 0.3,档案占一半预算,不拆分问题,书库有 4 个孤岛),一套按本书各章修正后的「审计」设计。
推导
# Chapter 8. p15: Seven automatic flags for a profile-conditioned agent.
# F1 dimension correlation above 0.6, so five scores act as one (p01);
# F2 a dimension's alpha below 0.80 (p03);
# F3 no blind-pick test of the reports against strangers' (p04);
# F4 profile inputs not checked, error rate above 0.25 (p06);
# F5 the profile given more than a quarter of the context budget (p07);
# F6 cross-book questions answered by one blended retrieval (p10);
# F7 volumes with no keyword link to the rest of the library (p11).
DESIGNS = {
"naive": dict(phi=0.8, alpha=0.66, blind=False, err=0.3,
share=0.5, split=False, islands=4),
"audited": dict(phi=0.3, alpha=0.91, blind=True, err=0.1,
share=0.07, split=True, islands=0),
}
def flags(d):
return [
("F1 dimensions correlate above 0.6", d["phi"] > 0.6),
("F2 alpha below 0.80", d["alpha"] < 0.80),
("F3 no blind-pick test", not d["blind"]),
("F4 profile error above 0.25", d["err"] > 0.25),
("F5 profile above a quarter of budget", d["share"] > 0.25),
("F6 no question splitting", not d["split"]),
("F7 isolated volumes in the library", d["islands"] > 0),
]
if __name__ == "__main__":
for name, d in DESIGNS.items():
fl = flags(d)
n = sum(f for _, f in fl)
print(" {} design: {} of {} flags".format(name, n, len(fl)))
for label, f in fl:
print(" [{}] {}".format("X" if f else " ", label))
print("\nReading: the naive agent trips every flag. The last one,")
print("four volumes with no keyword link, was true of the library")
print("in its 73-volume edition and is cleared in the 82-volume one")
print("(p11). Clearing flags does not show that the profile")
print("predicts anything, which only the experiment of p16 can.")
# naive design: 7 of 7 flags
# [X] F1 dimensions correlate above 0.6
# [X] F2 alpha below 0.80
# [X] F3 no blind-pick test
# [X] F4 profile error above 0.25
# [X] F5 profile above a quarter of budget
# [X] F6 no question splitting
# [X] F7 isolated volumes in the library
# audited design: 0 of 7 flags
# [ ] F1 dimensions correlate above 0.6
# [ ] F2 alpha below 0.80
# [ ] F3 no blind-pick test
# [ ] F4 profile error above 0.25
# [ ] F5 profile above a quarter of budget
# [ ] F6 no question splitting
# [ ] F7 isolated volumes in the library
#
# Reading: the naive agent trips every flag. The last one,
# four volumes with no keyword link, was true of the library
# in its 73-volume edition and is cleared in the 82-volume one
# (p11). Clearing flags does not show that the profile
# predicts anything, which only the experiment of p16 can.朴素智能体七项全中,其中最后一项是本书库只有 73 卷时的状态:当时有四卷书与其余各卷没有共同关键词;纳入全部 82 卷后,第六章显示这一项已不再触发。审计设计七项都不中,但这只说明它的设置与本书的模型一致,并不说明档案真的能预测什么。后者只能由实验回答。
一项预注册的盲比实验
假说 · 尚未验证
研究方案。 每位用户提出一个问题,看到两个不标来源的回答:一个由使用本人档案的智能体生成,另一个由同一智能体换入一位陌生人的档案生成;用户选出更好的一个。对照组不是「去掉档案」,而是「换成别人的档案」,这样除了档案属于谁之外,一切都相同。主要结局:选中本人档案回答的比例,无效应时为 0.5。次要结局:第二章的报告盲选命中率。检验:双侧 0.05,把握度 0.90。由与作者无经济关系的独立分析者按预先登记的代码执行;结果不论正负都发表。
推导
# Chapter 8. p16: Sample sizes for a pre-registered blind comparison.
# (a) Blind pairwise preference: each user puts one question and sees
# two unlabeled answers, one from the profile-conditioned agent and
# one from the same agent with a stranger's profile swapped in; the
# user picks the better. Under no effect the pick rate is 0.5.
# Users needed to detect 0.55 or 0.60 (two-sided 0.05, power 0.90).
# (b) Blind pick of reports (p04): the weakest case found there, a
# hit rate of 0.5726, and the strongest, 0.7587.
# (c) Answer-level grading by two blind raters agreeing with kappa
# 0.6: the effective sample shrinks by the reliability of the
# rating; n / 0.6 users are needed instead of n.
import math
from statistics import NormalDist
def n_needed(p, alpha=0.05, power=0.90):
z = NormalDist().inv_cdf
a = z(1 - alpha / 2) * 0.5
b = z(power) * math.sqrt(p * (1 - p))
return math.ceil(((a + b) / (p - 0.5)) ** 2)
if __name__ == "__main__":
for label, p in (("(a) preference 0.55", 0.55),
("(a) preference 0.60", 0.60),
("(b) blind pick 0.5726", 0.5726),
("(b) blind pick 0.7587", 0.7587)):
print(" {:24s} users: {:,d}".format(label, n_needed(p)))
print(" (c) preference 0.55 with rating reliability 0.6: {:,d}"
.format(math.ceil(n_needed(0.55) / 0.6)))
print("\nReading: a modest real preference needs about a thousand")
print("users; a strong one a few hundred. Swapping in a stranger's")
print("profile, not removing the profile, is the right control:")
print("it keeps everything the same except whose profile it is.")
# (a) preference 0.55 users: 1,047
# (a) preference 0.60 users: 259
# (b) blind pick 0.5726 users: 495
# (b) blind pick 0.7587 users: 35
# (c) preference 0.55 with rating reliability 0.6: 1,745
#
# Reading: a modest real preference needs about a thousand
# users; a strong one a few hundred. Swapping in a stranger's
# profile, not removing the profile, is the right control:
# it keeps everything the same except whose profile it is.要检出 0.55 的偏好,需要 1,047 位用户;0.60 只需 259 位。若偏好由两位评分者盲评、评分信度只有 0.6,需要 1,745 位。报告盲选的命中率在第二章最不利的情形(0.5726)下需要 495 人,最有利的情形(0.7587)只需 35 人。这些数字都不大:无论结果是正是负,这项实验都值得做。
小结
七个问题(维度相关、信度、盲选、档案错误、档案份额、问题拆分、书库孤岛)都可以写成自动审计。检验档案是否真的有用的实验,应当以「换成陌生人的档案」为对照,检出 0.55 的偏好需要 1,047 位用户。
Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).