5D Library › 五维灵商量化架构、先验条件化生成与检索增强认知体 · English

第八章 七项审计与一项盲比实验

七个问题

本章把全书归结为七个可以由脚本检查的问题,用于任何一个「个人档案 + 书库检索」的智能体,包括作者自己的。它们不是心理测评或经营建议。

  1. 五个维度的相关是否高于 0.6,以致五个分数实际上是一个?(第一章)
  2. 是否有维度的 α\alpha 低于 0.80?(第二章)
  3. 报告是否从未做过与陌生人报告对照的盲选测验?(第二章)
  4. 档案的输入是否未经核对,错误率高于 0.25?(第三章)
  5. 档案是否占了上下文预算的四分之一以上?(第四章)
  6. 跨书问题是否只用一次混合检索作答?(第五章)
  7. 书库中是否有与其余各卷没有关键词联系的孤岛?(第六章)

审计程序

程序把七个问题写成七个自动标志,用于两套设计:一套「朴素」智能体(维度相关 0.8,每维 2 题,没有盲选测验,档案错误率 0.3,档案占一半预算,不拆分问题,书库有 4 个孤岛),一套按本书各章修正后的「审计」设计。

推导

# Chapter 8. p15: Seven automatic flags for a profile-conditioned agent.
# F1 dimension correlation above 0.6, so five scores act as one (p01);
# F2 a dimension's alpha below 0.80 (p03);
# F3 no blind-pick test of the reports against strangers' (p04);
# F4 profile inputs not checked, error rate above 0.25 (p06);
# F5 the profile given more than a quarter of the context budget (p07);
# F6 cross-book questions answered by one blended retrieval (p10);
# F7 volumes with no keyword link to the rest of the library (p11).

DESIGNS = {
    "naive": dict(phi=0.8, alpha=0.66, blind=False, err=0.3,
                  share=0.5, split=False, islands=4),
    "audited": dict(phi=0.3, alpha=0.91, blind=True, err=0.1,
                    share=0.07, split=True, islands=0),
}

def flags(d):
    return [
        ("F1 dimensions correlate above 0.6", d["phi"] > 0.6),
        ("F2 alpha below 0.80", d["alpha"] < 0.80),
        ("F3 no blind-pick test", not d["blind"]),
        ("F4 profile error above 0.25", d["err"] > 0.25),
        ("F5 profile above a quarter of budget", d["share"] > 0.25),
        ("F6 no question splitting", not d["split"]),
        ("F7 isolated volumes in the library", d["islands"] > 0),
    ]

if __name__ == "__main__":
    for name, d in DESIGNS.items():
        fl = flags(d)
        n = sum(f for _, f in fl)
        print("   {} design: {} of {} flags".format(name, n, len(fl)))
        for label, f in fl:
            print("     [{}] {}".format("X" if f else " ", label))
    print("\nReading: the naive agent trips every flag. The last one,")
    print("four volumes with no keyword link, was true of the library")
    print("in its 73-volume edition and is cleared in the 82-volume one")
    print("(p11). Clearing flags does not show that the profile")
    print("predicts anything, which only the experiment of p16 can.")
#    naive design: 7 of 7 flags
#      [X] F1 dimensions correlate above 0.6
#      [X] F2 alpha below 0.80
#      [X] F3 no blind-pick test
#      [X] F4 profile error above 0.25
#      [X] F5 profile above a quarter of budget
#      [X] F6 no question splitting
#      [X] F7 isolated volumes in the library
#    audited design: 0 of 7 flags
#      [ ] F1 dimensions correlate above 0.6
#      [ ] F2 alpha below 0.80
#      [ ] F3 no blind-pick test
#      [ ] F4 profile error above 0.25
#      [ ] F5 profile above a quarter of budget
#      [ ] F6 no question splitting
#      [ ] F7 isolated volumes in the library
#
# Reading: the naive agent trips every flag. The last one,
# four volumes with no keyword link, was true of the library
# in its 73-volume edition and is cleared in the 82-volume one
# (p11). Clearing flags does not show that the profile
# predicts anything, which only the experiment of p16 can.

朴素智能体七项全中,其中最后一项是本书库只有 73 卷时的状态:当时有四卷书与其余各卷没有共同关键词;纳入全部 82 卷后,第六章显示这一项已不再触发。审计设计七项都不中,但这只说明它的设置与本书的模型一致,并不说明档案真的能预测什么。后者只能由实验回答。

一项预注册的盲比实验

假说 · 尚未验证

研究方案。 每位用户提出一个问题,看到两个不标来源的回答:一个由使用本人档案的智能体生成,另一个由同一智能体换入一位陌生人的档案生成;用户选出更好的一个。对照组不是「去掉档案」,而是「换成别人的档案」,这样除了档案属于谁之外,一切都相同。主要结局:选中本人档案回答的比例,无效应时为 0.5。次要结局:第二章的报告盲选命中率。检验:双侧 0.05,把握度 0.90。由与作者无经济关系的独立分析者按预先登记的代码执行;结果不论正负都发表。

推导

# Chapter 8. p16: Sample sizes for a pre-registered blind comparison.
# (a) Blind pairwise preference: each user puts one question and sees
#     two unlabeled answers, one from the profile-conditioned agent and
#     one from the same agent with a stranger's profile swapped in; the
#     user picks the better. Under no effect the pick rate is 0.5.
#     Users needed to detect 0.55 or 0.60 (two-sided 0.05, power 0.90).
# (b) Blind pick of reports (p04): the weakest case found there, a
#     hit rate of 0.5726, and the strongest, 0.7587.
# (c) Answer-level grading by two blind raters agreeing with kappa
#     0.6: the effective sample shrinks by the reliability of the
#     rating; n / 0.6 users are needed instead of n.
import math
from statistics import NormalDist

def n_needed(p, alpha=0.05, power=0.90):
    z = NormalDist().inv_cdf
    a = z(1 - alpha / 2) * 0.5
    b = z(power) * math.sqrt(p * (1 - p))
    return math.ceil(((a + b) / (p - 0.5)) ** 2)

if __name__ == "__main__":
    for label, p in (("(a) preference 0.55", 0.55),
                     ("(a) preference 0.60", 0.60),
                     ("(b) blind pick 0.5726", 0.5726),
                     ("(b) blind pick 0.7587", 0.7587)):
        print("   {:24s} users: {:,d}".format(label, n_needed(p)))
    print("   (c) preference 0.55 with rating reliability 0.6: {:,d}"
          .format(math.ceil(n_needed(0.55) / 0.6)))
    print("\nReading: a modest real preference needs about a thousand")
    print("users; a strong one a few hundred. Swapping in a stranger's")
    print("profile, not removing the profile, is the right control:")
    print("it keeps everything the same except whose profile it is.")
#    (a) preference 0.55      users: 1,047
#    (a) preference 0.60      users: 259
#    (b) blind pick 0.5726    users: 495
#    (b) blind pick 0.7587    users: 35
#    (c) preference 0.55 with rating reliability 0.6: 1,745
#
# Reading: a modest real preference needs about a thousand
# users; a strong one a few hundred. Swapping in a stranger's
# profile, not removing the profile, is the right control:
# it keeps everything the same except whose profile it is.

要检出 0.55 的偏好,需要 1,047 位用户;0.60 只需 259 位。若偏好由两位评分者盲评、评分信度只有 0.6,需要 1,745 位。报告盲选的命中率在第二章最不利的情形(0.5726)下需要 495 人,最有利的情形(0.7587)只需 35 人。这些数字都不大:无论结果是正是负,这项实验都值得做。

小结

七个问题(维度相关、信度、盲选、档案错误、档案份额、问题拆分、书库孤岛)都可以写成自动审计。检验档案是否真的有用的实验,应当以「换成陌生人的档案」为对照,检出 0.55 的偏好需要 1,047 位用户。

Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).