5D Library › Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents · 中文

Chapter 8. A Seven-Flag Audit and a Blind Comparison

Seven questions

This chapter turns the book into seven questions a script can check against any “personal profile plus library retrieval” agent, the author’s own included. They are not psychological assessment or business advice.

  1. Do the five dimensions correlate above 0.6, so that five scores are really one? (Chapter 1)
  2. Does any dimension have an α\alpha below 0.80? (Chapter 2)
  3. Have the reports never been put through a blind-pick test against strangers’ reports? (Chapter 2)
  4. Are the profile’s inputs unchecked, with an error rate above 0.25? (Chapter 3)
  5. Does the profile take more than a quarter of the context budget? (Chapter 4)
  6. Are cross-book questions answered with a single blended retrieval? (Chapter 5)
  7. Does the library have islands with no keyword link to the other volumes? (Chapter 6)

The audit as a program

The program writes the seven questions as seven automatic flags and applies them to two designs: a “naive” agent (dimension correlation 0.8, 2 items per dimension, no blind-pick test, a profile error rate of 0.3, the profile taking half the budget, no question splitting, 4 islands in the library) and an “audited” design corrected chapter by chapter in this book.

Calculation

# Chapter 8. p15: Seven automatic flags for a profile-conditioned agent.
# F1 dimension correlation above 0.6, so five scores act as one (p01);
# F2 a dimension's alpha below 0.80 (p03);
# F3 no blind-pick test of the reports against strangers' (p04);
# F4 profile inputs not checked, error rate above 0.25 (p06);
# F5 the profile given more than a quarter of the context budget (p07);
# F6 cross-book questions answered by one blended retrieval (p10);
# F7 volumes with no keyword link to the rest of the library (p11).

DESIGNS = {
    "naive": dict(phi=0.8, alpha=0.66, blind=False, err=0.3,
                  share=0.5, split=False, islands=4),
    "audited": dict(phi=0.3, alpha=0.91, blind=True, err=0.1,
                    share=0.07, split=True, islands=0),
}

def flags(d):
    return [
        ("F1 dimensions correlate above 0.6", d["phi"] > 0.6),
        ("F2 alpha below 0.80", d["alpha"] < 0.80),
        ("F3 no blind-pick test", not d["blind"]),
        ("F4 profile error above 0.25", d["err"] > 0.25),
        ("F5 profile above a quarter of budget", d["share"] > 0.25),
        ("F6 no question splitting", not d["split"]),
        ("F7 isolated volumes in the library", d["islands"] > 0),
    ]

if __name__ == "__main__":
    for name, d in DESIGNS.items():
        fl = flags(d)
        n = sum(f for _, f in fl)
        print("   {} design: {} of {} flags".format(name, n, len(fl)))
        for label, f in fl:
            print("     [{}] {}".format("X" if f else " ", label))
    print("\nReading: the naive agent trips every flag. The last one,")
    print("four volumes with no keyword link, was true of the library")
    print("in its 73-volume edition and is cleared in the 82-volume one")
    print("(p11). Clearing flags does not show that the profile")
    print("predicts anything, which only the experiment of p16 can.")
#    naive design: 7 of 7 flags
#      [X] F1 dimensions correlate above 0.6
#      [X] F2 alpha below 0.80
#      [X] F3 no blind-pick test
#      [X] F4 profile error above 0.25
#      [X] F5 profile above a quarter of budget
#      [X] F6 no question splitting
#      [X] F7 isolated volumes in the library
#    audited design: 0 of 7 flags
#      [ ] F1 dimensions correlate above 0.6
#      [ ] F2 alpha below 0.80
#      [ ] F3 no blind-pick test
#      [ ] F4 profile error above 0.25
#      [ ] F5 profile above a quarter of budget
#      [ ] F6 no question splitting
#      [ ] F7 isolated volumes in the library
#
# Reading: the naive agent trips every flag. The last one,
# four volumes with no keyword link, was true of the library
# in its 73-volume edition and is cleared in the 82-volume one
# (p11). Clearing flags does not show that the profile
# predicts anything, which only the experiment of p16 can.

The naive agent trips all seven flags, the last of which was the state of this library when it had 73 volumes: four volumes then shared no keyword with the rest; with all 82 volumes, Chapter 6 shows that this flag no longer trips. The audited design trips none, but that says only that its settings agree with this book’s models, not that the profile really predicts anything. Only an experiment can answer that.

A pre-registered blind comparison

Hypothesis · not established

Study protocol. Each user asks one question and sees two answers with no indication of source: one generated by the agent with the user’s own profile, the other by the same agent with a stranger’s profile swapped in; the user picks the better. The control is not “no profile” but “someone else’s profile”, so that everything is the same except whose profile it is. Primary outcome: the share of picks going to the own-profile answer, 0.5 under no effect. Secondary outcome: the report blind-pick hit rate of Chapter 2. Test: two-sided at 0.05 with power 0.90. Carried out with pre-registered code by an independent analyst with no financial relationship with the author; results published whether positive or negative.

Calculation

# Chapter 8. p16: Sample sizes for a pre-registered blind comparison.
# (a) Blind pairwise preference: each user puts one question and sees
#     two unlabeled answers, one from the profile-conditioned agent and
#     one from the same agent with a stranger's profile swapped in; the
#     user picks the better. Under no effect the pick rate is 0.5.
#     Users needed to detect 0.55 or 0.60 (two-sided 0.05, power 0.90).
# (b) Blind pick of reports (p04): the weakest case found there, a
#     hit rate of 0.5726, and the strongest, 0.7587.
# (c) Answer-level grading by two blind raters agreeing with kappa
#     0.6: the effective sample shrinks by the reliability of the
#     rating; n / 0.6 users are needed instead of n.
import math
from statistics import NormalDist

def n_needed(p, alpha=0.05, power=0.90):
    z = NormalDist().inv_cdf
    a = z(1 - alpha / 2) * 0.5
    b = z(power) * math.sqrt(p * (1 - p))
    return math.ceil(((a + b) / (p - 0.5)) ** 2)

if __name__ == "__main__":
    for label, p in (("(a) preference 0.55", 0.55),
                     ("(a) preference 0.60", 0.60),
                     ("(b) blind pick 0.5726", 0.5726),
                     ("(b) blind pick 0.7587", 0.7587)):
        print("   {:24s} users: {:,d}".format(label, n_needed(p)))
    print("   (c) preference 0.55 with rating reliability 0.6: {:,d}"
          .format(math.ceil(n_needed(0.55) / 0.6)))
    print("\nReading: a modest real preference needs about a thousand")
    print("users; a strong one a few hundred. Swapping in a stranger's")
    print("profile, not removing the profile, is the right control:")
    print("it keeps everything the same except whose profile it is.")
#    (a) preference 0.55      users: 1,047
#    (a) preference 0.60      users: 259
#    (b) blind pick 0.5726    users: 495
#    (b) blind pick 0.7587    users: 35
#    (c) preference 0.55 with rating reliability 0.6: 1,745
#
# Reading: a modest real preference needs about a thousand
# users; a strong one a few hundred. Swapping in a stranger's
# profile, not removing the profile, is the right control:
# it keeps everything the same except whose profile it is.

Detecting a preference of 0.55 needs 1,047 users; 0.60 needs only 259. If preferences are judged blind by two raters with a rating reliability of only 0.6, 1,745 users are needed. The report blind-pick hit rate needs 495 people in the least favourable case of Chapter 2 (0.5726) and only 35 in the most favourable (0.7587). None of these numbers is large: whether the result is positive or negative, the experiment is worth running.

Summary

Seven questions (dimension correlation, reliability, blind pick, profile errors, profile share, question splitting, library islands) can all be written as an automatic audit. The experiment that tests whether the profile really helps should use “a stranger’s profile” as its control; detecting a preference of 0.55 needs 1,047 users.

Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).