5D Library › Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents · 中文

Chapter 2. Reliability, the “Shape” of a Profile and a Blind-Pick Test

How many items does a dimension need?

The reliability of a scale is usually given by Cronbach’s α\alpha (Cronbach 1951). When every item relates to its dimension in the same way (here a loading of 0.7, so two items of one dimension correlate 0.49), α\alpha for kk items equals the Spearman-Brown value kr/(1+(k−1)r)k r/(1 + (k-1)r).

A personal profile has a further problem that is easy to miss: what users are shown is often not the five scores themselves but their “shape”: which dimension is high, which low. The reliability of the difference between two dimension scores is (α−r12)/(1−r12)(\alpha – r_{12})/(1 – r_{12}), where r12r_{12} is the correlation of the two observed scores. The more the dimensions correlate, the less true signal the difference holds.

Calculation

# Chapter 2. p03: Reliability: alpha, Spearman-Brown and profile shape.
# Items load 0.7 on their dimension (p01), so two items of the same
# dimension correlate r = 0.49. For k items, Cronbach's alpha equals
# the Spearman-Brown value k r / (1 + (k - 1) r). The program gives
# alpha for k = 2, 4, 8, 10 and the k needed for 0.80 and 0.90. The
# reliability of a difference between two dimension scores (the
# "shape" of a profile) is (alpha - r12) / (1 - r12), where r12 is the
# correlation of the two observed scores, alpha phi. Finally, alpha is
# estimated from 400 simulated samples of 60 and of 300 respondents
# (4 items, phi = 0.3) to show its sampling spread (5th-95th centile).
import math
import random

R = 0.49

def sb(k, r=R):
    return k * r / (1 + (k - 1) * r)

def k_for(target, r=R):
    return math.ceil(target * (1 - r) / (r * (1 - target)))

def sample_alpha(rng, n, k=4, phi=0.3):
    rows = []
    for _ in range(n):
        g = rng.gauss(0, 1)
        f = math.sqrt(phi) * g + math.sqrt(1 - phi) * rng.gauss(0, 1)
        rows.append([0.7 * f + math.sqrt(1 - R) * rng.gauss(0, 1)
                     for _ in range(k)])
    def var(x):
        m = sum(x) / len(x)
        return sum((v - m) ** 2 for v in x) / (len(x) - 1)
    items = sum(var([r[j] for r in rows]) for j in range(k))
    total = var([sum(r) for r in rows])
    return k / (k - 1) * (1 - items / total)

if __name__ == "__main__":
    print("   items per dimension: " + "  ".join(
        "k={}: {:.4f}".format(k, sb(k)) for k in (2, 4, 8, 10)))
    print("   k needed for alpha 0.80: {}, for 0.90: {}".format(
        k_for(0.80), k_for(0.90)))
    a4 = sb(4)
    print("   reliability of a difference of two scores (k = 4):")
    for phi in (0.0, 0.3, 0.6, 0.9):
        r12 = a4 * phi
        print("     phi {:.1f}: {:.4f}".format(phi, (a4 - r12) / (1 - r12)))
    rng = random.Random(76003)
    for n in (60, 300):
        est = sorted(sample_alpha(rng, n) for _ in range(400))
        print("   alpha from {:3d} respondents: 5th {:.3f}, 95th {:.3f}"
              .format(n, est[20], est[379]))
    print("\nReading: four items give each dimension an alpha near 0.79;")
    print("0.90 needs ten. The shape of a profile is less reliable than")
    print("its parts, and much less when the dimensions correlate. A")
    print("pilot of 60 people cannot pin alpha down to better than about")
    print("a tenth.")
#    items per dimension: k=2: 0.6577  k=4: 0.7935  k=8: 0.8849  k=10: 0.9057
#    k needed for alpha 0.80: 5, for 0.90: 10
#    reliability of a difference of two scores (k = 4):
#      phi 0.0: 0.7935
#      phi 0.3: 0.7290
#      phi 0.6: 0.6059
#      phi 0.9: 0.2776
#    alpha from  60 respondents: 5th 0.701, 95th 0.855
#    alpha from 300 respondents: 5th 0.764, 95th 0.823
#
# Reading: four items give each dimension an alpha near 0.79;
# 0.90 needs ten. The shape of a profile is less reliable than
# its parts, and much less when the dimensions correlate. A
# pilot of 60 people cannot pin alpha down to better than about
# a tenth.

With 4 items per dimension α\alpha is 0.7935, with 2 only 0.6577; reaching 0.80 takes 5 items and 0.90 takes 10. The shape of the profile is less reliable: at a dimension correlation of 0.3, the difference of two dimensions has reliability 0.7290; at 0.6 it falls to 0.6059; at 0.9 only 0.2776 remains – “your F is higher than your G” is then almost pure noise. Finally, estimating α\alpha from a pilot of 60 people gives a 5th-95th centile range of 0.701 to 0.855; with 300 people it narrows to 0.764 to 0.823.

Does a report really describe its owner?

Even with reliable scores, “users find the report accurate” does not show that the report is valid. Forer gave students a personality test and then handed each the same generic description, which the students broadly rated as an accurate portrait of themselves (Forer 1949). This is the Barnum effect: vague statements fit everyone. Validity must be shown by discrimination: measures of the same trait by different methods should agree, and measures of different traits should come apart (Campbell and Fiske 1959).

A simple discrimination test is the blind pick: give each user two anonymous reports, her own and a stranger’s, and ask her to pick the one that fits her better. If the reports are generic, the hit rate is 0.5. The program lets users’ self-view correlate ρ\rho (0.5 or 0.3) with their true dimensions, takes the three reliabilities of the previous section, compares a report of the total alone with a report of all five scores, and computes the number of users needed to tell the hit rate from 0.5.

Calculation

# Chapter 2. p04: Does a profile describe its owner? A blind-pick test.
# A rating of "how well does this fit you?" cannot separate a real
# profile from a generic one (the Barnum effect). A forced choice can:
# each user sees two anonymous reports - her own and a stranger's - and
# picks the one that fits. Model: true dimensions t ~ N(0, Phi) with
# phi = 0.3; the user's self-view v_k = rho t_k + noise (rho = 0.5 or
# 0.3); a report shows m_k = sqrt(a) t_k + sqrt(1 - a) noise, where a is
# the reliability (p03). The user picks the report closer to v. With
# 20,000 simulated pairs per cell (fixed seed) the program reports the
# hit rate when reports show only a total score or all five scores; a
# generic report hits 0.5 by construction. The last column gives users
# needed to tell the hit rate from 0.5 (two-sided 0.05, power 0.90).
import math
import random
from statistics import NormalDist

def draw_t(rng, phi=0.3):
    g = rng.gauss(0, 1)
    return [math.sqrt(phi) * g + math.sqrt(1 - phi) * rng.gauss(0, 1)
            for _ in range(5)]

def hit_rate(rng, a, rho, dims, n=20000):
    hits = 0
    for _ in range(n):
        t, u = draw_t(rng), draw_t(rng)
        v = [rho * x + math.sqrt(1 - rho * rho) * rng.gauss(0, 1) for x in t]
        rep = lambda s: [math.sqrt(a) * x + math.sqrt(1 - a) * rng.gauss(0, 1)
                         for x in s]
        mo, ms = rep(t), rep(u)
        if dims == 1:
            mo, ms, vv = [sum(mo)], [sum(ms)], [sum(v)]
        else:
            vv = v
        d_own = sum((x - y) ** 2 for x, y in zip(mo, vv))
        d_str = sum((x - y) ** 2 for x, y in zip(ms, vv))
        hits += d_own < d_str
    return hits / n

def n_needed(p, alpha=0.05, power=0.90):
    z = NormalDist().inv_cdf
    a = z(1 - alpha / 2) * 0.5
    b = z(power) * math.sqrt(p * (1 - p))
    return math.ceil(((a + b) / (p - 0.5)) ** 2)

if __name__ == "__main__":
    rng = random.Random(76004)
    print("   rho  reliability  dims  hit rate  users needed")
    for rho in (0.5, 0.3):
        for a in (0.6577, 0.7935, 0.9057):
            for dims in (1, 5):
                h = hit_rate(rng, a, rho, dims)
                print("   {:.1f} {:12.4f} {:5d} {:9.4f} {:13,d}".format(
                    rho, a, dims, h, n_needed(h)))
    print("\nReading: even a reliable profile is recognised only")
    print("modestly, because people know themselves imperfectly; five")
    print("scores are recognised more often than a total. A few dozen")
    print("to a few hundred users suffice to show it, or to show that")
    print("a report does no better than a generic one.")
#    rho  reliability  dims  hit rate  users needed
#    0.5       0.6577     1    0.6354           140
#    0.5       0.6577     5    0.7094            56
#    0.5       0.7935     1    0.6444           122
#    0.5       0.7935     5    0.7386            42
#    0.5       0.9057     1    0.6505           112
#    0.5       0.9057     5    0.7587            35
#    0.3       0.6577     1    0.5726           495
#    0.3       0.6577     5    0.6273           158
#    0.3       0.7935     1    0.5830           377
#    0.3       0.7935     5    0.6432           124
#    0.3       0.9057     1    0.5810           397
#    0.3       0.9057     5    0.6472           118
#
# Reading: even a reliable profile is recognised only
# modestly, because people know themselves imperfectly; five
# scores are recognised more often than a total. A few dozen
# to a few hundred users suffice to show it, or to show that
# a report does no better than a generic one.

Even a reliable report is recognised only to a limited degree, because people’s knowledge of themselves is itself imperfect. At ρ=0.5\rho = 0.5 and reliability 0.9057, the five-score report is picked 0.7587 of the time and the total alone 0.6505; at ρ=0.3\rho = 0.3, 0.6472 and 0.5810. Five-score reports are always easier to recognise than totals. Showing a hit rate above 0.5 takes only 35 users in the most favourable case and 495 in the least. It is a cheap and strong test: it can show that a report carries information about its owner, and it can show that a report is no different from a generic description.

Hypothesis · not established

The author’s interpretation (not yet tested): five-dimensional SQ reports are picked at a rate well above 0.5 in a blind-pick test, and above a version that reports only the total. What would show this wrong: if, in a pre-registered blind-pick test, the hit rate did not differ significantly from 0.5, the reports’ “feeling of accuracy” would come from the Barnum effect, not from measurement.

Summary

Four items per dimension give an α\alpha near 0.79; 0.90 needs 10. A profile’s shape is less reliable than its scores: at a dimension correlation of 0.9 the difference of two dimensions has reliability 0.2776. Users’ ratings of accuracy cannot show validity; a blind-pick test can, with a few hundred users at most.

References

Campbell, Donald T., and Donald W. Fiske. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.” Psychological Bulletin 56 (2): 81–105.
Cronbach, Lee J. 1951. “Coefficient Alpha and the Internal Structure of Tests.” Psychometrika 16 (3): 297–334.
Forer, Bertram R. 1949. “The Fallacy of Personal Validation: A Classroom Demonstration of Gullibility.” Journal of Abnormal and Social Psychology 44 (1): 118–23.

Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).