5D Library › Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents · 中文
Chapter 2. Reliability, the “Shape” of a Profile and a Blind-Pick Test
How many items does a dimension need?
The reliability of a scale is usually given by Cronbach’s (Cronbach 1951). When every item relates to its dimension in the same way (here a loading of 0.7, so two items of one dimension correlate 0.49), for items equals the Spearman-Brown value .
A personal profile has a further problem that is easy to miss: what users are shown is often not the five scores themselves but their “shape”: which dimension is high, which low. The reliability of the difference between two dimension scores is , where is the correlation of the two observed scores. The more the dimensions correlate, the less true signal the difference holds.
Calculation
# Chapter 2. p03: Reliability: alpha, Spearman-Brown and profile shape.
# Items load 0.7 on their dimension (p01), so two items of the same
# dimension correlate r = 0.49. For k items, Cronbach's alpha equals
# the Spearman-Brown value k r / (1 + (k - 1) r). The program gives
# alpha for k = 2, 4, 8, 10 and the k needed for 0.80 and 0.90. The
# reliability of a difference between two dimension scores (the
# "shape" of a profile) is (alpha - r12) / (1 - r12), where r12 is the
# correlation of the two observed scores, alpha phi. Finally, alpha is
# estimated from 400 simulated samples of 60 and of 300 respondents
# (4 items, phi = 0.3) to show its sampling spread (5th-95th centile).
import math
import random
R = 0.49
def sb(k, r=R):
return k * r / (1 + (k - 1) * r)
def k_for(target, r=R):
return math.ceil(target * (1 - r) / (r * (1 - target)))
def sample_alpha(rng, n, k=4, phi=0.3):
rows = []
for _ in range(n):
g = rng.gauss(0, 1)
f = math.sqrt(phi) * g + math.sqrt(1 - phi) * rng.gauss(0, 1)
rows.append([0.7 * f + math.sqrt(1 - R) * rng.gauss(0, 1)
for _ in range(k)])
def var(x):
m = sum(x) / len(x)
return sum((v - m) ** 2 for v in x) / (len(x) - 1)
items = sum(var([r[j] for r in rows]) for j in range(k))
total = var([sum(r) for r in rows])
return k / (k - 1) * (1 - items / total)
if __name__ == "__main__":
print(" items per dimension: " + " ".join(
"k={}: {:.4f}".format(k, sb(k)) for k in (2, 4, 8, 10)))
print(" k needed for alpha 0.80: {}, for 0.90: {}".format(
k_for(0.80), k_for(0.90)))
a4 = sb(4)
print(" reliability of a difference of two scores (k = 4):")
for phi in (0.0, 0.3, 0.6, 0.9):
r12 = a4 * phi
print(" phi {:.1f}: {:.4f}".format(phi, (a4 - r12) / (1 - r12)))
rng = random.Random(76003)
for n in (60, 300):
est = sorted(sample_alpha(rng, n) for _ in range(400))
print(" alpha from {:3d} respondents: 5th {:.3f}, 95th {:.3f}"
.format(n, est[20], est[379]))
print("\nReading: four items give each dimension an alpha near 0.79;")
print("0.90 needs ten. The shape of a profile is less reliable than")
print("its parts, and much less when the dimensions correlate. A")
print("pilot of 60 people cannot pin alpha down to better than about")
print("a tenth.")
# items per dimension: k=2: 0.6577 k=4: 0.7935 k=8: 0.8849 k=10: 0.9057
# k needed for alpha 0.80: 5, for 0.90: 10
# reliability of a difference of two scores (k = 4):
# phi 0.0: 0.7935
# phi 0.3: 0.7290
# phi 0.6: 0.6059
# phi 0.9: 0.2776
# alpha from 60 respondents: 5th 0.701, 95th 0.855
# alpha from 300 respondents: 5th 0.764, 95th 0.823
#
# Reading: four items give each dimension an alpha near 0.79;
# 0.90 needs ten. The shape of a profile is less reliable than
# its parts, and much less when the dimensions correlate. A
# pilot of 60 people cannot pin alpha down to better than about
# a tenth.With 4 items per dimension is 0.7935, with 2 only 0.6577; reaching 0.80 takes 5 items and 0.90 takes 10. The shape of the profile is less reliable: at a dimension correlation of 0.3, the difference of two dimensions has reliability 0.7290; at 0.6 it falls to 0.6059; at 0.9 only 0.2776 remains – “your F is higher than your G” is then almost pure noise. Finally, estimating from a pilot of 60 people gives a 5th-95th centile range of 0.701 to 0.855; with 300 people it narrows to 0.764 to 0.823.
Does a report really describe its owner?
Even with reliable scores, “users find the report accurate” does not show that the report is valid. Forer gave students a personality test and then handed each the same generic description, which the students broadly rated as an accurate portrait of themselves (Forer 1949). This is the Barnum effect: vague statements fit everyone. Validity must be shown by discrimination: measures of the same trait by different methods should agree, and measures of different traits should come apart (Campbell and Fiske 1959).
A simple discrimination test is the blind pick: give each user two anonymous reports, her own and a stranger’s, and ask her to pick the one that fits her better. If the reports are generic, the hit rate is 0.5. The program lets users’ self-view correlate (0.5 or 0.3) with their true dimensions, takes the three reliabilities of the previous section, compares a report of the total alone with a report of all five scores, and computes the number of users needed to tell the hit rate from 0.5.
Calculation
# Chapter 2. p04: Does a profile describe its owner? A blind-pick test.
# A rating of "how well does this fit you?" cannot separate a real
# profile from a generic one (the Barnum effect). A forced choice can:
# each user sees two anonymous reports - her own and a stranger's - and
# picks the one that fits. Model: true dimensions t ~ N(0, Phi) with
# phi = 0.3; the user's self-view v_k = rho t_k + noise (rho = 0.5 or
# 0.3); a report shows m_k = sqrt(a) t_k + sqrt(1 - a) noise, where a is
# the reliability (p03). The user picks the report closer to v. With
# 20,000 simulated pairs per cell (fixed seed) the program reports the
# hit rate when reports show only a total score or all five scores; a
# generic report hits 0.5 by construction. The last column gives users
# needed to tell the hit rate from 0.5 (two-sided 0.05, power 0.90).
import math
import random
from statistics import NormalDist
def draw_t(rng, phi=0.3):
g = rng.gauss(0, 1)
return [math.sqrt(phi) * g + math.sqrt(1 - phi) * rng.gauss(0, 1)
for _ in range(5)]
def hit_rate(rng, a, rho, dims, n=20000):
hits = 0
for _ in range(n):
t, u = draw_t(rng), draw_t(rng)
v = [rho * x + math.sqrt(1 - rho * rho) * rng.gauss(0, 1) for x in t]
rep = lambda s: [math.sqrt(a) * x + math.sqrt(1 - a) * rng.gauss(0, 1)
for x in s]
mo, ms = rep(t), rep(u)
if dims == 1:
mo, ms, vv = [sum(mo)], [sum(ms)], [sum(v)]
else:
vv = v
d_own = sum((x - y) ** 2 for x, y in zip(mo, vv))
d_str = sum((x - y) ** 2 for x, y in zip(ms, vv))
hits += d_own < d_str
return hits / n
def n_needed(p, alpha=0.05, power=0.90):
z = NormalDist().inv_cdf
a = z(1 - alpha / 2) * 0.5
b = z(power) * math.sqrt(p * (1 - p))
return math.ceil(((a + b) / (p - 0.5)) ** 2)
if __name__ == "__main__":
rng = random.Random(76004)
print(" rho reliability dims hit rate users needed")
for rho in (0.5, 0.3):
for a in (0.6577, 0.7935, 0.9057):
for dims in (1, 5):
h = hit_rate(rng, a, rho, dims)
print(" {:.1f} {:12.4f} {:5d} {:9.4f} {:13,d}".format(
rho, a, dims, h, n_needed(h)))
print("\nReading: even a reliable profile is recognised only")
print("modestly, because people know themselves imperfectly; five")
print("scores are recognised more often than a total. A few dozen")
print("to a few hundred users suffice to show it, or to show that")
print("a report does no better than a generic one.")
# rho reliability dims hit rate users needed
# 0.5 0.6577 1 0.6354 140
# 0.5 0.6577 5 0.7094 56
# 0.5 0.7935 1 0.6444 122
# 0.5 0.7935 5 0.7386 42
# 0.5 0.9057 1 0.6505 112
# 0.5 0.9057 5 0.7587 35
# 0.3 0.6577 1 0.5726 495
# 0.3 0.6577 5 0.6273 158
# 0.3 0.7935 1 0.5830 377
# 0.3 0.7935 5 0.6432 124
# 0.3 0.9057 1 0.5810 397
# 0.3 0.9057 5 0.6472 118
#
# Reading: even a reliable profile is recognised only
# modestly, because people know themselves imperfectly; five
# scores are recognised more often than a total. A few dozen
# to a few hundred users suffice to show it, or to show that
# a report does no better than a generic one.Even a reliable report is recognised only to a limited degree, because people’s knowledge of themselves is itself imperfect. At and reliability 0.9057, the five-score report is picked 0.7587 of the time and the total alone 0.6505; at , 0.6472 and 0.5810. Five-score reports are always easier to recognise than totals. Showing a hit rate above 0.5 takes only 35 users in the most favourable case and 495 in the least. It is a cheap and strong test: it can show that a report carries information about its owner, and it can show that a report is no different from a generic description.
Hypothesis · not established
The author’s interpretation (not yet tested): five-dimensional SQ reports are picked at a rate well above 0.5 in a blind-pick test, and above a version that reports only the total. What would show this wrong: if, in a pre-registered blind-pick test, the hit rate did not differ significantly from 0.5, the reports’ “feeling of accuracy” would come from the Barnum effect, not from measurement.
Summary
Four items per dimension give an near 0.79; 0.90 needs 10. A profile’s shape is less reliable than its scores: at a dimension correlation of 0.9 the difference of two dimensions has reliability 0.2776. Users’ ratings of accuracy cannot show validity; a blind-pick test can, with a few hundred users at most.
References
Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).