5D Library › 五维灵商量化架构、先验条件化生成与检索增强认知体 · English

第三章 先验条件化:一份档案能让回答少多少不确定

把问题写成条件熵

作者的设计是:每位用户有一份约 8 KB 的 destiny_profile.json,向量 zz 记录其五维状态与相关信息;模型在回答问题 xx 时,把 zz 与检索到的段落 c1,…,ckc_1, \dots, c_k 一起放进提示,生成 P(y∣x,z,c)P(y \mid x, z, c)。提纲认为这样可以降低回答的条件熵 H(Y∣X)H(Y \mid X)。

信息论给出的答案很简单,也很严格 (Cover and Thomas 2006):加入 ZZ 后条件熵降低的量是条件互信息

I(Y;Z∣X)=H(Y∣X)−H(Y∣X,Z),I(Y; Z \mid X) = H(Y \mid X) – H(Y \mid X, Z),

它不超过 H(Y∣X)H(Y \mid X),也不超过 H(Z)H(Z)。也就是说,档案能带来的好处有两个上限:回答本身有多少不确定,以及回答真正依赖档案的程度。文件有多大并不在其中。

一个可以精确计算的例子

程序设 16 类问题、8 类建议、档案为五个维度各三档(243 种状态)。在 ONE 情形中,每类问题只与一个维度有关,合适的建议由该维度决定,以 0.7 的概率给出合适的建议;TWO 情形与两个维度有关;NONE 情形中建议与档案无关。

推导

# Chapter 3. p05: How much can a profile tell the model? Exact entropies.
# X: 16 kinds of question, equally likely. Z: a profile of five
# dimensions with three levels each (243 states, uniform), as stored in
# destiny_profile.json. Y: one of 8 kinds of advice. For question x
# the relevant dimension is k = x mod 5 (case ONE) or k and k + 1 (case
# TWO); the fitting advice is y* = (x + 3 z_k) mod 8 (case TWO adds
# 5 z_k+1), given with probability 0.7, any of the 8 otherwise. Case
# NONE: the advice ignores Z (y* = x mod 8). Computed exactly:
#     H(Y|X), H(Y|X,Z) and I(Y;Z|X) = H(Y|X) - H(Y|X,Z), in bits,
# set against H(Z) = log2 243 and the 65,536 bits of an 8 KB file.
import itertools
import math

def h(p):
    return -sum(x * math.log2(x) for x in p if x > 0)

def ystar(case, x, z):
    k = x % 5
    if case == "ONE":
        return (x + 3 * z[k]) % 8
    if case == "TWO":
        return (x + 3 * z[k] + 5 * z[(k + 1) % 5]) % 8
    return x % 8

def entropies(case, keep=0.7):
    zs = list(itertools.product(range(3), repeat=5))
    hyx = hyxz = 0.0
    for x in range(16):
        py = [0.0] * 8
        for z in zs:
            ys = ystar(case, x, z)
            for y in range(8):
                p = (1 - keep) / 8 + (keep if y == ys else 0.0)
                py[y] += p / len(zs)
        hyx += h(py) / 16
        hyxz += h([(1 - keep) / 8 + (keep if y == 0 else 0)
                   for y in range(8)]) / 16
    gain = hyx - hyxz
    return hyx, hyxz, gain if abs(gain) > 1e-12 else 0.0

if __name__ == "__main__":
    print("   H(Z) = log2 243 = {:.4f} bits; an 8 KB file: {:,d} bits"
          .format(math.log2(243), 8 * 1024 * 8))
    print("   case   H(Y|X)   H(Y|X,Z)  I(Y;Z|X)")
    for case in ("NONE", "ONE", "TWO"):
        a, b, c = entropies(case)
        print("   {:5s} {:8.4f} {:10.4f} {:9.4f}".format(case, a, b, c))
    print("   upper bound on any gain: H(Y) = log2 8 = 3 bits")
    print("\nReading: a profile can lower the uncertainty of the advice")
    print("by at most what the advice depends on: here about one bit,")
    print("against 7.9 bits of profile and 65,536 bits of file. If the")
    print("advice ignores the profile, the gain is exactly zero.")
    print("Conditioning helps only through a real dependence, which")
    print("must be shown, not assumed.")
#    H(Z) = log2 243 = 7.9248 bits; an 8 KB file: 65,536 bits
#    case   H(Y|X)   H(Y|X,Z)  I(Y;Z|X)
#    NONE    1.5674     1.5674    0.0000
#    ONE     2.4194     1.5674    0.8519
#    TWO     2.6781     1.5674    1.1107
#    upper bound on any gain: H(Y) = log2 8 = 3 bits
#
# Reading: a profile can lower the uncertainty of the advice
# by at most what the advice depends on: here about one bit,
# against 7.9 bits of profile and 65,536 bits of file. If the
# advice ignores the profile, the gain is exactly zero.
# Conditioning helps only through a real dependence, which
# must be shown, not assumed.

243 种档案状态的熵是 7.9248 比特,一个 8 KB 的文件可以装 65,536 比特;但在 ONE 情形中,档案只能把回答的条件熵从 2.4194 降到 1.5674,即 0.8519 比特;TWO 情形 1.1107 比特。在 NONE 情形中,档案带来的降低恰好为 0。任何档案的收益都不能超过建议本身的 3 比特。

这一结论的含义是:档案的价值不在于它写了多少,而在于回答是否真的依赖其中的内容。若回答对档案的依赖没有被证明,在提示中加入档案只是让回答「看起来」更个人化。

档案出错时

档案的输入可能有错:出生时辰记错、问卷随手作答、资料过时。程序让档案的每个维度以概率 ee 被随机替换,计算收益 I(Y;Z′∣X)I(Y; Z' \mid X)。

推导

# Chapter 3. p06: A noisy profile: how fast the gain disappears.
# Case ONE of p05 (advice depends on one dimension of three levels,
# kept with probability 0.7). The stored profile Z' is wrong on each
# dimension with probability e: the level is replaced by one of the
# three at random (a wrong birth hour, a careless answer, an outdated
# questionnaire). The gain I(Y;Z'|X) = H(Y|X) - H(Y|X,Z') is computed
# exactly for e from 0 to 1, with the share of the noiseless gain kept.
import math

def h(p):
    return -sum(x * math.log2(x) for x in p if x > 0)

def p_y(x, zk, keep=0.7):
    ys = (x + 3 * zk) % 8
    return [(1 - keep) / 8 + (keep if y == ys else 0.0) for y in range(8)]

def gain(e):
    hyx = hyxz = 0.0
    for x in range(16):
        prior = [sum(p_y(x, z)[y] for z in range(3)) / 3 for y in range(8)]
        hyx += h(prior) / 16
        for zp in range(3):
            post = [(1 - e) * (z == zp) + e / 3 for z in range(3)]
            norm = sum(post)
            py = [sum(post[z] * p_y(x, z)[y] for z in range(3)) / norm
                  for y in range(8)]
            hyxz += h(py) / 16 / 3
    return hyx - hyxz

if __name__ == "__main__":
    g0 = gain(0.0)
    print("   error rate e   I(Y;Z'|X) bits   share of noiseless gain")
    for e in (0.0, 0.1, 0.25, 0.5, 0.75, 1.0):
        g = max(0.0, gain(e))
        print("   {:12.2f} {:16.4f} {:25.4f}".format(e, g, g / g0))
    print("\nReading: the gain falls faster than the error rate rises;")
    print("a profile wrong one time in four keeps about half of what it")
    print("could give. Checking the inputs is worth more than adding")
    print("dimensions to an uncertain profile.")
#    error rate e   I(Y;Z'|X) bits   share of noiseless gain
#            0.00           0.8519                    1.0000
#            0.10           0.6711                    0.7878
#            0.25           0.4557                    0.5350
#            0.50           0.2013                    0.2363
#            0.75           0.0515                    0.0605
#            1.00           0.0000                    0.0000
#
# Reading: the gain falls faster than the error rate rises;
# a profile wrong one time in four keeps about half of what it
# could give. Checking the inputs is worth more than adding
# dimensions to an uncertain profile.
档案出错率与保留的信息增益。虚线为线性参照。

收益下降得比错误率上升得快:错误率 0.10 时保留 0.7878,0.25 时只剩 0.5350,0.50 时只剩 0.2363。核对档案的输入,比给一份不可靠的档案增加维度更值得。

假说 · 尚未验证

作者的诠释(尚未验证): 在作者的系统中,五维档案确实改变了合适建议的分布(I(Y;Z∣X)>0I(Y; Z \mid X) > 0),且出生时间等输入经过核对后,档案的错误率低于 0.10。什么会证明这一判断错:若在第八章的盲比实验中,用户对「本人档案」与「换入陌生人档案」生成的回答没有显著偏好,那么在实际使用中 I(Y;Z∣X)I(Y; Z \mid X) 接近于零。

小结

档案对回答的帮助以条件互信息衡量,它受回答的不确定性与回答对档案的依赖程度两个上限约束,与文件大小无关:本书例子中,7.9248 比特的档案只带来 0.8519 比特的收益;回答不依赖档案时为 0。档案四分之一出错时,收益只剩 0.5350。

参考文献

Cover, Thomas M., and Joy A. Thomas. 2006. Elements of Information Theory. 2nd ed. Wiley.

Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).