5D Library › 信息几何、费希尔信息度量与自然梯度优化的统计流形 · English

第五章 对偶平坦:指数族、勒让德对偶与广义勾股定理

指数族的两套坐标

前几章只用到度量。信息几何的另一半是联络:它规定什么样的曲线算「直线」(测地线),以及沿曲线怎样平移向量。统计流形上有两种自然的「直线」,它们最清楚地出现在指数族中。

指数族的密度可以写成 p(x;θ)=exp⁡{θ⋅T(x)−ψ(θ)}h(x)p(x;\theta) = \exp\{\theta\cdot T(x) – \psi(\theta)\}\,h(x),其中 θ\theta 称为自然参数,TT 是充分统计量,ψ\psi 是对数配分函数(累积量生成函数),它是凸函数。它的梯度给出另一套坐标——期望参数 η=∇ψ(θ)=Eθ[T]\eta = \nabla\psi(\theta) = E_\theta[T];它的黑塞矩阵就是费希尔度量,G(θ)=∇2ψ(θ)=Covθ(T)G(\theta) = \nabla^2\psi(\theta) = \mathrm{Cov}_\theta(T)。ψ\psi 的勒让德变换 ϕ(η)=θ⋅η−ψ(θ)\phi(\eta) = \theta\cdot\eta – \psi(\theta) 是第二个凸函数,它等于负熵(相对于 hh),并满足 θ=∇ϕ(η)\theta = \nabla\phi(\eta)、∇2ϕ(η)=G(θ)−1\nabla^2\phi(\eta) = G(\theta)^{-1}。两套坐标、两个势函数通过勒让德变换对偶 (Amari and Nagaoka 2000; Amari 2016)。

布雷格曼散度。 对凸函数 ψ\psi,布雷格曼 1967 年定义 (Bregman 1967)

Bψ(θ2,θ1)=ψ(θ2)−ψ(θ1)−∇ψ(θ1)⋅(θ2−θ1).B_\psi(\theta_2, \theta_1) = \psi(\theta_2) – \psi(\theta_1) – \nabla\psi(\theta_1)\cdot(\theta_2 – \theta_1) .

对指数族,Bψ(θ2,θ1)=D(pθ1∥pθ2)B_\psi(\theta_2, \theta_1) = D(p_{\theta_1}\,\|\,p_{\theta_2}):KL 散度就是对数配分函数的布雷格曼散度。巴纳吉等人 2005 年证明,正则指数族与一类正则布雷格曼散度之间是一一对应的 (Banerjee et al. 2005)。

两种平坦与对偶联络

在自然参数中走直线 θ(t)=(1−t)θ1+tθ2\theta(t) = (1-t)\theta_1 + t\theta_2,对应的是分布的几何混合:log⁡pt=(1−t)log⁡p1+tlog⁡p2−常数\log p_t = (1-t)\log p_1 + t\log p_2 – \text{常数}。这样的曲线称为 e-测地线(e 表示 exponential)。在期望参数中走直线,或者对离散分布直接在概率上走直线 pt=(1−t)p1+tp2p_t = (1-t)p_1 + t p_2,对应的是算术混合,称为 m-测地线(m 表示 mixture)。一个子流形如果在 θ\theta 坐标中是仿射子空间,就称为 e-平坦(例如独立模型这样在自然参数中由线性约束给出的子族;弯曲指数族虽是指数族的子族,却不是 e-平坦的);在混合意义下是仿射的,称为 m-平坦(例如「某些矩固定」的分布集合)。

甘利 1985 年把这两种联络放进一个单参数族——α-联络,α=1\alpha = 1 为 e-联络,α=−1\alpha = -1 为 m-联络,α=0\alpha = 0 为费希尔度量的列维—奇维塔联络 (Amari 1985);α 与 −α 联络相对于费希尔度量互为对偶。α-联络的定义用到一个三阶对称张量,即第一章提到的甘利—钦佐夫张量,它与费希尔度量一样在充分统计量下不变 (Ay et al. 2015)。e-联络与 m-联络都没有曲率,所以指数族是对偶平坦的:存在两套坐标,使两种「直线」分别是这两套坐标中的直线。

广义勾股定理

对偶平坦流形上最有用的结果是广义勾股定理。设 EE 是一个 e-平坦子流形,pp 是任意分布,q*q^* 是 pp 到 EE 的 m-投影,即 EE 中使 D(p∥q)D(p\,\|\,q) 最小的点(等价地,连接 pp 与 q*q^* 的 m-测地线与 EE 正交)。则对 EE 中任意一点 rr,

D(p∥r)=D(p∥q*)+D(q*∥r).D(p\,\|\,r) = D(p\,\|\,q^*) + D(q^*\,\|\,r) .

对偶地,若 MM 是 m-平坦子流形,q*q^* 是 pp 到 MM 的 e-投影(使 D(q∥p)D(q\,\|\,p) 最小),则对 MM 中任意 rr,D(r∥p)=D(r∥q*)+D(q*∥p)D(r\,\|\,p) = D(r\,\|\,q^*) + D(q^*\,\|\,p)。契萨尔 1975 年在「I-散度几何」中给出了后一种形式及其在约束极小化中的应用 (Csiszár 1975)。两种形式中,散度的方向与投影的种类必须匹配;作者框架只写了等式,没有写这一条件。

推导

# Dual flatness on a 3 x 3 table. The independence model E = {q(i,j) = a_i b_j} is an
# exponential family (e-flat). The m-projection of p onto E (minimise D(p||q), q in E)
# is the product of p's marginals, and the generalised Pythagorean theorem holds:
# D(p||r) = D(p||q*) + D(q*||r) for EVERY r in E. The identity is directional: with the
# arguments reversed it fails. Then the Legendre structure of the categorical family:
# psi(theta) = log sum exp, eta = grad psi, Hessian psi = Fisher, Bregman_psi = KL.
import math, random

def kl(p, q):
    return sum(a * math.log(a / b) for a, b in zip(p, q) if a > 0)

def product(a, b):
    return [x * y for x in a for y in b]

def marg(p):
    return ([sum(p[3 * i + j] for j in range(3)) for i in range(3)],
            [sum(p[3 * i + j] for i in range(3)) for j in range(3)])

def dirichlet(rng, k):
    g = [rng.gammavariate(1.0, 1.0) for _ in range(k)]; s = sum(g)
    return [x / s for x in g]

rng = random.Random(1975)
p = dirichlet(rng, 9); a, b = marg(p); q = product(a, b)
print(f"D(p||q*) = mutual information of p = {kl(p, q):.6f}")
print("   r in E     D(p||r)    D(p||q*) + D(q*||r)  | reversed: D(r||p) vs"
      " D(r||q*)+D(q*||p)")
for k in range(5):
    r = product(dirichlet(rng, 3), dirichlet(rng, 3))
    lhs, rhs = kl(p, r), kl(p, q) + kl(q, r)
    print(f"   r{k + 1}         {lhs:.6f}   {rhs:.6f}            |"
          f"          {kl(r, p):.6f} vs {kl(r, q) + kl(q, p):.6f}")
best = min(kl(p, product(dirichlet(rng, 3), dirichlet(rng, 3))) for _ in range(20000))
print(f"   min of D(p||r) over 20,000 random r in E: {best:.6f} (>= D(p||q*))")

theta = [0.3, -1.2, 0.8, 0.0]                       # natural parameters, 5th fixed at 0
def psi(t):
    return math.log(1 + sum(math.exp(x) for x in t))
def eta(t):                                         # expectation parameters = grad psi
    z = 1 + sum(math.exp(x) for x in t); return [math.exp(x) / z for x in t]
def dist(t):
    e = eta(t); return e + [1 - sum(e)]
h = 1e-5
hess = [[(eta([x + h * (i == j) for i, x in enumerate(theta)])[k]
          - eta([x - h * (i == j) for i, x in enumerate(theta)])[k]) / (2 * h)
         for j in range(4)] for k in range(4)]
e = eta(theta)
fisher = [[(e[i] if i == j else 0) - e[i] * e[j] for j in range(4)] for i in range(4)]
err = max(abs(hess[i][j] - fisher[i][j]) for i in range(4) for j in range(4))
print("categorical (5 cells): max |Hessian of psi - Fisher| "
      + ("< 1e-10" if err < 1e-10 else f"= {err:.1e}"))
t2 = [-0.5, 0.4, 0.1, 1.0]
breg = psi(t2) - psi(theta) - sum(a * (x - y) for a, x, y in zip(eta(theta), t2, theta))
print(f"   Bregman_psi(theta2, theta1) = {breg:.6f};  KL(p_theta1 || p_theta2) ="
      f" {kl(dist(theta), dist(t2)):.6f}")
phi = lambda t: sum(x * y for x, y in zip(theta_of(t), t)) - psi(theta_of(t))
def theta_of(et):                                  # inverse map eta -> theta
    last = 1 - sum(et); return [math.log(x / last) for x in et]
leg = abs(psi(theta) + phi(e) - sum(x * y for x, y in zip(theta, e)))
print("   Legendre: psi(theta) + phi(eta) - theta.eta "
      + ("< 1e-12;" if leg < 1e-12 else f"= {leg:.1e};")
      + f" phi(eta) = -entropy = {sum(x * math.log(x) for x in dist(theta)):.4f}")
print("alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):")
P1, Q1 = dist(theta), dist(t2)
for al in (-0.999, -0.5, 0.0, 0.5, 0.999):
    da = 4 / (1 - al * al) * (1 - sum(x ** ((1 - al) / 2) * y ** ((1 + al) / 2)
                                      for x, y in zip(P1, Q1)))
    print(f"   alpha = {al:+.3f}: {da:.6f}", end="")
    print("   (-> KL(p||q))" if al < -0.9 else "   (-> KL(q||p))" if al > 0.9 else
          "   (4 x squared Hellinger, symmetric)" if al == 0 else "")
print(f"   KL(p||q) = {kl(P1, Q1):.6f}, KL(q||p) = {kl(Q1, P1):.6f}")
# D(p||q*) = mutual information of p = 0.103351
#    r in E     D(p||r)    D(p||q*) + D(q*||r)  | reversed: D(r||p) vs D(r||q*)+D(q*||p)
#    r1         0.986857   0.986857            |          1.412998 vs 1.208860
#    r2         0.281492   0.281492            |          0.467797 vs 0.337878
#    r3         0.365068   0.365068            |          0.489205 vs 0.371257
#    r4         0.971558   0.971558            |          1.100003 vs 0.974635
#    r5         0.912109   0.912109            |          0.811763 vs 0.806931
#    min of D(p||r) over 20,000 random r in E: 0.111900 (>= D(p||q*))
# categorical (5 cells): max |Hessian of psi - Fisher| < 1e-10
#    Bregman_psi(theta2, theta1) = 0.360388;  KL(p_theta1 || p_theta2) = 0.360388
#    Legendre: psi(theta) + phi(eta) - theta.eta < 1e-12; phi(eta) = -entropy = -1.4606
# alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):
#    alpha = -0.999: 0.360387   (-> KL(p||q))
#    alpha = -0.500: 0.361804
#    alpha = +0.000: 0.367294   (4 x squared Hellinger, symmetric)
#    alpha = +0.500: 0.377191
#    alpha = +0.999: 0.391952   (-> KL(q||p))
#    KL(p||q) = 0.360388, KL(q||p) = 0.391987

程序取一个 3 × 3 列联表的分布 pp,EE 是独立模型(所有形如 aibja_ib_j 的分布,它是指数族,因而 e-平坦)。pp 到 EE 的 m-投影就是两个边际分布的乘积,D(p∥q*)D(p\,\|\,q^*) 是 pp 的互信息,0.103351。对 EE 中五个随机的 rr,等式两边逐位相同(例如 0.986857 与 0.986857)。交换散度的方向后,等式不成立:D(r∥p)D(r\,\|\,p) 为 1.412998 时,D(r∥q*)+D(q*∥p)D(r\,\|\,q^*) + D(q^*\,\|\,p) 为 1.208860。在 20,000 个随机的 rr 中,D(p∥r)D(p\,\|\,r) 的最小值为 0.111900,不低于互信息,这正是 q*q^* 是最近点的数值表现。第 11 本从信息论的角度讨论过互信息;这里看到的是它的几何意义:pp 到独立模型的「距离」。

程序的后半部分验证了勒让德结构:五类别分布的 ψ\psi 的数值黑塞矩阵与费希尔矩阵之差小于 10−1010^{-10};布雷格曼散度与 KL 散度都是 0.360388;ψ(θ)+ϕ(η)−θ⋅η\psi(\theta) + \phi(\eta) – \theta\cdot\eta 小于 10−1210^{-12},ϕ\phi 等于负熵 −1.4606。

α-散度。 与 α-联络对应的散度族是 Dα(p∥q)=41−α2(1−∑p(1−α)/2q(1+α)/2)D_\alpha(p\,\|\,q) = \frac{4}{1-\alpha^2}\big(1 – \sum p^{(1-\alpha)/2}q^{(1+\alpha)/2}\big)。程序中 α\alpha 接近 −1 时为 0.360387,趋于 D(p∥q)=0.360388D(p\,\|\,q) = 0.360388;接近 +1 时为 0.391952,趋于 D(q∥p)=0.391987D(q\,\|\,p) = 0.391987;α=0\alpha = 0 时为 0.367294,是对称的海林格距离平方的 4 倍(4(1−∑pq)4(1 – \sum\sqrt{pq}))。KL 散度的两个方向是这个族的两个端点。

投影算法:迭代比例拟合与 EM

许多经典算法可以读成交替投影。戴明与斯蒂芬 1940 年为了让抽样列联表与已知的边际总数一致,提出了迭代比例拟合:轮流按行、按列缩放,使行和、列和分别符合目标 (Deming and Stephan 1940)。契萨尔 1975 年证明,每一次缩放都是当前表到一个线性族(m-平坦,「行和固定」或「列和固定」)的 I-投影,算法收敛到同时满足两组边际、且与起始表的 KL 散度最小的表 (Csiszár 1975)。契萨尔与图斯纳迪 1984 年把这类交替极小化程序放进统一的几何框架 (Csiszár and Tusnády 1984);甘利 1995 年说明期望最大化(EM)算法与它的几何版本 em 算法也是在数据流形与模型流形之间交替做 e-投影与 m-投影 (Amari 1995)。

推导

# Iterative proportional fitting (Deming and Stephan 1940) as alternating I-projections
# (Csiszar 1975). Seed table s (4 x 4) and target margins; each row step and column step
# is the I-projection of the current table onto a linear family {q: margins fixed}. For
# the limit p*, which lies in both families, every step satisfies the Pythagorean
# identity D(p*||q_k) = D(p*||q_k+1) + D(q_k+1||q_k). The limit has the form a_i b_j s_ij
# (the e-flat family through s). Least squares onto the same margins gives a different
# table and can go negative.
import math

S = [[40, 2, 1, 1], [3, 30, 3, 2], [1, 2, 6, 1], [2, 1, 1, 4]]
tot = sum(map(sum, S)); S = [[x / tot for x in r] for r in S]
ROW, COL = [0.25, 0.25, 0.25, 0.25], [0.10, 0.20, 0.30, 0.40]

def kl(p, q):
    return sum(a * math.log(a / b) for ra, rb in zip(p, q) for a, b in zip(ra, rb) if a)

def rows(q):
    return [[x * ROW[i] / sum(r) for x in r] for i, r in enumerate(q)]

def cols(q):
    cs = [sum(r[j] for r in q) for j in range(4)]
    return [[x * COL[j] / cs[j] for j, x in enumerate(r)] for r in q]

seq = [S]
for k in range(400):
    seq.append(rows(seq[-1]) if k % 2 == 0 else cols(seq[-1]))
ps = seq[-1]
print(f"D(p*||seed) = {kl(ps, S):.6f}; sum of step divergences D(q_k+1||q_k) ="
      f" {sum(kl(seq[k + 1], seq[k]) for k in range(400)):.6f}")
print("  step   D(p*||q_k)    D(p*||q_k+1) + D(q_k+1||q_k)   max margin error")
for k in range(6):
    a, b = kl(ps, seq[k]), kl(ps, seq[k + 1]) + kl(seq[k + 1], seq[k])
    q = seq[k + 1]
    err = max(max(abs(sum(r) - ROW[i]) for i, r in enumerate(q)),
              max(abs(sum(r[j] for r in q) - COL[j]) for j in range(4)))
    print(f"   {k + 1:3d}   {a:.8f}    {b:.8f}                     {err:.2e}")
ratio = [[ps[i][j] / S[i][j] for j in range(4)] for i in range(4)]
rank1 = max(abs(ratio[i][j] * ratio[0][0] - ratio[i][0] * ratio[0][j])
            for i in range(4) for j in range(4))
print("limit / seed is a product a_i b_j: max 2x2 minor "
      + ("< 1e-12" if rank1 < 1e-12 else f"{rank1:.1e}"))
rs = [sum(r) for r in S]; cs = [sum(r[j] for r in S) for j in range(4)]
B = 0.0
for _ in range(200):                     # q = s + alpha_i + beta_j (unweighted LS)
    al = [(ROW[i] - rs[i] - B) / 4 for i in range(4)]; A = sum(al)
    be = [(COL[j] - cs[j] - A) / 4 for j in range(4)]; B = sum(be)
LS = [[S[i][j] + al[i] + be[j] for j in range(4)] for i in range(4)]
print("cell (1,1): seed {:.4f}, IPF {:.4f}, least squares {:.4f}".format(
    S[0][0], ps[0][0], LS[0][0]))
print(f"smallest cell: IPF {min(map(min, ps)):.4f},"
      f" least squares {min(map(min, LS)):+.4f}")
# D(p*||seed) = 0.731380; sum of step divergences D(q_k+1||q_k) = 0.731380
#   step   D(p*||q_k)    D(p*||q_k+1) + D(q_k+1||q_k)   max margin error
#      1   0.73137995    0.73137995                     2.35e-01
#      2   0.46345473    0.46345473                     1.53e-01
#      3   0.20169901    0.20169901                     9.96e-02
#      4   0.07947240    0.07947240                     7.65e-02
#      5   0.03256790    0.03256790                     3.72e-02
#      6   0.01083802    0.01083802                     2.82e-02
# limit / seed is a product a_i b_j: max 2x2 minor < 1e-12
# cell (1,1): seed 0.4000, IPF 0.0932, least squares 0.2625
# smallest cell: IPF 0.0015, least squares -0.0925

以 4 × 4 的起始表 ss 与目标边际为例,极限表 p*p^* 与起始表的散度为 0.731380,恰好等于全部步骤的散度 D(qk+1∥qk)D(q_{k+1}\,\|\,q_k) 之和。每一步都满足勾股等式:第一步 D(p*∥q0)=0.73137995D(p^*\,\|\,q_0) = 0.73137995,D(p*∥q1)+D(q1∥q0)D(p^*\,\|\,q_1) + D(q_1\,\|\,q_0) 也是 0.73137995,之后各步同样逐位相同,而边际误差从 0.235 逐步减小。极限表与起始表之比是 aibja_ib_j 的乘积形式(所有 2 × 2 子式小于 10−1210^{-12}),也就是说它落在经过 ss 的 e-平坦族中——这是对偶的另一半。

有一个历史细节值得一提:戴明与斯蒂芬的论文题目说的是「最小二乘调整」,他们原本把这个算法当作最小二乘解的近似;后来才弄清它的极限其实是最小 KL 散度的解,而不是最小二乘解。程序比较了两者:同样满足边际约束,起始表中 0.40 的格子在迭代比例拟合中变为 0.0932,在(不加权的)最小二乘中为 0.2625;最小二乘解中最小的格子是 −0.0925——一个负的「概率」,而迭代比例拟合的解始终为正(最小 0.0015)。

假说 · 尚未验证 · 投影顺序的可诊断性

作者的假说(尚未确立): 在用迭代比例拟合校准调查权重的实际应用中,若把每一步的勾股分解 D(p*∥qk)=D(p*∥qk+1)+D(qk+1∥qk)D(p^*\,\|\,q_k) = D(p^*\,\|\,q_{k+1}) + D(q_{k+1}\,\|\,q_k) 作为诊断量报告,可以比只报告边际误差更早地发现边际约束之间相互矛盾(无解)的情形。什么样的结果能证明它是错的: 在一组事先登记、含有可行与不可行约束的模拟校准问题中,基于勾股分解的停止规则识别不可行问题所需的迭代次数,不少于基于边际误差的规则。

本章小结

指数族有自然参数 θ\theta 与期望参数 η\eta 两套坐标,由凸函数 ψ\psi 与其勒让德对偶 ϕ\phi(负熵)联系;费希尔度量是 ψ\psi 的黑塞矩阵,KL 散度是 ψ\psi 的布雷格曼散度(程序中都是 0.360388)。e-测地线是几何混合,m-测地线是算术混合;α-联络把两者连成一族。广义勾股定理在「m-投影到 e-平坦子流形」(或其对偶)时成立,散度方向不能颠倒:程序中正确方向逐位相等,颠倒后 1.412998 对 1.208860。迭代比例拟合是交替的 I-投影,每一步都满足勾股等式;它的极限是最小 KL 解而不是最小二乘解,后者可以出现负的格子。

参考文献

Amari, Shun-ichi. 1985. Differential-Geometrical Methods in Statistics. Vol. 28. Lecture Notes in Statistics. Springer.
Amari, Shun-ichi. 1995. “Information Geometry of the EM and Em Algorithms for Neural Networks.” Neural Networks 8 (9): 1379–408.
Amari, Shun-ichi. 2016. Information Geometry and Its Applications. Vol. 194. Applied Mathematical Sciences. Springer.
Amari, Shun-ichi, and Hiroshi Nagaoka. 2000. Methods of Information Geometry. Vol. 191. Translations of Mathematical Monographs. American Mathematical Society; Oxford University Press.
Ay, Nihat, Jürgen Jost, Hông Vân Lê, and Lorenz Schwachhöfer. 2015. “Information Geometry and Sufficient Statistics.” Probability Theory and Related Fields 162 (1–2): 327–64. https://doi.org/10.1007/s00440-014-0574-8.
Banerjee, Arindam, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. 2005. “Clustering with Bregman Divergences.” Journal of Machine Learning Research 6: 1705–49.
Bregman, L. M. 1967. “The Relaxation Method of Finding the Common Point of Convex Sets and Its Application to the Solution of Problems in Convex Programming.” USSR Computational Mathematics and Mathematical Physics 7 (3): 200–217.
Csiszár, Imre. 1975. “II-Divergence Geometry of Probability Distributions and Minimization Problems.” Annals of Probability 3 (1): 146–58.
Csiszár, Imre, and Gábor Tusnády. 1984. “Information Geometry and Alternating Minimization Procedures.” Statistics & Decisions Supplement Issue 1: 205–37.
Deming, W. Edwards, and Frederick F. Stephan. 1940. “On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals Are Known.” Annals of Mathematical Statistics 11 (4): 427–44. https://doi.org/10.1214/aoms/1177731829.

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).