关于「对偶平坦:指数族、勒让德对偶与广义勾股定理」,《信息几何与自然梯度》给出的定量结论是什么?
结论:指数族有自然参数 \(\theta\) 与期望参数 \(\eta\) 两套坐标,由凸函数 \(\psi\) 与其勒让德对偶 \(\phi\)(负熵)联系;费希尔度量是 \(\psi\) 的黑塞矩阵,KL 散度是 \(\psi\) 的布雷格曼散度(程序中都是 0.360388)。e-测地线是几何混合,m-测地线是算术混合;α-联络把两者连成一族。广义勾股定理在「m-投影到 e-平坦子流形」(或其对偶)时成立,散度方向不能颠倒:程序中正确方向逐位相等,颠倒后 1.412998 对 1.208860。迭代比例拟合是交替的 I-投影,每一步都满足勾股等式;它的极限是最小 KL 解而不是最小二乘解,后者可以出现负的格子。
数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.360388, 1.208860, 1.412998
| D(p||q*) = mutual information of p | 0.103351 |
| r in E D(p||r) D(p||q*) + D(q*||r) | reversed: D(r||p) vs D(r||q*)+D(q*||p) | |
| r1 | 0.986857 0.986857 | 1.412998 vs 1.208860 |
| r2 | 0.281492 0.281492 | 0.467797 vs 0.337878 |
| r3 | 0.365068 0.365068 | 0.489205 vs 0.371257 |
| r4 | 0.971558 0.971558 | 1.100003 vs 0.974635 |
| r5 | 0.912109 0.912109 | 0.811763 vs 0.806931 |
| min of D(p||r) over 20,000 random r in E | 0.111900 (>= D(p||q*)) |
| categorical (5 cells): max |Hessian of psi - Fisher| < 1e-10 | |
| Bregman_psi(theta2, theta1) | 0.360388; KL(p_theta1 || p_theta2) = 0.360388 |
| Legendre: psi(theta) + phi(eta) - theta.eta < 1e-12; phi(eta) = -entropy | -1.4606 |
| alpha-divergences D_a(p||q) | 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)): |
| alpha | -0.999: 0.360387 (-> KL(p||q)) |
| alpha | -0.500: 0.361804 |
| alpha = +0.000 | 0.367294 (4 x squared Hellinger, symmetric) |
| alpha = +0.500 | 0.377191 |
| alpha = +0.999 | 0.391952 (-> KL(q||p)) |
| KL(p||q) | 0.360388, KL(q||p) = 0.391987 |
核心公式
查看完整验算程序
# Dual flatness on a 3 x 3 table. The independence model E = {q(i,j) = a_i b_j} is an
# exponential family (e-flat). The m-projection of p onto E (minimise D(p||q), q in E)
# is the product of p's marginals, and the generalised Pythagorean theorem holds:
# D(p||r) = D(p||q*) + D(q*||r) for EVERY r in E. The identity is directional: with the
# arguments reversed it fails. Then the Legendre structure of the categorical family:
# psi(theta) = log sum exp, eta = grad psi, Hessian psi = Fisher, Bregman_psi = KL.
import math, random
def kl(p, q):
return sum(a * math.log(a / b) for a, b in zip(p, q) if a > 0)
def product(a, b):
return [x * y for x in a for y in b]
def marg(p):
return ([sum(p[3 * i + j] for j in range(3)) for i in range(3)],
[sum(p[3 * i + j] for i in range(3)) for j in range(3)])
def dirichlet(rng, k):
g = [rng.gammavariate(1.0, 1.0) for _ in range(k)]; s = sum(g)
return [x / s for x in g]
rng = random.Random(1975)
p = dirichlet(rng, 9); a, b = marg(p); q = product(a, b)
print(f"D(p||q*) = mutual information of p = {kl(p, q):.6f}")
print(" r in E D(p||r) D(p||q*) + D(q*||r) | reversed: D(r||p) vs"
" D(r||q*)+D(q*||p)")
for k in range(5):
r = product(dirichlet(rng, 3), dirichlet(rng, 3))
lhs, rhs = kl(p, r), kl(p, q) + kl(q, r)
print(f" r{k + 1} {lhs:.6f} {rhs:.6f} |"
f" {kl(r, p):.6f} vs {kl(r, q) + kl(q, p):.6f}")
best = min(kl(p, product(dirichlet(rng, 3), dirichlet(rng, 3))) for _ in range(20000))
print(f" min of D(p||r) over 20,000 random r in E: {best:.6f} (>= D(p||q*))")
theta = [0.3, -1.2, 0.8, 0.0] # natural parameters, 5th fixed at 0
def psi(t):
return math.log(1 + sum(math.exp(x) for x in t))
def eta(t): # expectation parameters = grad psi
z = 1 + sum(math.exp(x) for x in t); return [math.exp(x) / z for x in t]
def dist(t):
e = eta(t); return e + [1 - sum(e)]
h = 1e-5
hess = [[(eta([x + h * (i == j) for i, x in enumerate(theta)])[k]
- eta([x - h * (i == j) for i, x in enumerate(theta)])[k]) / (2 * h)
for j in range(4)] for k in range(4)]
e = eta(theta)
fisher = [[(e[i] if i == j else 0) - e[i] * e[j] for j in range(4)] for i in range(4)]
err = max(abs(hess[i][j] - fisher[i][j]) for i in range(4) for j in range(4))
print("categorical (5 cells): max |Hessian of psi - Fisher| "
+ ("< 1e-10" if err < 1e-10 else f"= {err:.1e}"))
t2 = [-0.5, 0.4, 0.1, 1.0]
breg = psi(t2) - psi(theta) - sum(a * (x - y) for a, x, y in zip(eta(theta), t2, theta))
print(f" Bregman_psi(theta2, theta1) = {breg:.6f}; KL(p_theta1 || p_theta2) ="
f" {kl(dist(theta), dist(t2)):.6f}")
phi = lambda t: sum(x * y for x, y in zip(theta_of(t), t)) - psi(theta_of(t))
def theta_of(et): # inverse map eta -> theta
last = 1 - sum(et); return [math.log(x / last) for x in et]
leg = abs(psi(theta) + phi(e) - sum(x * y for x, y in zip(theta, e)))
print(" Legendre: psi(theta) + phi(eta) - theta.eta "
+ ("< 1e-12;" if leg < 1e-12 else f"= {leg:.1e};")
+ f" phi(eta) = -entropy = {sum(x * math.log(x) for x in dist(theta)):.4f}")
print("alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):")
P1, Q1 = dist(theta), dist(t2)
for al in (-0.999, -0.5, 0.0, 0.5, 0.999):
da = 4 / (1 - al * al) * (1 - sum(x ** ((1 - al) / 2) * y ** ((1 + al) / 2)
for x, y in zip(P1, Q1)))
print(f" alpha = {al:+.3f}: {da:.6f}", end="")
print(" (-> KL(p||q))" if al < -0.9 else " (-> KL(q||p))" if al > 0.9 else
" (4 x squared Hellinger, symmetric)" if al == 0 else "")
print(f" KL(p||q) = {kl(P1, Q1):.6f}, KL(q||p) = {kl(Q1, P1):.6f}")唯一数据源与引用
范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 5 章「对偶平坦:指数族、勒让德对偶与广义勾股定理」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}