关于「费希尔信息:统计流形上的度量」,《信息几何与自然梯度》给出的定量结论是什么?
结论:费希尔信息 \(g_{ij} = E[\partial_i\ell\,\partial_j\ell]\) 按张量规律变换,是参数空间上与坐标无关的度量,也是 KL 散度的二阶项(程序中 \(dp = 0.001\) 时比值为 0.9987)。KL 散度不对称、不满足三角不等式,平方根也不满足;费希尔—拉奥测地距离才是距离。钦佐夫定理说明在有限样本空间上,对马尔可夫嵌入不变的度量只有费希尔度量(至多差常数),这一结论已推广到一般样本空间;「唯一」是相对于这一不变性要求而言的。统计量只能丢失费希尔信息,充分统计量不丢失。
数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.001, 0.9987
| Bernoulli p | 0.3: Fisher information G = 1/(p(1-p)) = 4.7619 |
| dp D(p||p+dp) | 1/2 G dp^2 ratio D(p+dp||p) |
| 0.100 | 2.1601e-02 2.3810e-02 0.9072 2.2582e-02 |
| 0.030 | 2.0688e-03 2.1429e-03 0.9654 2.1046e-03 |
| 0.010 | 2.3517e-04 2.3810e-04 0.9877 2.3662e-04 |
| 0.001 | 2.3779e-06 2.3810e-06 0.9987 2.3794e-06 |
| normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2) | |
| (dmu, dsigma) D | 1/2 dt'G dt ratio |
| ( 0.200, 0.200) | 4.3433e-02 6.0000e-02 0.7239 |
| ( 0.050,-0.050) | 4.1084e-03 3.7500e-03 1.0956 |
| ( 0.010, 0.020) | 4.3508e-04 4.5000e-04 0.9668 |
| ( 0.001, 0.001) | 1.4973e-06 1.5000e-06 0.9982 |
| triangle test on Bernoulli a | 0.1, b = 0.5, c = 0.9: |
| KL: D(a||c) | 1.7578 > D(a||b) + D(b||c) = 0.3681 + 0.5108 = 0.8789 |
| sqrt KL | 1.3258 vs 0.6067 + 0.7147 = 1.3214 |
| Fisher-Rao: d(a,c) | 1.8546 = d(a,b) + d(b,c) = 1.8546 (b lies on the geodesic) |
| asymmetry: D(0.1||0.5) | 0.3681, D(0.5||0.1) = 0.5108 |
| Fisher-Rao triangle check on a 39^3 grid: max excess < 1e-12 (zero up to rounding) |
核心公式
查看完整验算程序
# The Fisher metric as the second-order term of the KL divergence, and why KL is not a
# distance. D(t || t + dt) = 1/2 dt' G(t) dt + O(|dt|^3): checked for a Bernoulli model
# and for a normal model in (mu, sigma); then asymmetry and a failed triangle inequality,
# against the Fisher-Rao geodesic distance 2 arccos(sum sqrt(p q)), which is a metric.
import math
def kl_bern(p, q):
return p * math.log(p / q) + (1 - p) * math.log((1 - p) / (1 - q))
def kl_norm(m1, s1, m2, s2): # D(N(m1, s1^2) || N(m2, s2^2))
return math.log(s2 / s1) + (s1 ** 2 + (m1 - m2) ** 2) / (2 * s2 ** 2) - 0.5
def fr_bern(p, q): # Fisher-Rao distance on Bernoulli
return 2 * math.acos(math.sqrt(p * q) + math.sqrt((1 - p) * (1 - q)))
p = 0.3; g = 1 / (p * (1 - p))
print(f"Bernoulli p = {p}: Fisher information G = 1/(p(1-p)) = {g:.4f}")
print(" dp D(p||p+dp) 1/2 G dp^2 ratio D(p+dp||p)")
for dp in (0.1, 0.03, 0.01, 0.001):
d, q = kl_bern(p, p + dp), 0.5 * g * dp * dp
print(f" {dp:6.3f} {d:12.4e} {q:12.4e} {d / q:7.4f} {kl_bern(p + dp, p):12.4e}")
s = 1.0; G = ((1 / s ** 2, 0.0), (0.0, 2 / s ** 2))
print("normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2)")
print(" (dmu, dsigma) D 1/2 dt'G dt ratio")
for dm, ds in ((0.2, 0.2), (0.05, -0.05), (0.01, 0.02), (0.001, 0.001)):
d = kl_norm(0.0, s, dm, s + ds)
q = 0.5 * (G[0][0] * dm * dm + G[1][1] * ds * ds)
print(f" ({dm:6.3f},{ds:6.3f}) {d:12.4e} {q:12.4e} {d / q:7.4f}")
a, b, c = 0.1, 0.5, 0.9
ab, bc, ac = kl_bern(a, b), kl_bern(b, c), kl_bern(a, c)
print(f"triangle test on Bernoulli a = {a}, b = {b}, c = {c}:")
print(f" KL: D(a||c) = {ac:.4f} > D(a||b) + D(b||c) = {ab:.4f} + {bc:.4f}"
f" = {ab + bc:.4f}")
print(f" sqrt KL: {math.sqrt(ac):.4f} vs {math.sqrt(ab):.4f} + {math.sqrt(bc):.4f}"
f" = {math.sqrt(ab) + math.sqrt(bc):.4f}")
fab, fbc, fac = fr_bern(a, b), fr_bern(b, c), fr_bern(a, c)
print(f" Fisher-Rao: d(a,c) = {fac:.4f} = d(a,b) + d(b,c) = {fab + fbc:.4f}"
" (b lies on the geodesic)")
print(f"asymmetry: D(0.1||0.5) = {kl_bern(0.1, 0.5):.4f}, D(0.5||0.1) ="
f" {kl_bern(0.5, 0.1):.4f}")
worst = 0.0
for i in range(1, 40): # random-free grid check of the metric
for j in range(1, 40):
for k in range(1, 40):
x, y, z = i / 40, j / 40, k / 40
worst = max(worst, fr_bern(x, z) - fr_bern(x, y) - fr_bern(y, z))
print("Fisher-Rao triangle check on a 39^3 grid: max excess "
+ ("< 1e-12 (zero up to rounding)" if worst < 1e-12 else f"{worst:.1e}"))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 1 章「费希尔信息:统计流形上的度量」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}