关于「费希尔信息:统计流形上的度量」,《信息几何与自然梯度》给出的定量结论是什么?

结论:费希尔信息 \(g_{ij} = E[\partial_i\ell\,\partial_j\ell]\) 按张量规律变换,是参数空间上与坐标无关的度量,也是 KL 散度的二阶项(程序中 \(dp = 0.001\) 时比值为 0.9987)。KL 散度不对称、不满足三角不等式,平方根也不满足;费希尔—拉奥测地距离才是距离。钦佐夫定理说明在有限样本空间上,对马尔可夫嵌入不变的度量只有费希尔度量(至多差常数),这一结论已推广到一般样本空间;「唯一」是相对于这一不变性要求而言的。统计量只能丢失费希尔信息,充分统计量不丢失。

数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.001, 0.9987

Bernoulli p0.3: Fisher information G = 1/(p(1-p)) = 4.7619
dp D(p||p+dp)1/2 G dp^2 ratio D(p+dp||p)
0.1002.1601e-02 2.3810e-02 0.9072 2.2582e-02
0.0302.0688e-03 2.1429e-03 0.9654 2.1046e-03
0.0102.3517e-04 2.3810e-04 0.9877 2.3662e-04
0.0012.3779e-06 2.3810e-06 0.9987 2.3794e-06
normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2)
(dmu, dsigma) D1/2 dt'G dt ratio
( 0.200, 0.200)4.3433e-02 6.0000e-02 0.7239
( 0.050,-0.050)4.1084e-03 3.7500e-03 1.0956
( 0.010, 0.020)4.3508e-04 4.5000e-04 0.9668
( 0.001, 0.001)1.4973e-06 1.5000e-06 0.9982
triangle test on Bernoulli a0.1, b = 0.5, c = 0.9:
KL: D(a||c)1.7578 > D(a||b) + D(b||c) = 0.3681 + 0.5108 = 0.8789
sqrt KL1.3258 vs 0.6067 + 0.7147 = 1.3214
Fisher-Rao: d(a,c)1.8546 = d(a,b) + d(b,c) = 1.8546 (b lies on the geodesic)
asymmetry: D(0.1||0.5)0.3681, D(0.5||0.1) = 0.5108
Fisher-Rao triangle check on a 39^3 grid: max excess < 1e-12 (zero up to rounding)

核心公式

\[g_{ij}(\theta) = E_\theta\!\left[\partial_i \ell\, \partial_j \ell\right] = -E_\theta\!\left[\partial_i\partial_j \ell\right].\]
查看完整验算程序
# The Fisher metric as the second-order term of the KL divergence, and why KL is not a
# distance. D(t || t + dt) = 1/2 dt' G(t) dt + O(|dt|^3): checked for a Bernoulli model
# and for a normal model in (mu, sigma); then asymmetry and a failed triangle inequality,
# against the Fisher-Rao geodesic distance 2 arccos(sum sqrt(p q)), which is a metric.
import math

def kl_bern(p, q):
    return p * math.log(p / q) + (1 - p) * math.log((1 - p) / (1 - q))

def kl_norm(m1, s1, m2, s2):                     # D(N(m1, s1^2) || N(m2, s2^2))
    return math.log(s2 / s1) + (s1 ** 2 + (m1 - m2) ** 2) / (2 * s2 ** 2) - 0.5

def fr_bern(p, q):                               # Fisher-Rao distance on Bernoulli
    return 2 * math.acos(math.sqrt(p * q) + math.sqrt((1 - p) * (1 - q)))

p = 0.3; g = 1 / (p * (1 - p))
print(f"Bernoulli p = {p}: Fisher information G = 1/(p(1-p)) = {g:.4f}")
print("   dp        D(p||p+dp)    1/2 G dp^2    ratio    D(p+dp||p)")
for dp in (0.1, 0.03, 0.01, 0.001):
    d, q = kl_bern(p, p + dp), 0.5 * g * dp * dp
    print(f"  {dp:6.3f}  {d:12.4e}  {q:12.4e}  {d / q:7.4f}  {kl_bern(p + dp, p):12.4e}")

s = 1.0; G = ((1 / s ** 2, 0.0), (0.0, 2 / s ** 2))
print("normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2)")
print("   (dmu, dsigma)     D           1/2 dt'G dt   ratio")
for dm, ds in ((0.2, 0.2), (0.05, -0.05), (0.01, 0.02), (0.001, 0.001)):
    d = kl_norm(0.0, s, dm, s + ds)
    q = 0.5 * (G[0][0] * dm * dm + G[1][1] * ds * ds)
    print(f"  ({dm:6.3f},{ds:6.3f})  {d:12.4e}  {q:12.4e}  {d / q:7.4f}")

a, b, c = 0.1, 0.5, 0.9
ab, bc, ac = kl_bern(a, b), kl_bern(b, c), kl_bern(a, c)
print(f"triangle test on Bernoulli a = {a}, b = {b}, c = {c}:")
print(f"   KL: D(a||c) = {ac:.4f} > D(a||b) + D(b||c) = {ab:.4f} + {bc:.4f}"
      f" = {ab + bc:.4f}")
print(f"   sqrt KL: {math.sqrt(ac):.4f} vs {math.sqrt(ab):.4f} + {math.sqrt(bc):.4f}"
      f" = {math.sqrt(ab) + math.sqrt(bc):.4f}")
fab, fbc, fac = fr_bern(a, b), fr_bern(b, c), fr_bern(a, c)
print(f"   Fisher-Rao: d(a,c) = {fac:.4f} = d(a,b) + d(b,c) = {fab + fbc:.4f}"
      " (b lies on the geodesic)")
print(f"asymmetry: D(0.1||0.5) = {kl_bern(0.1, 0.5):.4f}, D(0.5||0.1) ="
      f" {kl_bern(0.5, 0.1):.4f}")
worst = 0.0
for i in range(1, 40):                           # random-free grid check of the metric
    for j in range(1, 40):
        for k in range(1, 40):
            x, y, z = i / 40, j / 40, k / 40
            worst = max(worst, fr_bern(x, z) - fr_bern(x, y) - fr_bern(y, z))
print("Fisher-Rao triangle check on a 39^3 grid: max excess "
      + ("< 1e-12 (zero up to rounding)" if worst < 1e-12 else f"{worst:.1e}"))

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 1 章「费希尔信息:统计流形上的度量」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

本书其他章节结论