What does Information Geometry and the Natural Gradient conclude, in numbers, about “Fisher Information: a Metric on Statistical Manifolds”?

Answer: The Fisher information \(g_{ij} = E[\partial_i\ell\,\partial_j\ell]\) transforms as a tensor; it is a coordinate-free metric on parameter space and the second-order term of the KL divergence (a ratio of 0.9987 at \(dp = 0.001\) in the program). The KL divergence is asymmetric and fails the triangle inequality, as does its square root; the Fisher–Rao geodesic distance is a distance. Čencov's theorem shows that on finite sample spaces the only metric invariant under Markov embeddings is the Fisher metric, up to a constant, and the result has been extended to general sample spaces; "unique" is relative to this invariance requirement. Statistics can only lose Fisher information; sufficient statistics lose none.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.001, 0.9987

Bernoulli p0.3: Fisher information G = 1/(p(1-p)) = 4.7619
dp D(p||p+dp)1/2 G dp^2 ratio D(p+dp||p)
0.1002.1601e-02 2.3810e-02 0.9072 2.2582e-02
0.0302.0688e-03 2.1429e-03 0.9654 2.1046e-03
0.0102.3517e-04 2.3810e-04 0.9877 2.3662e-04
0.0012.3779e-06 2.3810e-06 0.9987 2.3794e-06
normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2)
(dmu, dsigma) D1/2 dt'G dt ratio
( 0.200, 0.200)4.3433e-02 6.0000e-02 0.7239
( 0.050,-0.050)4.1084e-03 3.7500e-03 1.0956
( 0.010, 0.020)4.3508e-04 4.5000e-04 0.9668
( 0.001, 0.001)1.4973e-06 1.5000e-06 0.9982
triangle test on Bernoulli a0.1, b = 0.5, c = 0.9:
KL: D(a||c)1.7578 > D(a||b) + D(b||c) = 0.3681 + 0.5108 = 0.8789
sqrt KL1.3258 vs 0.6067 + 0.7147 = 1.3214
Fisher-Rao: d(a,c)1.8546 = d(a,b) + d(b,c) = 1.8546 (b lies on the geodesic)
asymmetry: D(0.1||0.5)0.3681, D(0.5||0.1) = 0.5108
Fisher-Rao triangle check on a 39^3 grid: max excess < 1e-12 (zero up to rounding)

Key formula

\[g_{ij}(\theta) = E_\theta\!\left[\partial_i \ell\, \partial_j \ell\right] = -E_\theta\!\left[\partial_i\partial_j \ell\right].\]
Show the full program
# The Fisher metric as the second-order term of the KL divergence, and why KL is not a
# distance. D(t || t + dt) = 1/2 dt' G(t) dt + O(|dt|^3): checked for a Bernoulli model
# and for a normal model in (mu, sigma); then asymmetry and a failed triangle inequality,
# against the Fisher-Rao geodesic distance 2 arccos(sum sqrt(p q)), which is a metric.
import math

def kl_bern(p, q):
    return p * math.log(p / q) + (1 - p) * math.log((1 - p) / (1 - q))

def kl_norm(m1, s1, m2, s2):                     # D(N(m1, s1^2) || N(m2, s2^2))
    return math.log(s2 / s1) + (s1 ** 2 + (m1 - m2) ** 2) / (2 * s2 ** 2) - 0.5

def fr_bern(p, q):                               # Fisher-Rao distance on Bernoulli
    return 2 * math.acos(math.sqrt(p * q) + math.sqrt((1 - p) * (1 - q)))

p = 0.3; g = 1 / (p * (1 - p))
print(f"Bernoulli p = {p}: Fisher information G = 1/(p(1-p)) = {g:.4f}")
print("   dp        D(p||p+dp)    1/2 G dp^2    ratio    D(p+dp||p)")
for dp in (0.1, 0.03, 0.01, 0.001):
    d, q = kl_bern(p, p + dp), 0.5 * g * dp * dp
    print(f"  {dp:6.3f}  {d:12.4e}  {q:12.4e}  {d / q:7.4f}  {kl_bern(p + dp, p):12.4e}")

s = 1.0; G = ((1 / s ** 2, 0.0), (0.0, 2 / s ** 2))
print("normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2)")
print("   (dmu, dsigma)     D           1/2 dt'G dt   ratio")
for dm, ds in ((0.2, 0.2), (0.05, -0.05), (0.01, 0.02), (0.001, 0.001)):
    d = kl_norm(0.0, s, dm, s + ds)
    q = 0.5 * (G[0][0] * dm * dm + G[1][1] * ds * ds)
    print(f"  ({dm:6.3f},{ds:6.3f})  {d:12.4e}  {q:12.4e}  {d / q:7.4f}")

a, b, c = 0.1, 0.5, 0.9
ab, bc, ac = kl_bern(a, b), kl_bern(b, c), kl_bern(a, c)
print(f"triangle test on Bernoulli a = {a}, b = {b}, c = {c}:")
print(f"   KL: D(a||c) = {ac:.4f} > D(a||b) + D(b||c) = {ab:.4f} + {bc:.4f}"
      f" = {ab + bc:.4f}")
print(f"   sqrt KL: {math.sqrt(ac):.4f} vs {math.sqrt(ab):.4f} + {math.sqrt(bc):.4f}"
      f" = {math.sqrt(ab) + math.sqrt(bc):.4f}")
fab, fbc, fac = fr_bern(a, b), fr_bern(b, c), fr_bern(a, c)
print(f"   Fisher-Rao: d(a,c) = {fac:.4f} = d(a,b) + d(b,c) = {fab + fbc:.4f}"
      " (b lies on the geodesic)")
print(f"asymmetry: D(0.1||0.5) = {kl_bern(0.1, 0.5):.4f}, D(0.5||0.1) ="
      f" {kl_bern(0.5, 0.1):.4f}")
worst = 0.0
for i in range(1, 40):                           # random-free grid check of the metric
    for j in range(1, 40):
        for k in range(1, 40):
            x, y, z = i / 40, j / 40, k / 40
            worst = max(worst, fr_bern(x, z) - fr_bern(x, y) - fr_bern(y, z))
print("Fisher-Rao triangle check on a 39^3 grid: max excess "
      + ("< 1e-12 (zero up to rounding)" if worst < 1e-12 else f"{worst:.1e}"))

Single source and citation

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 1: Fisher Information: a Metric on Statistical Manifolds. 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

Other chapter answers in this book