5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文
Chapter 1 Fisher Information: a Metric on Statistical Manifolds
From the score to a metric
Let a family of probability distributions be described by parameters , and write the log-likelihood as . Its derivative with respect to a parameter is called the score; under regularity conditions (differentiation under the integral sign is allowed and the support does not depend on the parameter) its expectation is zero. In two papers of 1922 and 1925, Fisher took the variance of the score as the measure of the “information” contained in one observation (Fisher 1922, 1925):
The second equality also relies on the regularity conditions. The information in independent observations is times that of one. A Bernoulli distribution with has information ; the normal distribution has information matrix in the coordinates .
Why can this matrix be regarded as a “metric”? The key is how it changes under a change of coordinates. If is another parametrisation with Jacobian , the chain rule gives
This is exactly the transformation law of a Riemannian metric tensor: the same tangent vector (an “infinitesimal change” of the distribution) has the same length in both coordinate systems. The Fisher information therefore defines a notion of length on parameter space that does not depend on coordinates, and a family of distributions together with this metric is called a statistical manifold. The idea was first put forward by Rao in 1945 (Rao 1945) and later developed systematically into information geometry by Amari, Čencov and others (Amari 1985, 2016; Amari and Nagaoka 2000). Introductory surveys include Nielsen’s (Nielsen 2020) and the tutorial by Ly and co-authors written for psychologists (Ly et al. 2017).
Note
On “coordinates”. In this book “coordinates”, “parametrisation” and “chart” all mean the same thing: giving each member of a family of distributions a label. A change of coordinates changes only the labels, not the distributions. Any quantity that depends on the labelling rather than on the distributions themselves is not a “geometric property” of the family.
The second-order expansion of the KL divergence
In 1951 Kullback and Leibler defined the “information for discrimination” between two distributions (Kullback and Leibler 1951), now called the KL divergence:
with equality if and only if (almost everywhere). Take to be a nearby member of the same family and expand about : the zeroth-order term is zero; the first-order term is the expected score, also zero; the second-order term is
So the Fisher metric is the “curvature” of the KL divergence near the diagonal. This also shows why the KL divergence itself does not depend on coordinates (it depends only on the two distributions), while its second-order approximation is a quadratic form that transforms as a tensor.
Calculation
# The Fisher metric as the second-order term of the KL divergence, and why KL is not a
# distance. D(t || t + dt) = 1/2 dt' G(t) dt + O(|dt|^3): checked for a Bernoulli model
# and for a normal model in (mu, sigma); then asymmetry and a failed triangle inequality,
# against the Fisher-Rao geodesic distance 2 arccos(sum sqrt(p q)), which is a metric.
import math
def kl_bern(p, q):
return p * math.log(p / q) + (1 - p) * math.log((1 - p) / (1 - q))
def kl_norm(m1, s1, m2, s2): # D(N(m1, s1^2) || N(m2, s2^2))
return math.log(s2 / s1) + (s1 ** 2 + (m1 - m2) ** 2) / (2 * s2 ** 2) - 0.5
def fr_bern(p, q): # Fisher-Rao distance on Bernoulli
return 2 * math.acos(math.sqrt(p * q) + math.sqrt((1 - p) * (1 - q)))
p = 0.3; g = 1 / (p * (1 - p))
print(f"Bernoulli p = {p}: Fisher information G = 1/(p(1-p)) = {g:.4f}")
print(" dp D(p||p+dp) 1/2 G dp^2 ratio D(p+dp||p)")
for dp in (0.1, 0.03, 0.01, 0.001):
d, q = kl_bern(p, p + dp), 0.5 * g * dp * dp
print(f" {dp:6.3f} {d:12.4e} {q:12.4e} {d / q:7.4f} {kl_bern(p + dp, p):12.4e}")
s = 1.0; G = ((1 / s ** 2, 0.0), (0.0, 2 / s ** 2))
print("normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2)")
print(" (dmu, dsigma) D 1/2 dt'G dt ratio")
for dm, ds in ((0.2, 0.2), (0.05, -0.05), (0.01, 0.02), (0.001, 0.001)):
d = kl_norm(0.0, s, dm, s + ds)
q = 0.5 * (G[0][0] * dm * dm + G[1][1] * ds * ds)
print(f" ({dm:6.3f},{ds:6.3f}) {d:12.4e} {q:12.4e} {d / q:7.4f}")
a, b, c = 0.1, 0.5, 0.9
ab, bc, ac = kl_bern(a, b), kl_bern(b, c), kl_bern(a, c)
print(f"triangle test on Bernoulli a = {a}, b = {b}, c = {c}:")
print(f" KL: D(a||c) = {ac:.4f} > D(a||b) + D(b||c) = {ab:.4f} + {bc:.4f}"
f" = {ab + bc:.4f}")
print(f" sqrt KL: {math.sqrt(ac):.4f} vs {math.sqrt(ab):.4f} + {math.sqrt(bc):.4f}"
f" = {math.sqrt(ab) + math.sqrt(bc):.4f}")
fab, fbc, fac = fr_bern(a, b), fr_bern(b, c), fr_bern(a, c)
print(f" Fisher-Rao: d(a,c) = {fac:.4f} = d(a,b) + d(b,c) = {fab + fbc:.4f}"
" (b lies on the geodesic)")
print(f"asymmetry: D(0.1||0.5) = {kl_bern(0.1, 0.5):.4f}, D(0.5||0.1) ="
f" {kl_bern(0.5, 0.1):.4f}")
worst = 0.0
for i in range(1, 40): # random-free grid check of the metric
for j in range(1, 40):
for k in range(1, 40):
x, y, z = i / 40, j / 40, k / 40
worst = max(worst, fr_bern(x, z) - fr_bern(x, y) - fr_bern(y, z))
print("Fisher-Rao triangle check on a 39^3 grid: max excess "
+ ("< 1e-12 (zero up to rounding)" if worst < 1e-12 else f"{worst:.1e}"))
# Bernoulli p = 0.3: Fisher information G = 1/(p(1-p)) = 4.7619
# dp D(p||p+dp) 1/2 G dp^2 ratio D(p+dp||p)
# 0.100 2.1601e-02 2.3810e-02 0.9072 2.2582e-02
# 0.030 2.0688e-03 2.1429e-03 0.9654 2.1046e-03
# 0.010 2.3517e-04 2.3810e-04 0.9877 2.3662e-04
# 0.001 2.3779e-06 2.3810e-06 0.9987 2.3794e-06
# normal (mu, sigma) at (0, 1): G = diag(1/sigma^2, 2/sigma^2)
# (dmu, dsigma) D 1/2 dt'G dt ratio
# ( 0.200, 0.200) 4.3433e-02 6.0000e-02 0.7239
# ( 0.050,-0.050) 4.1084e-03 3.7500e-03 1.0956
# ( 0.010, 0.020) 4.3508e-04 4.5000e-04 0.9668
# ( 0.001, 0.001) 1.4973e-06 1.5000e-06 0.9982
# triangle test on Bernoulli a = 0.1, b = 0.5, c = 0.9:
# KL: D(a||c) = 1.7578 > D(a||b) + D(b||c) = 0.3681 + 0.5108 = 0.8789
# sqrt KL: 1.3258 vs 0.6067 + 0.7147 = 1.3214
# Fisher-Rao: d(a,c) = 1.8546 = d(a,b) + d(b,c) = 1.8546 (b lies on the geodesic)
# asymmetry: D(0.1||0.5) = 0.3681, D(0.5||0.1) = 0.5108
# Fisher-Rao triangle check on a 39^3 grid: max excess < 1e-12 (zero up to rounding)The program verifies the expansion. The Bernoulli distribution at has information 4.7619; the ratio of KL to its quadratic approximation is 0.9072 at , 0.9877 at and 0.9987 at , the error being of the order of , so the approximation is poor once the third-order term matters. In the normal model the ratio is only 0.7239 at and 0.9982 at .
The second half of the program shows why the KL divergence is not a distance. First, it is asymmetric: but ; the first and last columns of the table also show that and differ, agreeing only to second order. Second, it fails the triangle inequality: with , , , is twice . Some think the square root gives a distance; the program shows that fails too: 1.3258 exceeds 1.3214.
A genuine distance is the geodesic distance obtained by integrating the metric along curves and taking the shortest path. For discrete distributions, the map sends the probability simplex onto part of a sphere of radius 2, and the Fisher metric becomes exactly the Euclidean metric on the sphere, so the Fisher–Rao distance has the closed form . Rao’s 1945 paper already proposed such a geodesic distance to measure how different two populations are (Rao 1945). In the program, , and lie on one geodesic, so ; a check of the triangle inequality on grid points finds no violation.
Note
The logarithmic score and KL. Book 26 discussed strictly proper scoring rules. The expected logarithmic score differs from the KL divergence only by an entropy term that does not depend on the forecast, so “scoring forecasts with the log score” is equivalent to “measuring the KL divergence between the forecast and the true distribution”. The connection is only pointed out here; the derivation is not repeated.
Čencov’s theorem: “unique” in what sense?
Since many metrics can be defined on parameter space (the Euclidean one, for instance), why the Fisher metric? Čencov’s answer is an invariance requirement (Čencov 1982). Let the sample space be finite, so that distributions are points of a simplex. Consider “refining” one category into several sub-categories in fixed proportions (mathematically a Markov embedding, or congruent embedding). The refinement leaves the content of statistical inference unchanged: knowing the refined outcome, one recovers the original by merging in fixed proportions, and conversely; so refinement is a sufficient transformation. Čencov proved that a metric invariant under all such embeddings can only be a constant multiple of the Fisher metric.
Evidence
The conditions of the theorem. Čencov’s original result (Russian original 1972, English translation 1982) concerns finite sample spaces and the probability simplex (Čencov 1982). In 1986 Campbell extended it to the cone of unnormalised positive measures (Campbell 1986). In 2015 Ay, Jost, Lê and Schwachhöfer introduced “parametrised measure models” and proved on general (possibly infinite) sample spaces that the second- and third-order tensors invariant under sufficient statistics can only be constant multiples of the Fisher metric and the Amari–Čencov tensor, deriving the monotonicity of Fisher information from a decomposition of Markov morphisms (Ay et al. 2015); their monograph gives a systematic treatment (Ay et al. 2017). In 2016 Bauer, Bruveris and Michor proved that on the space of smooth positive densities on a compact manifold without boundary, the only Riemannian metrics invariant under all diffeomorphisms are constant multiples of the Fisher–Rao metric (for manifolds of dimension at least 2) (Bauer et al. 2016). “Unique” is always relative to a stated invariance requirement and a stated space.
The program below checks these properties by hand on a three-category distribution: the Fisher length of a tangent vector is unchanged by every refinement, while the Euclidean length changes; merging categories (a statistic) can only shrink the Fisher length, with equality only when the merge is sufficient in that direction; finally, for ten Bernoulli trials, it computes how much information several statistics retain.
Calculation
# Chentsov's theorem in a finite sample space, checked by hand. A tangent vector v at a
# distribution p on {1..k} (sum v = 0) has Fisher length sum v_i^2 / p_i. (1) A congruent
# Markov embedding (split category i into parts with fixed ratios) leaves it unchanged;
# the Euclidean length sum v_i^2 does not. (2) Merging categories (a statistic) can only
# shrink it - the information inequality - with equality iff the merge is sufficient
# along v. (3) For n Bernoulli trials, the count keeps all n/(p(1-p)); other statistics
# keep less.
import math
def fisher(p, v):
return sum(x * x / q for x, q in zip(v, p))
def euclid(p, v):
return sum(x * x for x in v)
def split(p, v, i, ratios): # congruent embedding of category i
P = p[:i] + [p[i] * r for r in ratios] + p[i + 1:]
V = v[:i] + [v[i] * r for r in ratios] + v[i + 1:]
return P, V
def merge(p, v, groups):
return [sum(p[j] for j in g) for g in groups], [sum(v[j] for j in g) for g in groups]
p = [0.2, 0.3, 0.5]; v = [0.1, -0.04, -0.06]
print(f"p = {p}, v = {v}")
print(f" original: Fisher {fisher(p, v):.6f} Euclidean {euclid(p, v):.6f}")
for i, r in ((0, (0.5, 0.5)), (2, (0.1, 0.3, 0.6)), (1, (0.9, 0.1))):
P, V = split(p, v, i, r)
print(f" split cat {i + 1} as {r}: Fisher {fisher(P, V):.6f}"
f" Euclidean {euclid(P, V):.6f}")
for g in (((0, 1), (2,)), ((0,), (1, 2)), ((0, 2), (1,))):
P, V = merge(p, v, g)
print(f" merge {g}: Fisher {fisher(P, V):.6f} (<= {fisher(p, v):.6f})")
v2 = [0.15, -0.15 * 0.3 / 0.8, -0.15 * 0.5 / 0.8] # proportional to p on {2,3}
P, V = merge(p, v2, ((0,), (1, 2)))
print(f" v' = ({v2[0]:.3f}, {v2[1]:.4f}, {v2[2]:.4f}) is proportional to p on {{2,3}}:")
print(f" merge ((0,), (1, 2)): Fisher {fisher(P, V):.6f} = {fisher(p, v2):.6f}"
" (merge is sufficient along v')")
n, pr = 10, 0.3
full = n / (pr * (1 - pr))
def stat_info(pmf_of_stat, h=1e-5): # Fisher information of a statistic
a, b = pmf_of_stat(pr + h), pmf_of_stat(pr - h)
c = pmf_of_stat(pr)
return sum(((x - y) / (2 * h)) ** 2 / z for x, y, z in zip(a, b, c) if z > 0)
count = lambda q: [math.comb(n, k) * q ** k * (1 - q) ** (n - k) for k in range(n + 1)]
first = lambda q: [1 - q, q]
atleast3 = lambda q: [sum(count(q)[:3]), sum(count(q)[3:])]
parity = lambda q: [sum(count(q)[0::2]), sum(count(q)[1::2])]
print(f"n = {n} Bernoulli({pr}) trials: full-sample information {full:.4f}")
for lab, f in (("count of successes (sufficient)", count), ("first trial only", first),
("indicator count >= 3", atleast3), ("parity of count", parity)):
i = stat_info(f)
print(f" {lab:32s} {i:10.4g} ({i / full:7.2%})")
# p = [0.2, 0.3, 0.5], v = [0.1, -0.04, -0.06]
# original: Fisher 0.062533 Euclidean 0.015200
# split cat 1 as (0.5, 0.5): Fisher 0.062533 Euclidean 0.010200
# split cat 3 as (0.1, 0.3, 0.6): Fisher 0.062533 Euclidean 0.013256
# split cat 2 as (0.9, 0.1): Fisher 0.062533 Euclidean 0.014912
# merge ((0, 1), (2,)): Fisher 0.014400 (<= 0.062533)
# merge ((0,), (1, 2)): Fisher 0.062500 (<= 0.062533)
# merge ((0, 2), (1,)): Fisher 0.007619 (<= 0.062533)
# v' = (0.150, -0.0562, -0.0937) is proportional to p on {2,3}:
# merge ((0,), (1, 2)): Fisher 0.140625 = 0.140625 (merge is sufficient along v')
# n = 10 Bernoulli(0.3) trials: full-sample information 47.6190
# count of successes (sufficient) 47.62 (100.00%)
# first trial only 4.762 ( 10.00%)
# indicator count >= 3 30.14 ( 63.28%)
# parity of count 2.749e-05 ( 0.00%)The squared Fisher length of the tangent vector at is 0.062533, and still 0.062533 after each of three refinements; the squared Euclidean length goes from 0.015200 to 0.010200, 0.013256 and 0.014912 depending on the refinement. The Euclidean metric fails Čencov’s invariance requirement, and that is why it is ruled out: for the same statistical problem, the “distance” would change merely because one category was recorded as two.
Merging categories gives Fisher lengths of 0.014400, 0.062500 and 0.007619, none exceeding the original 0.062533. This is the information inequality (the data-processing inequality for Fisher information): no statistic can increase information. The second merge loses almost nothing because is nearly proportional to on the two merged categories; replacing by an exactly proportional gives 0.140625 before and after, with no loss at all. The example shows that sufficiency is relative to a direction: the same merge can be sufficient for some changes of the parameters and not for others.
Ten Bernoulli trials carry total information 47.62, all of which (100%) is kept by the sufficient statistic, the number of successes; the first trial alone keeps 10%; the indicator “at least 3 successes” keeps 63.28%; the parity of the number of successes keeps only , almost nothing. The last example is a reminder that whether a statistic is “useful” has nothing to do with whether it looks “complicated”; it depends on how sensitive its distribution is to the parameter.
What the metric can and cannot do
The Fisher metric gives the following chapters a common language: the Cramér–Rao bound of Chapter 2 is the inverse of the metric matrix; Jeffreys’s prior in Chapter 3 is the volume element induced by the metric; the natural gradient of Chapter 4 uses the metric to turn the gradient, a covector, into a tangent vector; and the dual connections of Chapter 5 and the singularities of Chapter 6 are built on it. But it is a local, second-order quantity. First, it describes how distinguishable distributions are near one point; further away, the KL divergence, the geodesic distance and the quadratic approximation can differ a great deal (the ratio of 0.7239 in the program). Second, it depends on the model: the same data have entirely different Fisher information in different model families; it describes how precisely the parameters could be estimated if the model were right, not whether the model is right. Third, when a model is not identifiable, the metric matrix becomes singular at some points and is no longer a Riemannian metric there; that is the subject of Chapter 6.
Hypothesis · not established · Setting learning rates by KL step
The author’s hypothesis (not established): in training medium-sized neural networks, capping each step by the KL change it causes in the model’s output distribution (rather than by the Euclidean length of the parameter update) substantially reduces the sensitivity of the final loss to how the network is parametrised (for example whether weights are normalised, or the scale of the activation functions). What would show it wrong: on a pre-registered set of parametrisation variants and tasks, the variance of final validation loss across parametrisations is not reduced by KL-capped steps relative to parameter-capped steps (the confidence interval for the difference includes zero or shows an increase). The test requires real training and is not done in this book; the discrete-step analysis of Chapter 4 shows that even KL-capped steps are invariant only to first order.
Summary
The Fisher information transforms as a tensor; it is a coordinate-free metric on parameter space and the second-order term of the KL divergence (a ratio of 0.9987 at in the program). The KL divergence is asymmetric and fails the triangle inequality, as does its square root; the Fisher–Rao geodesic distance is a distance. Čencov’s theorem shows that on finite sample spaces the only metric invariant under Markov embeddings is the Fisher metric, up to a constant, and the result has been extended to general sample spaces; “unique” is relative to this invariance requirement. Statistics can only lose Fisher information; sufficient statistics lose none.
References
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).