5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文
Chapter 5 Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem
Two coordinate systems for an exponential family
The previous chapters used only the metric. The other half of information geometry is the connection: it specifies which curves count as “straight” (geodesics) and how vectors are transported along curves. Statistical manifolds carry two natural kinds of “straight line”, and they appear most clearly in exponential families.
An exponential family has densities , where is the natural parameter, the sufficient statistic and the log-partition (cumulant-generating) function, which is convex. Its gradient gives a second coordinate system — the expectation parameters ; its Hessian is the Fisher metric, . The Legendre transform of , , is a second convex function; it equals the negative entropy (relative to ) and satisfies and . The two coordinate systems and two potentials are dual through the Legendre transform (Amari and Nagaoka 2000; Amari 2016).
Bregman divergences. For a convex function , Bregman in 1967 defined (Bregman 1967)
For an exponential family, : the KL divergence is the Bregman divergence of the log-partition function. Banerjee and co-authors proved in 2005 that regular exponential families correspond one-to-one with a class of regular Bregman divergences (Banerjee et al. 2005).
Two kinds of flatness and dual connections
Walking a straight line in the natural parameters, , corresponds to a geometric mixture of distributions: . Such a curve is called an e-geodesic (e for exponential). Walking a straight line in the expectation parameters, or, for discrete distributions, directly in the probabilities, , corresponds to an arithmetic mixture and is called an m-geodesic (m for mixture). A submanifold that is an affine subspace in the coordinates is called e-flat (for example a subfamily cut out by linear constraints on the natural parameters, such as the independence model; a curved exponential family is a subfamily but is not e-flat); one that is affine in the mixture sense is called m-flat (for example the set of distributions with some moments fixed).
In 1985 Amari placed these two connections in a one-parameter family — the α-connections, with the e-connection, the m-connection and the Levi-Civita connection of the Fisher metric (Amari 1985); the α- and −α-connections are dual with respect to the Fisher metric. The definition of the α-connections uses a symmetric third-order tensor, the Amari–Čencov tensor mentioned in Chapter 1, which, like the Fisher metric, is invariant under sufficient statistics (Ay et al. 2015). Both the e- and m-connections have zero curvature, so exponential families are dually flat: there are two coordinate systems in which the two kinds of “straight line” are, respectively, straight lines.
The generalised Pythagorean theorem
The most useful result on a dually flat manifold is the generalised Pythagorean theorem. Let be an e-flat submanifold, any distribution, and the m-projection of onto , i.e. the point of minimising (equivalently, the m-geodesic joining and is orthogonal to ). Then for any point of ,
Dually, if is an m-flat submanifold and is the e-projection of onto (minimising ), then for any in , . Csiszár gave the latter form and its use in constrained minimisation in his 1975 “I-divergence geometry” (Csiszár 1975). In both forms, the direction of the divergence and the kind of projection must match; the author’s framework states only the equation, not this condition.
Calculation
# Dual flatness on a 3 x 3 table. The independence model E = {q(i,j) = a_i b_j} is an
# exponential family (e-flat). The m-projection of p onto E (minimise D(p||q), q in E)
# is the product of p's marginals, and the generalised Pythagorean theorem holds:
# D(p||r) = D(p||q*) + D(q*||r) for EVERY r in E. The identity is directional: with the
# arguments reversed it fails. Then the Legendre structure of the categorical family:
# psi(theta) = log sum exp, eta = grad psi, Hessian psi = Fisher, Bregman_psi = KL.
import math, random
def kl(p, q):
return sum(a * math.log(a / b) for a, b in zip(p, q) if a > 0)
def product(a, b):
return [x * y for x in a for y in b]
def marg(p):
return ([sum(p[3 * i + j] for j in range(3)) for i in range(3)],
[sum(p[3 * i + j] for i in range(3)) for j in range(3)])
def dirichlet(rng, k):
g = [rng.gammavariate(1.0, 1.0) for _ in range(k)]; s = sum(g)
return [x / s for x in g]
rng = random.Random(1975)
p = dirichlet(rng, 9); a, b = marg(p); q = product(a, b)
print(f"D(p||q*) = mutual information of p = {kl(p, q):.6f}")
print(" r in E D(p||r) D(p||q*) + D(q*||r) | reversed: D(r||p) vs"
" D(r||q*)+D(q*||p)")
for k in range(5):
r = product(dirichlet(rng, 3), dirichlet(rng, 3))
lhs, rhs = kl(p, r), kl(p, q) + kl(q, r)
print(f" r{k + 1} {lhs:.6f} {rhs:.6f} |"
f" {kl(r, p):.6f} vs {kl(r, q) + kl(q, p):.6f}")
best = min(kl(p, product(dirichlet(rng, 3), dirichlet(rng, 3))) for _ in range(20000))
print(f" min of D(p||r) over 20,000 random r in E: {best:.6f} (>= D(p||q*))")
theta = [0.3, -1.2, 0.8, 0.0] # natural parameters, 5th fixed at 0
def psi(t):
return math.log(1 + sum(math.exp(x) for x in t))
def eta(t): # expectation parameters = grad psi
z = 1 + sum(math.exp(x) for x in t); return [math.exp(x) / z for x in t]
def dist(t):
e = eta(t); return e + [1 - sum(e)]
h = 1e-5
hess = [[(eta([x + h * (i == j) for i, x in enumerate(theta)])[k]
- eta([x - h * (i == j) for i, x in enumerate(theta)])[k]) / (2 * h)
for j in range(4)] for k in range(4)]
e = eta(theta)
fisher = [[(e[i] if i == j else 0) - e[i] * e[j] for j in range(4)] for i in range(4)]
err = max(abs(hess[i][j] - fisher[i][j]) for i in range(4) for j in range(4))
print("categorical (5 cells): max |Hessian of psi - Fisher| "
+ ("< 1e-10" if err < 1e-10 else f"= {err:.1e}"))
t2 = [-0.5, 0.4, 0.1, 1.0]
breg = psi(t2) - psi(theta) - sum(a * (x - y) for a, x, y in zip(eta(theta), t2, theta))
print(f" Bregman_psi(theta2, theta1) = {breg:.6f}; KL(p_theta1 || p_theta2) ="
f" {kl(dist(theta), dist(t2)):.6f}")
phi = lambda t: sum(x * y for x, y in zip(theta_of(t), t)) - psi(theta_of(t))
def theta_of(et): # inverse map eta -> theta
last = 1 - sum(et); return [math.log(x / last) for x in et]
leg = abs(psi(theta) + phi(e) - sum(x * y for x, y in zip(theta, e)))
print(" Legendre: psi(theta) + phi(eta) - theta.eta "
+ ("< 1e-12;" if leg < 1e-12 else f"= {leg:.1e};")
+ f" phi(eta) = -entropy = {sum(x * math.log(x) for x in dist(theta)):.4f}")
print("alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):")
P1, Q1 = dist(theta), dist(t2)
for al in (-0.999, -0.5, 0.0, 0.5, 0.999):
da = 4 / (1 - al * al) * (1 - sum(x ** ((1 - al) / 2) * y ** ((1 + al) / 2)
for x, y in zip(P1, Q1)))
print(f" alpha = {al:+.3f}: {da:.6f}", end="")
print(" (-> KL(p||q))" if al < -0.9 else " (-> KL(q||p))" if al > 0.9 else
" (4 x squared Hellinger, symmetric)" if al == 0 else "")
print(f" KL(p||q) = {kl(P1, Q1):.6f}, KL(q||p) = {kl(Q1, P1):.6f}")
# D(p||q*) = mutual information of p = 0.103351
# r in E D(p||r) D(p||q*) + D(q*||r) | reversed: D(r||p) vs D(r||q*)+D(q*||p)
# r1 0.986857 0.986857 | 1.412998 vs 1.208860
# r2 0.281492 0.281492 | 0.467797 vs 0.337878
# r3 0.365068 0.365068 | 0.489205 vs 0.371257
# r4 0.971558 0.971558 | 1.100003 vs 0.974635
# r5 0.912109 0.912109 | 0.811763 vs 0.806931
# min of D(p||r) over 20,000 random r in E: 0.111900 (>= D(p||q*))
# categorical (5 cells): max |Hessian of psi - Fisher| < 1e-10
# Bregman_psi(theta2, theta1) = 0.360388; KL(p_theta1 || p_theta2) = 0.360388
# Legendre: psi(theta) + phi(eta) - theta.eta < 1e-12; phi(eta) = -entropy = -1.4606
# alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):
# alpha = -0.999: 0.360387 (-> KL(p||q))
# alpha = -0.500: 0.361804
# alpha = +0.000: 0.367294 (4 x squared Hellinger, symmetric)
# alpha = +0.500: 0.377191
# alpha = +0.999: 0.391952 (-> KL(q||p))
# KL(p||q) = 0.360388, KL(q||p) = 0.391987The program takes a distribution on a 3 × 3 contingency table, with the independence model (all distributions of the form , an exponential family and hence e-flat). The m-projection of onto is the product of its two marginals, and is the mutual information of , 0.103351. For five random in the two sides agree digit for digit (for example 0.986857 and 0.986857). With the direction of the divergences swapped, the equation fails: where is 1.412998, is 1.208860. Over 20,000 random , the smallest is 0.111900, not below the mutual information — the numerical sign that is the nearest point. Book 11 discussed mutual information from the information-theoretic side; here we see its geometric meaning: the “distance” of from the independence model.
The second half of the program verifies the Legendre structure: the numerical Hessian of for a five-category distribution differs from the Fisher matrix by less than ; the Bregman divergence and the KL divergence are both 0.360388; is below , and equals the negative entropy, −1.4606.
α-divergences. The family of divergences corresponding to the α-connections is . In the program it is 0.360387 with near −1, tending to ; 0.391952 near +1, tending to ; and 0.367294 at , four times the symmetric squared Hellinger distance (). The two directions of the KL divergence are the two ends of this family.
Projection algorithms: iterative proportional fitting and EM
Many classical algorithms can be read as alternating projections. In 1940 Deming and Stephan, wishing to make a sampled contingency table agree with known marginal totals, proposed iterative proportional fitting: rescale rows and columns in turn so that the row sums and column sums match their targets (Deming and Stephan 1940). In 1975 Csiszár proved that each rescaling is the I-projection of the current table onto a linear family (m-flat: “row sums fixed” or “column sums fixed”), and that the algorithm converges to the table that satisfies both sets of margins and has the smallest KL divergence from the starting table (Csiszár 1975). Csiszár and Tusnády in 1984 put such alternating-minimisation procedures in a unified geometric framework (Csiszár and Tusnády 1984); Amari in 1995 showed that the expectation–maximisation (EM) algorithm and its geometric version, the em algorithm, also alternate e- and m-projections between a data manifold and a model manifold (Amari 1995).
Calculation
# Iterative proportional fitting (Deming and Stephan 1940) as alternating I-projections
# (Csiszar 1975). Seed table s (4 x 4) and target margins; each row step and column step
# is the I-projection of the current table onto a linear family {q: margins fixed}. For
# the limit p*, which lies in both families, every step satisfies the Pythagorean
# identity D(p*||q_k) = D(p*||q_k+1) + D(q_k+1||q_k). The limit has the form a_i b_j s_ij
# (the e-flat family through s). Least squares onto the same margins gives a different
# table and can go negative.
import math
S = [[40, 2, 1, 1], [3, 30, 3, 2], [1, 2, 6, 1], [2, 1, 1, 4]]
tot = sum(map(sum, S)); S = [[x / tot for x in r] for r in S]
ROW, COL = [0.25, 0.25, 0.25, 0.25], [0.10, 0.20, 0.30, 0.40]
def kl(p, q):
return sum(a * math.log(a / b) for ra, rb in zip(p, q) for a, b in zip(ra, rb) if a)
def rows(q):
return [[x * ROW[i] / sum(r) for x in r] for i, r in enumerate(q)]
def cols(q):
cs = [sum(r[j] for r in q) for j in range(4)]
return [[x * COL[j] / cs[j] for j, x in enumerate(r)] for r in q]
seq = [S]
for k in range(400):
seq.append(rows(seq[-1]) if k % 2 == 0 else cols(seq[-1]))
ps = seq[-1]
print(f"D(p*||seed) = {kl(ps, S):.6f}; sum of step divergences D(q_k+1||q_k) ="
f" {sum(kl(seq[k + 1], seq[k]) for k in range(400)):.6f}")
print(" step D(p*||q_k) D(p*||q_k+1) + D(q_k+1||q_k) max margin error")
for k in range(6):
a, b = kl(ps, seq[k]), kl(ps, seq[k + 1]) + kl(seq[k + 1], seq[k])
q = seq[k + 1]
err = max(max(abs(sum(r) - ROW[i]) for i, r in enumerate(q)),
max(abs(sum(r[j] for r in q) - COL[j]) for j in range(4)))
print(f" {k + 1:3d} {a:.8f} {b:.8f} {err:.2e}")
ratio = [[ps[i][j] / S[i][j] for j in range(4)] for i in range(4)]
rank1 = max(abs(ratio[i][j] * ratio[0][0] - ratio[i][0] * ratio[0][j])
for i in range(4) for j in range(4))
print("limit / seed is a product a_i b_j: max 2x2 minor "
+ ("< 1e-12" if rank1 < 1e-12 else f"{rank1:.1e}"))
rs = [sum(r) for r in S]; cs = [sum(r[j] for r in S) for j in range(4)]
B = 0.0
for _ in range(200): # q = s + alpha_i + beta_j (unweighted LS)
al = [(ROW[i] - rs[i] - B) / 4 for i in range(4)]; A = sum(al)
be = [(COL[j] - cs[j] - A) / 4 for j in range(4)]; B = sum(be)
LS = [[S[i][j] + al[i] + be[j] for j in range(4)] for i in range(4)]
print("cell (1,1): seed {:.4f}, IPF {:.4f}, least squares {:.4f}".format(
S[0][0], ps[0][0], LS[0][0]))
print(f"smallest cell: IPF {min(map(min, ps)):.4f},"
f" least squares {min(map(min, LS)):+.4f}")
# D(p*||seed) = 0.731380; sum of step divergences D(q_k+1||q_k) = 0.731380
# step D(p*||q_k) D(p*||q_k+1) + D(q_k+1||q_k) max margin error
# 1 0.73137995 0.73137995 2.35e-01
# 2 0.46345473 0.46345473 1.53e-01
# 3 0.20169901 0.20169901 9.96e-02
# 4 0.07947240 0.07947240 7.65e-02
# 5 0.03256790 0.03256790 3.72e-02
# 6 0.01083802 0.01083802 2.82e-02
# limit / seed is a product a_i b_j: max 2x2 minor < 1e-12
# cell (1,1): seed 0.4000, IPF 0.0932, least squares 0.2625
# smallest cell: IPF 0.0015, least squares -0.0925With a 4 × 4 starting table and target margins, the limiting table has divergence 0.731380 from the starting table, exactly the sum of the step divergences over all steps. Every step satisfies the Pythagorean equation: at the first step , and is also 0.73137995; the later steps agree digit for digit likewise, while the margin error falls from 0.235. The ratio of the limiting table to the starting table has the product form (all 2 × 2 minors below ), i.e. it lies in the e-flat family through — the other half of the duality.
A historical detail is worth mentioning. Deming and Stephan’s title speaks of a “least squares adjustment”; they originally regarded the algorithm as an approximation to the least-squares solution, and only later was it understood that its limit is the minimum-KL solution, not the least-squares one. The program compares the two: with the same margin constraints, the cell that is 0.40 in the starting table becomes 0.0932 under iterative proportional fitting and 0.2625 under (unweighted) least squares; the smallest cell of the least-squares solution is −0.0925 — a negative “probability” — while the iterative-proportional-fitting solution stays positive (smallest 0.0015).
Hypothesis · not established · Diagnosing with the projection steps
The author’s hypothesis (not established): in real applications of iterative proportional fitting to calibrate survey weights, reporting the Pythagorean decomposition at each step as a diagnostic reveals mutually contradictory (infeasible) margin constraints earlier than reporting the margin error alone. What would show it wrong: on a pre-registered set of simulated calibration problems with feasible and infeasible constraints, a stopping rule based on the Pythagorean decomposition needs no fewer iterations to flag infeasible problems than a rule based on the margin error.
Summary
An exponential family has two coordinate systems, the natural parameters and the expectation parameters , linked by the convex function and its Legendre dual (the negative entropy); the Fisher metric is the Hessian of , and the KL divergence is the Bregman divergence of (both 0.360388 in the program). e-geodesics are geometric mixtures and m-geodesics arithmetic mixtures; the α-connections join the two in a family. The generalised Pythagorean theorem holds for “m-projection onto an e-flat submanifold” (or its dual), and the direction of the divergence cannot be reversed: in the program the correct direction agrees digit for digit, while the reversed one gives 1.412998 against 1.208860. Iterative proportional fitting is alternating I-projection, each step satisfying the Pythagorean equation; its limit is the minimum-KL solution, not the least-squares one, which can contain negative cells.
References
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).