5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文

Chapter 5 Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem

Two coordinate systems for an exponential family

The previous chapters used only the metric. The other half of information geometry is the connection: it specifies which curves count as “straight” (geodesics) and how vectors are transported along curves. Statistical manifolds carry two natural kinds of “straight line”, and they appear most clearly in exponential families.

An exponential family has densities p(x;θ)=exp⁡{θ⋅T(x)−ψ(θ)}h(x)p(x;\theta) = \exp\{\theta\cdot T(x) – \psi(\theta)\}\,h(x), where θ\theta is the natural parameter, TT the sufficient statistic and ψ\psi the log-partition (cumulant-generating) function, which is convex. Its gradient gives a second coordinate system — the expectation parameters η=∇ψ(θ)=Eθ[T]\eta = \nabla\psi(\theta) = E_\theta[T]; its Hessian is the Fisher metric, G(θ)=∇2ψ(θ)=Covθ(T)G(\theta) = \nabla^2\psi(\theta) = \mathrm{Cov}_\theta(T). The Legendre transform of ψ\psi, ϕ(η)=θ⋅η−ψ(θ)\phi(\eta) = \theta\cdot\eta – \psi(\theta), is a second convex function; it equals the negative entropy (relative to hh) and satisfies θ=∇ϕ(η)\theta = \nabla\phi(\eta) and ∇2ϕ(η)=G(θ)−1\nabla^2\phi(\eta) = G(\theta)^{-1}. The two coordinate systems and two potentials are dual through the Legendre transform (Amari and Nagaoka 2000; Amari 2016).

Bregman divergences. For a convex function ψ\psi, Bregman in 1967 defined (Bregman 1967)

Bψ(θ2,θ1)=ψ(θ2)−ψ(θ1)−∇ψ(θ1)⋅(θ2−θ1).B_\psi(\theta_2, \theta_1) = \psi(\theta_2) – \psi(\theta_1) – \nabla\psi(\theta_1)\cdot(\theta_2 – \theta_1) .

For an exponential family, Bψ(θ2,θ1)=D(pθ1∥pθ2)B_\psi(\theta_2, \theta_1) = D(p_{\theta_1}\,\|\,p_{\theta_2}): the KL divergence is the Bregman divergence of the log-partition function. Banerjee and co-authors proved in 2005 that regular exponential families correspond one-to-one with a class of regular Bregman divergences (Banerjee et al. 2005).

Two kinds of flatness and dual connections

Walking a straight line in the natural parameters, θ(t)=(1−t)θ1+tθ2\theta(t) = (1-t)\theta_1 + t\theta_2, corresponds to a geometric mixture of distributions: log⁡pt=(1−t)log⁡p1+tlog⁡p2−const\log p_t = (1-t)\log p_1 + t\log p_2 – \text{const}. Such a curve is called an e-geodesic (e for exponential). Walking a straight line in the expectation parameters, or, for discrete distributions, directly in the probabilities, pt=(1−t)p1+tp2p_t = (1-t)p_1 + t p_2, corresponds to an arithmetic mixture and is called an m-geodesic (m for mixture). A submanifold that is an affine subspace in the θ\theta coordinates is called e-flat (for example a subfamily cut out by linear constraints on the natural parameters, such as the independence model; a curved exponential family is a subfamily but is not e-flat); one that is affine in the mixture sense is called m-flat (for example the set of distributions with some moments fixed).

In 1985 Amari placed these two connections in a one-parameter family — the α-connections, with α=1\alpha = 1 the e-connection, α=−1\alpha = -1 the m-connection and α=0\alpha = 0 the Levi-Civita connection of the Fisher metric (Amari 1985); the α- and −α-connections are dual with respect to the Fisher metric. The definition of the α-connections uses a symmetric third-order tensor, the Amari–Čencov tensor mentioned in Chapter 1, which, like the Fisher metric, is invariant under sufficient statistics (Ay et al. 2015). Both the e- and m-connections have zero curvature, so exponential families are dually flat: there are two coordinate systems in which the two kinds of “straight line” are, respectively, straight lines.

The generalised Pythagorean theorem

The most useful result on a dually flat manifold is the generalised Pythagorean theorem. Let EE be an e-flat submanifold, pp any distribution, and q*q^* the m-projection of pp onto EE, i.e. the point of EE minimising D(p∥q)D(p\,\|\,q) (equivalently, the m-geodesic joining pp and q*q^* is orthogonal to EE). Then for any point rr of EE,

D(p∥r)=D(p∥q*)+D(q*∥r).D(p\,\|\,r) = D(p\,\|\,q^*) + D(q^*\,\|\,r) .

Dually, if MM is an m-flat submanifold and q*q^* is the e-projection of pp onto MM (minimising D(q∥p)D(q\,\|\,p)), then for any rr in MM, D(r∥p)=D(r∥q*)+D(q*∥p)D(r\,\|\,p) = D(r\,\|\,q^*) + D(q^*\,\|\,p). Csiszár gave the latter form and its use in constrained minimisation in his 1975 “I-divergence geometry” (Csiszár 1975). In both forms, the direction of the divergence and the kind of projection must match; the author’s framework states only the equation, not this condition.

Calculation

# Dual flatness on a 3 x 3 table. The independence model E = {q(i,j) = a_i b_j} is an
# exponential family (e-flat). The m-projection of p onto E (minimise D(p||q), q in E)
# is the product of p's marginals, and the generalised Pythagorean theorem holds:
# D(p||r) = D(p||q*) + D(q*||r) for EVERY r in E. The identity is directional: with the
# arguments reversed it fails. Then the Legendre structure of the categorical family:
# psi(theta) = log sum exp, eta = grad psi, Hessian psi = Fisher, Bregman_psi = KL.
import math, random

def kl(p, q):
    return sum(a * math.log(a / b) for a, b in zip(p, q) if a > 0)

def product(a, b):
    return [x * y for x in a for y in b]

def marg(p):
    return ([sum(p[3 * i + j] for j in range(3)) for i in range(3)],
            [sum(p[3 * i + j] for i in range(3)) for j in range(3)])

def dirichlet(rng, k):
    g = [rng.gammavariate(1.0, 1.0) for _ in range(k)]; s = sum(g)
    return [x / s for x in g]

rng = random.Random(1975)
p = dirichlet(rng, 9); a, b = marg(p); q = product(a, b)
print(f"D(p||q*) = mutual information of p = {kl(p, q):.6f}")
print("   r in E     D(p||r)    D(p||q*) + D(q*||r)  | reversed: D(r||p) vs"
      " D(r||q*)+D(q*||p)")
for k in range(5):
    r = product(dirichlet(rng, 3), dirichlet(rng, 3))
    lhs, rhs = kl(p, r), kl(p, q) + kl(q, r)
    print(f"   r{k + 1}         {lhs:.6f}   {rhs:.6f}            |"
          f"          {kl(r, p):.6f} vs {kl(r, q) + kl(q, p):.6f}")
best = min(kl(p, product(dirichlet(rng, 3), dirichlet(rng, 3))) for _ in range(20000))
print(f"   min of D(p||r) over 20,000 random r in E: {best:.6f} (>= D(p||q*))")

theta = [0.3, -1.2, 0.8, 0.0]                       # natural parameters, 5th fixed at 0
def psi(t):
    return math.log(1 + sum(math.exp(x) for x in t))
def eta(t):                                         # expectation parameters = grad psi
    z = 1 + sum(math.exp(x) for x in t); return [math.exp(x) / z for x in t]
def dist(t):
    e = eta(t); return e + [1 - sum(e)]
h = 1e-5
hess = [[(eta([x + h * (i == j) for i, x in enumerate(theta)])[k]
          - eta([x - h * (i == j) for i, x in enumerate(theta)])[k]) / (2 * h)
         for j in range(4)] for k in range(4)]
e = eta(theta)
fisher = [[(e[i] if i == j else 0) - e[i] * e[j] for j in range(4)] for i in range(4)]
err = max(abs(hess[i][j] - fisher[i][j]) for i in range(4) for j in range(4))
print("categorical (5 cells): max |Hessian of psi - Fisher| "
      + ("< 1e-10" if err < 1e-10 else f"= {err:.1e}"))
t2 = [-0.5, 0.4, 0.1, 1.0]
breg = psi(t2) - psi(theta) - sum(a * (x - y) for a, x, y in zip(eta(theta), t2, theta))
print(f"   Bregman_psi(theta2, theta1) = {breg:.6f};  KL(p_theta1 || p_theta2) ="
      f" {kl(dist(theta), dist(t2)):.6f}")
phi = lambda t: sum(x * y for x, y in zip(theta_of(t), t)) - psi(theta_of(t))
def theta_of(et):                                  # inverse map eta -> theta
    last = 1 - sum(et); return [math.log(x / last) for x in et]
leg = abs(psi(theta) + phi(e) - sum(x * y for x, y in zip(theta, e)))
print("   Legendre: psi(theta) + phi(eta) - theta.eta "
      + ("< 1e-12;" if leg < 1e-12 else f"= {leg:.1e};")
      + f" phi(eta) = -entropy = {sum(x * math.log(x) for x in dist(theta)):.4f}")
print("alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):")
P1, Q1 = dist(theta), dist(t2)
for al in (-0.999, -0.5, 0.0, 0.5, 0.999):
    da = 4 / (1 - al * al) * (1 - sum(x ** ((1 - al) / 2) * y ** ((1 + al) / 2)
                                      for x, y in zip(P1, Q1)))
    print(f"   alpha = {al:+.3f}: {da:.6f}", end="")
    print("   (-> KL(p||q))" if al < -0.9 else "   (-> KL(q||p))" if al > 0.9 else
          "   (4 x squared Hellinger, symmetric)" if al == 0 else "")
print(f"   KL(p||q) = {kl(P1, Q1):.6f}, KL(q||p) = {kl(Q1, P1):.6f}")
# D(p||q*) = mutual information of p = 0.103351
#    r in E     D(p||r)    D(p||q*) + D(q*||r)  | reversed: D(r||p) vs D(r||q*)+D(q*||p)
#    r1         0.986857   0.986857            |          1.412998 vs 1.208860
#    r2         0.281492   0.281492            |          0.467797 vs 0.337878
#    r3         0.365068   0.365068            |          0.489205 vs 0.371257
#    r4         0.971558   0.971558            |          1.100003 vs 0.974635
#    r5         0.912109   0.912109            |          0.811763 vs 0.806931
#    min of D(p||r) over 20,000 random r in E: 0.111900 (>= D(p||q*))
# categorical (5 cells): max |Hessian of psi - Fisher| < 1e-10
#    Bregman_psi(theta2, theta1) = 0.360388;  KL(p_theta1 || p_theta2) = 0.360388
#    Legendre: psi(theta) + phi(eta) - theta.eta < 1e-12; phi(eta) = -entropy = -1.4606
# alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):
#    alpha = -0.999: 0.360387   (-> KL(p||q))
#    alpha = -0.500: 0.361804
#    alpha = +0.000: 0.367294   (4 x squared Hellinger, symmetric)
#    alpha = +0.500: 0.377191
#    alpha = +0.999: 0.391952   (-> KL(q||p))
#    KL(p||q) = 0.360388, KL(q||p) = 0.391987

The program takes a distribution pp on a 3 × 3 contingency table, with EE the independence model (all distributions of the form aibja_ib_j, an exponential family and hence e-flat). The m-projection of pp onto EE is the product of its two marginals, and D(p∥q*)D(p\,\|\,q^*) is the mutual information of pp, 0.103351. For five random rr in EE the two sides agree digit for digit (for example 0.986857 and 0.986857). With the direction of the divergences swapped, the equation fails: where D(r∥p)D(r\,\|\,p) is 1.412998, D(r∥q*)+D(q*∥p)D(r\,\|\,q^*) + D(q^*\,\|\,p) is 1.208860. Over 20,000 random rr, the smallest D(p∥r)D(p\,\|\,r) is 0.111900, not below the mutual information — the numerical sign that q*q^* is the nearest point. Book 11 discussed mutual information from the information-theoretic side; here we see its geometric meaning: the “distance” of pp from the independence model.

The second half of the program verifies the Legendre structure: the numerical Hessian of ψ\psi for a five-category distribution differs from the Fisher matrix by less than 10−1010^{-10}; the Bregman divergence and the KL divergence are both 0.360388; ψ(θ)+ϕ(η)−θ⋅η\psi(\theta) + \phi(\eta) – \theta\cdot\eta is below 10−1210^{-12}, and ϕ\phi equals the negative entropy, −1.4606.

α-divergences. The family of divergences corresponding to the α-connections is Dα(p∥q)=41−α2(1−∑p(1−α)/2q(1+α)/2)D_\alpha(p\,\|\,q) = \frac{4}{1-\alpha^2}\big(1 – \sum p^{(1-\alpha)/2}q^{(1+\alpha)/2}\big). In the program it is 0.360387 with α\alpha near −1, tending to D(p∥q)=0.360388D(p\,\|\,q) = 0.360388; 0.391952 near +1, tending to D(q∥p)=0.391987D(q\,\|\,p) = 0.391987; and 0.367294 at α=0\alpha = 0, four times the symmetric squared Hellinger distance (4(1−∑pq)4(1 – \sum\sqrt{pq})). The two directions of the KL divergence are the two ends of this family.

Projection algorithms: iterative proportional fitting and EM

Many classical algorithms can be read as alternating projections. In 1940 Deming and Stephan, wishing to make a sampled contingency table agree with known marginal totals, proposed iterative proportional fitting: rescale rows and columns in turn so that the row sums and column sums match their targets (Deming and Stephan 1940). In 1975 Csiszár proved that each rescaling is the I-projection of the current table onto a linear family (m-flat: “row sums fixed” or “column sums fixed”), and that the algorithm converges to the table that satisfies both sets of margins and has the smallest KL divergence from the starting table (Csiszár 1975). Csiszár and Tusnády in 1984 put such alternating-minimisation procedures in a unified geometric framework (Csiszár and Tusnády 1984); Amari in 1995 showed that the expectation–maximisation (EM) algorithm and its geometric version, the em algorithm, also alternate e- and m-projections between a data manifold and a model manifold (Amari 1995).

Calculation

# Iterative proportional fitting (Deming and Stephan 1940) as alternating I-projections
# (Csiszar 1975). Seed table s (4 x 4) and target margins; each row step and column step
# is the I-projection of the current table onto a linear family {q: margins fixed}. For
# the limit p*, which lies in both families, every step satisfies the Pythagorean
# identity D(p*||q_k) = D(p*||q_k+1) + D(q_k+1||q_k). The limit has the form a_i b_j s_ij
# (the e-flat family through s). Least squares onto the same margins gives a different
# table and can go negative.
import math

S = [[40, 2, 1, 1], [3, 30, 3, 2], [1, 2, 6, 1], [2, 1, 1, 4]]
tot = sum(map(sum, S)); S = [[x / tot for x in r] for r in S]
ROW, COL = [0.25, 0.25, 0.25, 0.25], [0.10, 0.20, 0.30, 0.40]

def kl(p, q):
    return sum(a * math.log(a / b) for ra, rb in zip(p, q) for a, b in zip(ra, rb) if a)

def rows(q):
    return [[x * ROW[i] / sum(r) for x in r] for i, r in enumerate(q)]

def cols(q):
    cs = [sum(r[j] for r in q) for j in range(4)]
    return [[x * COL[j] / cs[j] for j, x in enumerate(r)] for r in q]

seq = [S]
for k in range(400):
    seq.append(rows(seq[-1]) if k % 2 == 0 else cols(seq[-1]))
ps = seq[-1]
print(f"D(p*||seed) = {kl(ps, S):.6f}; sum of step divergences D(q_k+1||q_k) ="
      f" {sum(kl(seq[k + 1], seq[k]) for k in range(400)):.6f}")
print("  step   D(p*||q_k)    D(p*||q_k+1) + D(q_k+1||q_k)   max margin error")
for k in range(6):
    a, b = kl(ps, seq[k]), kl(ps, seq[k + 1]) + kl(seq[k + 1], seq[k])
    q = seq[k + 1]
    err = max(max(abs(sum(r) - ROW[i]) for i, r in enumerate(q)),
              max(abs(sum(r[j] for r in q) - COL[j]) for j in range(4)))
    print(f"   {k + 1:3d}   {a:.8f}    {b:.8f}                     {err:.2e}")
ratio = [[ps[i][j] / S[i][j] for j in range(4)] for i in range(4)]
rank1 = max(abs(ratio[i][j] * ratio[0][0] - ratio[i][0] * ratio[0][j])
            for i in range(4) for j in range(4))
print("limit / seed is a product a_i b_j: max 2x2 minor "
      + ("< 1e-12" if rank1 < 1e-12 else f"{rank1:.1e}"))
rs = [sum(r) for r in S]; cs = [sum(r[j] for r in S) for j in range(4)]
B = 0.0
for _ in range(200):                     # q = s + alpha_i + beta_j (unweighted LS)
    al = [(ROW[i] - rs[i] - B) / 4 for i in range(4)]; A = sum(al)
    be = [(COL[j] - cs[j] - A) / 4 for j in range(4)]; B = sum(be)
LS = [[S[i][j] + al[i] + be[j] for j in range(4)] for i in range(4)]
print("cell (1,1): seed {:.4f}, IPF {:.4f}, least squares {:.4f}".format(
    S[0][0], ps[0][0], LS[0][0]))
print(f"smallest cell: IPF {min(map(min, ps)):.4f},"
      f" least squares {min(map(min, LS)):+.4f}")
# D(p*||seed) = 0.731380; sum of step divergences D(q_k+1||q_k) = 0.731380
#   step   D(p*||q_k)    D(p*||q_k+1) + D(q_k+1||q_k)   max margin error
#      1   0.73137995    0.73137995                     2.35e-01
#      2   0.46345473    0.46345473                     1.53e-01
#      3   0.20169901    0.20169901                     9.96e-02
#      4   0.07947240    0.07947240                     7.65e-02
#      5   0.03256790    0.03256790                     3.72e-02
#      6   0.01083802    0.01083802                     2.82e-02
# limit / seed is a product a_i b_j: max 2x2 minor < 1e-12
# cell (1,1): seed 0.4000, IPF 0.0932, least squares 0.2625
# smallest cell: IPF 0.0015, least squares -0.0925

With a 4 × 4 starting table ss and target margins, the limiting table p*p^* has divergence 0.731380 from the starting table, exactly the sum of the step divergences D(qk+1∥qk)D(q_{k+1}\,\|\,q_k) over all steps. Every step satisfies the Pythagorean equation: at the first step D(p*∥q0)=0.73137995D(p^*\,\|\,q_0) = 0.73137995, and D(p*∥q1)+D(q1∥q0)D(p^*\,\|\,q_1) + D(q_1\,\|\,q_0) is also 0.73137995; the later steps agree digit for digit likewise, while the margin error falls from 0.235. The ratio of the limiting table to the starting table has the product form aibja_ib_j (all 2 × 2 minors below 10−1210^{-12}), i.e. it lies in the e-flat family through ss — the other half of the duality.

A historical detail is worth mentioning. Deming and Stephan’s title speaks of a “least squares adjustment”; they originally regarded the algorithm as an approximation to the least-squares solution, and only later was it understood that its limit is the minimum-KL solution, not the least-squares one. The program compares the two: with the same margin constraints, the cell that is 0.40 in the starting table becomes 0.0932 under iterative proportional fitting and 0.2625 under (unweighted) least squares; the smallest cell of the least-squares solution is −0.0925 — a negative “probability” — while the iterative-proportional-fitting solution stays positive (smallest 0.0015).

Hypothesis · not established · Diagnosing with the projection steps

The author’s hypothesis (not established): in real applications of iterative proportional fitting to calibrate survey weights, reporting the Pythagorean decomposition D(p*∥qk)=D(p*∥qk+1)+D(qk+1∥qk)D(p^*\,\|\,q_k) = D(p^*\,\|\,q_{k+1}) + D(q_{k+1}\,\|\,q_k) at each step as a diagnostic reveals mutually contradictory (infeasible) margin constraints earlier than reporting the margin error alone. What would show it wrong: on a pre-registered set of simulated calibration problems with feasible and infeasible constraints, a stopping rule based on the Pythagorean decomposition needs no fewer iterations to flag infeasible problems than a rule based on the margin error.

Summary

An exponential family has two coordinate systems, the natural parameters θ\theta and the expectation parameters η\eta, linked by the convex function ψ\psi and its Legendre dual ϕ\phi (the negative entropy); the Fisher metric is the Hessian of ψ\psi, and the KL divergence is the Bregman divergence of ψ\psi (both 0.360388 in the program). e-geodesics are geometric mixtures and m-geodesics arithmetic mixtures; the α-connections join the two in a family. The generalised Pythagorean theorem holds for “m-projection onto an e-flat submanifold” (or its dual), and the direction of the divergence cannot be reversed: in the program the correct direction agrees digit for digit, while the reversed one gives 1.412998 against 1.208860. Iterative proportional fitting is alternating I-projection, each step satisfying the Pythagorean equation; its limit is the minimum-KL solution, not the least-squares one, which can contain negative cells.

References

Amari, Shun-ichi. 1985. Differential-Geometrical Methods in Statistics. Vol. 28. Lecture Notes in Statistics. Springer.
Amari, Shun-ichi. 1995. “Information Geometry of the EM and Em Algorithms for Neural Networks.” Neural Networks 8 (9): 1379–408.
Amari, Shun-ichi. 2016. Information Geometry and Its Applications. Vol. 194. Applied Mathematical Sciences. Springer.
Amari, Shun-ichi, and Hiroshi Nagaoka. 2000. Methods of Information Geometry. Vol. 191. Translations of Mathematical Monographs. American Mathematical Society; Oxford University Press.
Ay, Nihat, Jürgen Jost, Hông Vân Lê, and Lorenz Schwachhöfer. 2015. “Information Geometry and Sufficient Statistics.” Probability Theory and Related Fields 162 (1–2): 327–64. https://doi.org/10.1007/s00440-014-0574-8.
Banerjee, Arindam, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. 2005. “Clustering with Bregman Divergences.” Journal of Machine Learning Research 6: 1705–49.
Bregman, L. M. 1967. “The Relaxation Method of Finding the Common Point of Convex Sets and Its Application to the Solution of Problems in Convex Programming.” USSR Computational Mathematics and Mathematical Physics 7 (3): 200–217.
Csiszár, Imre. 1975. “II-Divergence Geometry of Probability Distributions and Minimization Problems.” Annals of Probability 3 (1): 146–58.
Csiszár, Imre, and Gábor Tusnády. 1984. “Information Geometry and Alternating Minimization Procedures.” Statistics & Decisions Supplement Issue 1: 205–37.
Deming, W. Edwards, and Frederick F. Stephan. 1940. “On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals Are Known.” Annals of Mathematical Statistics 11 (4): 427–44. https://doi.org/10.1214/aoms/1177731829.

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).