What does Information Geometry and the Natural Gradient conclude, in numbers, about “Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem”?

Answer: An exponential family has two coordinate systems, the natural parameters \(\theta\) and the expectation parameters \(\eta\), linked by the convex function \(\psi\) and its Legendre dual \(\phi\) (the negative entropy); the Fisher metric is the Hessian of \(\psi\), and the KL divergence is the Bregman divergence of \(\psi\) (both 0.360388 in the program). e-geodesics are geometric mixtures and m-geodesics arithmetic mixtures; the α-connections join the two in a family. The generalised Pythagorean theorem holds for "m-projection onto an e-flat submanifold" (or its dual), and the direction of the divergence cannot be reversed: in the program the correct direction agrees digit for digit, while the reversed one gives 1.412998 against 1.208860. Iterative proportional fitting is alternating I-projection, each step satisfying the Pythagorean equation; its limit is the minimum-KL solution, not the least-squares one, which can contain negative cells.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.360388, 1.208860, 1.412998

D(p||q*) = mutual information of p0.103351
r in E D(p||r) D(p||q*) + D(q*||r) | reversed: D(r||p) vs D(r||q*)+D(q*||p)
r10.986857 0.986857 | 1.412998 vs 1.208860
r20.281492 0.281492 | 0.467797 vs 0.337878
r30.365068 0.365068 | 0.489205 vs 0.371257
r40.971558 0.971558 | 1.100003 vs 0.974635
r50.912109 0.912109 | 0.811763 vs 0.806931
min of D(p||r) over 20,000 random r in E0.111900 (>= D(p||q*))
categorical (5 cells): max |Hessian of psi - Fisher| < 1e-10
Bregman_psi(theta2, theta1)0.360388; KL(p_theta1 || p_theta2) = 0.360388
Legendre: psi(theta) + phi(eta) - theta.eta < 1e-12; phi(eta) = -entropy-1.4606
alpha-divergences D_a(p||q)4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):
alpha-0.999: 0.360387 (-> KL(p||q))
alpha-0.500: 0.361804
alpha = +0.0000.367294 (4 x squared Hellinger, symmetric)
alpha = +0.5000.377191
alpha = +0.9990.391952 (-> KL(q||p))
KL(p||q)0.360388, KL(q||p) = 0.391987

Key formula

\[B_\psi(\theta_2, \theta_1) = \psi(\theta_2) - \psi(\theta_1) - \nabla\psi(\theta_1)\cdot(\theta_2 - \theta_1) .\]
Show the full program
# Dual flatness on a 3 x 3 table. The independence model E = {q(i,j) = a_i b_j} is an
# exponential family (e-flat). The m-projection of p onto E (minimise D(p||q), q in E)
# is the product of p's marginals, and the generalised Pythagorean theorem holds:
# D(p||r) = D(p||q*) + D(q*||r) for EVERY r in E. The identity is directional: with the
# arguments reversed it fails. Then the Legendre structure of the categorical family:
# psi(theta) = log sum exp, eta = grad psi, Hessian psi = Fisher, Bregman_psi = KL.
import math, random

def kl(p, q):
    return sum(a * math.log(a / b) for a, b in zip(p, q) if a > 0)

def product(a, b):
    return [x * y for x in a for y in b]

def marg(p):
    return ([sum(p[3 * i + j] for j in range(3)) for i in range(3)],
            [sum(p[3 * i + j] for i in range(3)) for j in range(3)])

def dirichlet(rng, k):
    g = [rng.gammavariate(1.0, 1.0) for _ in range(k)]; s = sum(g)
    return [x / s for x in g]

rng = random.Random(1975)
p = dirichlet(rng, 9); a, b = marg(p); q = product(a, b)
print(f"D(p||q*) = mutual information of p = {kl(p, q):.6f}")
print("   r in E     D(p||r)    D(p||q*) + D(q*||r)  | reversed: D(r||p) vs"
      " D(r||q*)+D(q*||p)")
for k in range(5):
    r = product(dirichlet(rng, 3), dirichlet(rng, 3))
    lhs, rhs = kl(p, r), kl(p, q) + kl(q, r)
    print(f"   r{k + 1}         {lhs:.6f}   {rhs:.6f}            |"
          f"          {kl(r, p):.6f} vs {kl(r, q) + kl(q, p):.6f}")
best = min(kl(p, product(dirichlet(rng, 3), dirichlet(rng, 3))) for _ in range(20000))
print(f"   min of D(p||r) over 20,000 random r in E: {best:.6f} (>= D(p||q*))")

theta = [0.3, -1.2, 0.8, 0.0]                       # natural parameters, 5th fixed at 0
def psi(t):
    return math.log(1 + sum(math.exp(x) for x in t))
def eta(t):                                         # expectation parameters = grad psi
    z = 1 + sum(math.exp(x) for x in t); return [math.exp(x) / z for x in t]
def dist(t):
    e = eta(t); return e + [1 - sum(e)]
h = 1e-5
hess = [[(eta([x + h * (i == j) for i, x in enumerate(theta)])[k]
          - eta([x - h * (i == j) for i, x in enumerate(theta)])[k]) / (2 * h)
         for j in range(4)] for k in range(4)]
e = eta(theta)
fisher = [[(e[i] if i == j else 0) - e[i] * e[j] for j in range(4)] for i in range(4)]
err = max(abs(hess[i][j] - fisher[i][j]) for i in range(4) for j in range(4))
print("categorical (5 cells): max |Hessian of psi - Fisher| "
      + ("< 1e-10" if err < 1e-10 else f"= {err:.1e}"))
t2 = [-0.5, 0.4, 0.1, 1.0]
breg = psi(t2) - psi(theta) - sum(a * (x - y) for a, x, y in zip(eta(theta), t2, theta))
print(f"   Bregman_psi(theta2, theta1) = {breg:.6f};  KL(p_theta1 || p_theta2) ="
      f" {kl(dist(theta), dist(t2)):.6f}")
phi = lambda t: sum(x * y for x, y in zip(theta_of(t), t)) - psi(theta_of(t))
def theta_of(et):                                  # inverse map eta -> theta
    last = 1 - sum(et); return [math.log(x / last) for x in et]
leg = abs(psi(theta) + phi(e) - sum(x * y for x, y in zip(theta, e)))
print("   Legendre: psi(theta) + phi(eta) - theta.eta "
      + ("< 1e-12;" if leg < 1e-12 else f"= {leg:.1e};")
      + f" phi(eta) = -entropy = {sum(x * math.log(x) for x in dist(theta)):.4f}")
print("alpha-divergences D_a(p||q) = 4/(1-a^2) (1 - sum p^((1-a)/2) q^((1+a)/2)):")
P1, Q1 = dist(theta), dist(t2)
for al in (-0.999, -0.5, 0.0, 0.5, 0.999):
    da = 4 / (1 - al * al) * (1 - sum(x ** ((1 - al) / 2) * y ** ((1 + al) / 2)
                                      for x, y in zip(P1, Q1)))
    print(f"   alpha = {al:+.3f}: {da:.6f}", end="")
    print("   (-> KL(p||q))" if al < -0.9 else "   (-> KL(q||p))" if al > 0.9 else
          "   (4 x squared Hellinger, symmetric)" if al == 0 else "")
print(f"   KL(p||q) = {kl(P1, Q1):.6f}, KL(q||p) = {kl(Q1, P1):.6f}")

Single source and citation

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 5: Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem. 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

Other chapter answers in this book