What does Information Geometry and the Natural Gradient conclude, in numbers, about “The Natural Gradient: Invariance, Speed and Approximation”?
Answer: The natural gradient \(G{-1}\nabla L\) is the steepest-descent direction under the Fisher metric. Its invariance holds exactly for the continuous-time flow and only to first order for discrete steps: in the program the gap between charts is \(2.5\times10{-7}\) at step size 0.004 and \(2.1\times10^6\) at step size 1. The framework's "proof" should read: when the Fisher equals the Hessian, the natural gradient is Newton's method. On ill-conditioned logistic regression it takes the fewest steps but costs more than gradient descent when conditioning is good; on a small network the exact natural gradient takes the fewest steps at about six times Adam's cost, and K-FAC misses the target. The empirical Fisher is not the Fisher: its scale changes a hundredfold with the true noise, and it stalls far from the optimum. Published benchmarks show no optimiser dominating on every task.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.004, 2.1, 2.5
| reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu 0.8647, sigma 1.8999 | ||||||
| natural gradient: KL(flow || end point) per chart; spread = largest KL between charts | ||||||
| eta | K | A (mu, sigma) | B (mu, log s) | C (mu, s^2) | D expectation | spread |
| 1.000 | 2 | 1.60e-01 | 7.17e+00 | 4.84e-03 | 4.84e-03 | 2.1e+06 |
| 0.500 | 4 | 1.57e-02 | 2.17e+00 | 1.98e-03 | 1.51e-03 | 9.5e+01 |
| 0.100 | 20 | 2.17e-04 | 7.23e-04 | 7.89e-05 | 5.81e-05 | 4.5e-04 |
| 0.020 | 100 | 7.49e-06 | 1.45e-05 | 3.10e-06 | 2.27e-06 | 6.5e-06 |
| 0.004 | 500 | 2.83e-07 | 5.43e-07 | 1.16e-07 | 8.44e-08 | 2.5e-07 |
| plain gradient: KL(flow || end point) per chart; spread = largest KL between charts | ||||||
| eta | K | A (mu, sigma) | B (mu, log s) | C (mu, s^2) | D expectation | spread |
| 1.000 | 2 | 2.51e+00 | 1.62e+01 | 8.50e-01 | 7.07e-01 | 2.4e+13 |
| 0.500 | 4 | 1.83e+00 | 6.17e+00 | 4.59e-01 | 3.80e-01 | 7.4e+04 |
| 0.100 | 20 | 3.54e-01 | 1.79e-02 | 1.03e-02 | 8.62e-03 | 9.0e-01 |
| 0.020 | 100 | 1.50e-02 | 1.88e-02 | 9.98e-02 | 1.56e-01 | 1.8e-01 |
| 0.004 | 500 | 1.32e-02 | 1.89e-02 | 1.09e-01 | 1.67e-01 | 1.9e-01 |
| one natural step of size 1 in chart D lands on (mu, sigma) = (1.0000, 2.0000), the MLE |
Key formula
Show the full program
# Is the natural gradient "parametrisation invariant"? Fit N(mu, sigma^2) to data with
# mean 1 and variance 4 (average negative log-likelihood), starting at (0, 0.5), in four
# coordinate systems: A (mu, sigma), B (mu, log sigma), C (mu, sigma^2) and D the
# expectation parameters (mu, mu^2 + sigma^2). K steps of size eta, eta * K = 2 fixed.
# The natural-gradient FLOW is the same in every chart; discrete steps differ by O(eta).
# Plain gradient descent differs by O(1). At eta = 1 the chart matters a great deal.
import math
M, V = 1.0, 4.0
CHARTS = { # to (mu, sigma), from (mu, sigma), Jacobian d(mu, sigma)/d(theta)
"A (mu, sigma)": (lambda t: t, lambda m, s: (m, s),
lambda t: ((1, 0), (0, 1))),
"B (mu, log s)": (lambda t: (t[0], math.exp(t[1])), lambda m, s: (m, math.log(s)),
lambda t: ((1, 0), (0, math.exp(t[1])))),
"C (mu, s^2)": (lambda t: (t[0], math.sqrt(t[1])), lambda m, s: (m, s * s),
lambda t: ((1, 0), (0, 0.5 / math.sqrt(t[1])))),
"D expectation": (lambda t: (t[0], math.sqrt(t[1] - t[0] ** 2)),
lambda m, s: (m, m * m + s * s),
lambda t: ((1, 0), (-t[0] / math.sqrt(t[1] - t[0] ** 2),
0.5 / math.sqrt(t[1] - t[0] ** 2))))}
def grad_ms(m, s): # gradient of log s + ((M-m)^2 + V)/(2 s^2)
q = (M - m) ** 2 + V
return (-(M - m) / s ** 2, 1 / s - q / s ** 3)
def run(chart, eta, K, natural=True):
to, frm, jac = CHARTS[chart]; t = list(frm(0.0, 0.5))
for _ in range(K):
m, s = to(t); gm, gs = grad_ms(m, s); J = jac(t)
g = [J[0][0] * gm + J[1][0] * gs, J[0][1] * gm + J[1][1] * gs] # J' grad
if natural: # F = J' diag(1/s^2, 2/s^2) J
a = (J[0][0] ** 2 + 2 * J[1][0] ** 2) / s ** 2
b = (J[0][0] * J[0][1] + 2 * J[1][0] * J[1][1]) / s ** 2
d = (J[0][1] ** 2 + 2 * J[1][1] ** 2) / s ** 2
det = a * d - b * b; g = [(d * g[0] - b * g[1]) / det,
(a * g[1] - b * g[0]) / det]
t = [t[0] - eta * g[0], t[1] - eta * g[1]]
try:
m, s = to(t)
except (ValueError, OverflowError):
return None
if not (s > 0) or abs(m) > 1e6:
return None
return to(t)
def kl(a, b):
return math.log(b[1] / a[1]) + (a[1] ** 2 + (a[0] - b[0]) ** 2) / (2 * b[1] ** 2) - .5
flow = run("A (mu, sigma)", 1e-4, 20_000)
print(f"reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu {flow[0]:.4f},"
f" sigma {flow[1]:.4f}")
for natural in (True, False):
print(("natural gradient" if natural else "plain gradient") +
": KL(flow || end point) per chart; spread = largest KL between charts")
print(" eta K " + " ".join(f"{c:>13s}" for c in CHARTS) + " spread")
for eta in (1.0, 0.5, 0.1, 0.02, 0.004):
K = round(2 / eta); ends = [run(c, eta, K, natural) for c in CHARTS]
cells = [f"{kl(flow, e):13.2e}" if e else f"{'diverged':>13s}" for e in ends]
ok = [e for e in ends if e]
spr = max(kl(a, b) for a in ok for b in ok) if len(ok) > 1 else float("nan")
print(f" {eta:5.3f} {K:5d} " + " ".join(cells) + f" {spr:8.1e}")
one = run("D expectation", 1.0, 1)
print(f"one natural step of size 1 in chart D lands on (mu, sigma) = ({one[0]:.4f},"
f" {one[1]:.4f}), the MLE")Single source and citation
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 4: The Natural Gradient: Invariance, Speed and Approximation. 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}Other chapter answers in this book
- Chapter 1 · Fisher Information: a Metric on Statistical Manifolds
- Chapter 2 · The Cramér–Rao Bound: Attained, Not Attained and "Beaten"
- Chapter 3 · Jeffreys's Prior: Invariance and Its Price
- Chapter 5 · Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem
- Chapter 6 · Singular Learning Theory: When the Fisher Matrix Degenerates
- Chapter 8 · Practice and a Pre-Registered Test