What does Information Geometry and the Natural Gradient conclude, in numbers, about “The Natural Gradient: Invariance, Speed and Approximation”?

Answer: The natural gradient \(G{-1}\nabla L\) is the steepest-descent direction under the Fisher metric. Its invariance holds exactly for the continuous-time flow and only to first order for discrete steps: in the program the gap between charts is \(2.5\times10{-7}\) at step size 0.004 and \(2.1\times10^6\) at step size 1. The framework's "proof" should read: when the Fisher equals the Hessian, the natural gradient is Newton's method. On ill-conditioned logistic regression it takes the fewest steps but costs more than gradient descent when conditioning is good; on a small network the exact natural gradient takes the fewest steps at about six times Adam's cost, and K-FAC misses the target. The empirical Fisher is not the Fisher: its scale changes a hundredfold with the true noise, and it stalls far from the optimum. Published benchmarks show no optimiser dominating on every task.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.004, 2.1, 2.5

reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu 0.8647, sigma 1.8999
natural gradient: KL(flow || end point) per chart; spread = largest KL between charts
etaKA (mu, sigma)B (mu, log s)C (mu, s^2)D expectationspread
1.00021.60e-017.17e+004.84e-034.84e-032.1e+06
0.50041.57e-022.17e+001.98e-031.51e-039.5e+01
0.100202.17e-047.23e-047.89e-055.81e-054.5e-04
0.0201007.49e-061.45e-053.10e-062.27e-066.5e-06
0.0045002.83e-075.43e-071.16e-078.44e-082.5e-07
plain gradient: KL(flow || end point) per chart; spread = largest KL between charts
etaKA (mu, sigma)B (mu, log s)C (mu, s^2)D expectationspread
1.00022.51e+001.62e+018.50e-017.07e-012.4e+13
0.50041.83e+006.17e+004.59e-013.80e-017.4e+04
0.100203.54e-011.79e-021.03e-028.62e-039.0e-01
0.0201001.50e-021.88e-029.98e-021.56e-011.8e-01
0.0045001.32e-021.89e-021.09e-011.67e-011.9e-01
one natural step of size 1 in chart D lands on (mu, sigma) = (1.0000, 2.0000), the MLE

Key formula

\[d\theta \;\propto\; -\,G(\theta)^{-1}\nabla L(\theta) ,\]
Show the full program
# Is the natural gradient "parametrisation invariant"? Fit N(mu, sigma^2) to data with
# mean 1 and variance 4 (average negative log-likelihood), starting at (0, 0.5), in four
# coordinate systems: A (mu, sigma), B (mu, log sigma), C (mu, sigma^2) and D the
# expectation parameters (mu, mu^2 + sigma^2). K steps of size eta, eta * K = 2 fixed.
# The natural-gradient FLOW is the same in every chart; discrete steps differ by O(eta).
# Plain gradient descent differs by O(1). At eta = 1 the chart matters a great deal.
import math

M, V = 1.0, 4.0
CHARTS = {   # to (mu, sigma), from (mu, sigma), Jacobian d(mu, sigma)/d(theta)
    "A (mu, sigma)": (lambda t: t, lambda m, s: (m, s),
                      lambda t: ((1, 0), (0, 1))),
    "B (mu, log s)": (lambda t: (t[0], math.exp(t[1])), lambda m, s: (m, math.log(s)),
                      lambda t: ((1, 0), (0, math.exp(t[1])))),
    "C (mu, s^2)": (lambda t: (t[0], math.sqrt(t[1])), lambda m, s: (m, s * s),
                    lambda t: ((1, 0), (0, 0.5 / math.sqrt(t[1])))),
    "D expectation": (lambda t: (t[0], math.sqrt(t[1] - t[0] ** 2)),
                      lambda m, s: (m, m * m + s * s),
                      lambda t: ((1, 0), (-t[0] / math.sqrt(t[1] - t[0] ** 2),
                                          0.5 / math.sqrt(t[1] - t[0] ** 2))))}

def grad_ms(m, s):                       # gradient of log s + ((M-m)^2 + V)/(2 s^2)
    q = (M - m) ** 2 + V
    return (-(M - m) / s ** 2, 1 / s - q / s ** 3)

def run(chart, eta, K, natural=True):
    to, frm, jac = CHARTS[chart]; t = list(frm(0.0, 0.5))
    for _ in range(K):
        m, s = to(t); gm, gs = grad_ms(m, s); J = jac(t)
        g = [J[0][0] * gm + J[1][0] * gs, J[0][1] * gm + J[1][1] * gs]   # J' grad
        if natural:                                       # F = J' diag(1/s^2, 2/s^2) J
            a = (J[0][0] ** 2 + 2 * J[1][0] ** 2) / s ** 2
            b = (J[0][0] * J[0][1] + 2 * J[1][0] * J[1][1]) / s ** 2
            d = (J[0][1] ** 2 + 2 * J[1][1] ** 2) / s ** 2
            det = a * d - b * b; g = [(d * g[0] - b * g[1]) / det,
                                      (a * g[1] - b * g[0]) / det]
        t = [t[0] - eta * g[0], t[1] - eta * g[1]]
        try:
            m, s = to(t)
        except (ValueError, OverflowError):
            return None
        if not (s > 0) or abs(m) > 1e6:
            return None
    return to(t)

def kl(a, b):
    return math.log(b[1] / a[1]) + (a[1] ** 2 + (a[0] - b[0]) ** 2) / (2 * b[1] ** 2) - .5

flow = run("A (mu, sigma)", 1e-4, 20_000)
print(f"reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu {flow[0]:.4f},"
      f" sigma {flow[1]:.4f}")
for natural in (True, False):
    print(("natural gradient" if natural else "plain gradient") +
          ": KL(flow || end point) per chart; spread = largest KL between charts")
    print("     eta     K    " + "  ".join(f"{c:>13s}" for c in CHARTS) + "   spread")
    for eta in (1.0, 0.5, 0.1, 0.02, 0.004):
        K = round(2 / eta); ends = [run(c, eta, K, natural) for c in CHARTS]
        cells = [f"{kl(flow, e):13.2e}" if e else f"{'diverged':>13s}" for e in ends]
        ok = [e for e in ends if e]
        spr = max(kl(a, b) for a in ok for b in ok) if len(ok) > 1 else float("nan")
        print(f"   {eta:5.3f} {K:5d}    " + "  ".join(cells) + f"  {spr:8.1e}")
one = run("D expectation", 1.0, 1)
print(f"one natural step of size 1 in chart D lands on (mu, sigma) = ({one[0]:.4f},"
      f" {one[1]:.4f}), the MLE")

Single source and citation

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 4: The Natural Gradient: Invariance, Speed and Approximation. 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

Other chapter answers in this book