关于「自然梯度:不变性、速度与近似」,《信息几何与自然梯度》给出的定量结论是什么?

结论:自然梯度 \(G{-1}\nabla L\) 是费希尔度量下的最陡下降方向。它的不变性对连续时间的流严格成立,对离散步只在一阶成立:程序中步长 0.004 时坐标间差距为 \(2.5\times10{-7}\),步长 1 时达到 \(2.1\times10^6\)。作者框架所说的「证明」应改为:费希尔等于黑塞矩阵时自然梯度就是牛顿法。在病态逻辑回归上它的步数最少,但按计算量在条件良好时不如梯度下降;在一个小网络上,精确自然梯度步数最少而计算量约为 Adam 的六倍,K-FAC 未达到目标。经验费希尔不是费希尔:它的尺度随真实噪声改变一百倍,远离最优点时停滞。已发表的基准显示没有一个优化器在所有任务上占优。

数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.004, 2.1, 2.5

reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu 0.8647, sigma 1.8999
natural gradient: KL(flow || end point) per chart; spread = largest KL between charts
etaKA (mu, sigma)B (mu, log s)C (mu, s^2)D expectationspread
1.00021.60e-017.17e+004.84e-034.84e-032.1e+06
0.50041.57e-022.17e+001.98e-031.51e-039.5e+01
0.100202.17e-047.23e-047.89e-055.81e-054.5e-04
0.0201007.49e-061.45e-053.10e-062.27e-066.5e-06
0.0045002.83e-075.43e-071.16e-078.44e-082.5e-07
plain gradient: KL(flow || end point) per chart; spread = largest KL between charts
etaKA (mu, sigma)B (mu, log s)C (mu, s^2)D expectationspread
1.00022.51e+001.62e+018.50e-017.07e-012.4e+13
0.50041.83e+006.17e+004.59e-013.80e-017.4e+04
0.100203.54e-011.79e-021.03e-028.62e-039.0e-01
0.0201001.50e-021.88e-029.98e-021.56e-011.8e-01
0.0045001.32e-021.89e-021.09e-011.67e-011.9e-01
one natural step of size 1 in chart D lands on (mu, sigma) = (1.0000, 2.0000), the MLE

核心公式

\[d\theta \;\propto\; -\,G(\theta)^{-1}\nabla L(\theta) ,\]
查看完整验算程序
# Is the natural gradient "parametrisation invariant"? Fit N(mu, sigma^2) to data with
# mean 1 and variance 4 (average negative log-likelihood), starting at (0, 0.5), in four
# coordinate systems: A (mu, sigma), B (mu, log sigma), C (mu, sigma^2) and D the
# expectation parameters (mu, mu^2 + sigma^2). K steps of size eta, eta * K = 2 fixed.
# The natural-gradient FLOW is the same in every chart; discrete steps differ by O(eta).
# Plain gradient descent differs by O(1). At eta = 1 the chart matters a great deal.
import math

M, V = 1.0, 4.0
CHARTS = {   # to (mu, sigma), from (mu, sigma), Jacobian d(mu, sigma)/d(theta)
    "A (mu, sigma)": (lambda t: t, lambda m, s: (m, s),
                      lambda t: ((1, 0), (0, 1))),
    "B (mu, log s)": (lambda t: (t[0], math.exp(t[1])), lambda m, s: (m, math.log(s)),
                      lambda t: ((1, 0), (0, math.exp(t[1])))),
    "C (mu, s^2)": (lambda t: (t[0], math.sqrt(t[1])), lambda m, s: (m, s * s),
                    lambda t: ((1, 0), (0, 0.5 / math.sqrt(t[1])))),
    "D expectation": (lambda t: (t[0], math.sqrt(t[1] - t[0] ** 2)),
                      lambda m, s: (m, m * m + s * s),
                      lambda t: ((1, 0), (-t[0] / math.sqrt(t[1] - t[0] ** 2),
                                          0.5 / math.sqrt(t[1] - t[0] ** 2))))}

def grad_ms(m, s):                       # gradient of log s + ((M-m)^2 + V)/(2 s^2)
    q = (M - m) ** 2 + V
    return (-(M - m) / s ** 2, 1 / s - q / s ** 3)

def run(chart, eta, K, natural=True):
    to, frm, jac = CHARTS[chart]; t = list(frm(0.0, 0.5))
    for _ in range(K):
        m, s = to(t); gm, gs = grad_ms(m, s); J = jac(t)
        g = [J[0][0] * gm + J[1][0] * gs, J[0][1] * gm + J[1][1] * gs]   # J' grad
        if natural:                                       # F = J' diag(1/s^2, 2/s^2) J
            a = (J[0][0] ** 2 + 2 * J[1][0] ** 2) / s ** 2
            b = (J[0][0] * J[0][1] + 2 * J[1][0] * J[1][1]) / s ** 2
            d = (J[0][1] ** 2 + 2 * J[1][1] ** 2) / s ** 2
            det = a * d - b * b; g = [(d * g[0] - b * g[1]) / det,
                                      (a * g[1] - b * g[0]) / det]
        t = [t[0] - eta * g[0], t[1] - eta * g[1]]
        try:
            m, s = to(t)
        except (ValueError, OverflowError):
            return None
        if not (s > 0) or abs(m) > 1e6:
            return None
    return to(t)

def kl(a, b):
    return math.log(b[1] / a[1]) + (a[1] ** 2 + (a[0] - b[0]) ** 2) / (2 * b[1] ** 2) - .5

flow = run("A (mu, sigma)", 1e-4, 20_000)
print(f"reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu {flow[0]:.4f},"
      f" sigma {flow[1]:.4f}")
for natural in (True, False):
    print(("natural gradient" if natural else "plain gradient") +
          ": KL(flow || end point) per chart; spread = largest KL between charts")
    print("     eta     K    " + "  ".join(f"{c:>13s}" for c in CHARTS) + "   spread")
    for eta in (1.0, 0.5, 0.1, 0.02, 0.004):
        K = round(2 / eta); ends = [run(c, eta, K, natural) for c in CHARTS]
        cells = [f"{kl(flow, e):13.2e}" if e else f"{'diverged':>13s}" for e in ends]
        ok = [e for e in ends if e]
        spr = max(kl(a, b) for a in ok for b in ok) if len(ok) > 1 else float("nan")
        print(f"   {eta:5.3f} {K:5d}    " + "  ".join(cells) + f"  {spr:8.1e}")
one = run("D expectation", 1.0, 1)
print(f"one natural step of size 1 in chart D lands on (mu, sigma) = ({one[0]:.4f},"
      f" {one[1]:.4f}), the MLE")

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 4 章「自然梯度:不变性、速度与近似」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

本书其他章节结论