关于「自然梯度:不变性、速度与近似」,《信息几何与自然梯度》给出的定量结论是什么?
结论:自然梯度 \(G{-1}\nabla L\) 是费希尔度量下的最陡下降方向。它的不变性对连续时间的流严格成立,对离散步只在一阶成立:程序中步长 0.004 时坐标间差距为 \(2.5\times10{-7}\),步长 1 时达到 \(2.1\times10^6\)。作者框架所说的「证明」应改为:费希尔等于黑塞矩阵时自然梯度就是牛顿法。在病态逻辑回归上它的步数最少,但按计算量在条件良好时不如梯度下降;在一个小网络上,精确自然梯度步数最少而计算量约为 Adam 的六倍,K-FAC 未达到目标。经验费希尔不是费希尔:它的尺度随真实噪声改变一百倍,远离最优点时停滞。已发表的基准显示没有一个优化器在所有任务上占优。
数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.004, 2.1, 2.5
| reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu 0.8647, sigma 1.8999 | ||||||
| natural gradient: KL(flow || end point) per chart; spread = largest KL between charts | ||||||
| eta | K | A (mu, sigma) | B (mu, log s) | C (mu, s^2) | D expectation | spread |
| 1.000 | 2 | 1.60e-01 | 7.17e+00 | 4.84e-03 | 4.84e-03 | 2.1e+06 |
| 0.500 | 4 | 1.57e-02 | 2.17e+00 | 1.98e-03 | 1.51e-03 | 9.5e+01 |
| 0.100 | 20 | 2.17e-04 | 7.23e-04 | 7.89e-05 | 5.81e-05 | 4.5e-04 |
| 0.020 | 100 | 7.49e-06 | 1.45e-05 | 3.10e-06 | 2.27e-06 | 6.5e-06 |
| 0.004 | 500 | 2.83e-07 | 5.43e-07 | 1.16e-07 | 8.44e-08 | 2.5e-07 |
| plain gradient: KL(flow || end point) per chart; spread = largest KL between charts | ||||||
| eta | K | A (mu, sigma) | B (mu, log s) | C (mu, s^2) | D expectation | spread |
| 1.000 | 2 | 2.51e+00 | 1.62e+01 | 8.50e-01 | 7.07e-01 | 2.4e+13 |
| 0.500 | 4 | 1.83e+00 | 6.17e+00 | 4.59e-01 | 3.80e-01 | 7.4e+04 |
| 0.100 | 20 | 3.54e-01 | 1.79e-02 | 1.03e-02 | 8.62e-03 | 9.0e-01 |
| 0.020 | 100 | 1.50e-02 | 1.88e-02 | 9.98e-02 | 1.56e-01 | 1.8e-01 |
| 0.004 | 500 | 1.32e-02 | 1.89e-02 | 1.09e-01 | 1.67e-01 | 1.9e-01 |
| one natural step of size 1 in chart D lands on (mu, sigma) = (1.0000, 2.0000), the MLE |
核心公式
查看完整验算程序
# Is the natural gradient "parametrisation invariant"? Fit N(mu, sigma^2) to data with
# mean 1 and variance 4 (average negative log-likelihood), starting at (0, 0.5), in four
# coordinate systems: A (mu, sigma), B (mu, log sigma), C (mu, sigma^2) and D the
# expectation parameters (mu, mu^2 + sigma^2). K steps of size eta, eta * K = 2 fixed.
# The natural-gradient FLOW is the same in every chart; discrete steps differ by O(eta).
# Plain gradient descent differs by O(1). At eta = 1 the chart matters a great deal.
import math
M, V = 1.0, 4.0
CHARTS = { # to (mu, sigma), from (mu, sigma), Jacobian d(mu, sigma)/d(theta)
"A (mu, sigma)": (lambda t: t, lambda m, s: (m, s),
lambda t: ((1, 0), (0, 1))),
"B (mu, log s)": (lambda t: (t[0], math.exp(t[1])), lambda m, s: (m, math.log(s)),
lambda t: ((1, 0), (0, math.exp(t[1])))),
"C (mu, s^2)": (lambda t: (t[0], math.sqrt(t[1])), lambda m, s: (m, s * s),
lambda t: ((1, 0), (0, 0.5 / math.sqrt(t[1])))),
"D expectation": (lambda t: (t[0], math.sqrt(t[1] - t[0] ** 2)),
lambda m, s: (m, m * m + s * s),
lambda t: ((1, 0), (-t[0] / math.sqrt(t[1] - t[0] ** 2),
0.5 / math.sqrt(t[1] - t[0] ** 2))))}
def grad_ms(m, s): # gradient of log s + ((M-m)^2 + V)/(2 s^2)
q = (M - m) ** 2 + V
return (-(M - m) / s ** 2, 1 / s - q / s ** 3)
def run(chart, eta, K, natural=True):
to, frm, jac = CHARTS[chart]; t = list(frm(0.0, 0.5))
for _ in range(K):
m, s = to(t); gm, gs = grad_ms(m, s); J = jac(t)
g = [J[0][0] * gm + J[1][0] * gs, J[0][1] * gm + J[1][1] * gs] # J' grad
if natural: # F = J' diag(1/s^2, 2/s^2) J
a = (J[0][0] ** 2 + 2 * J[1][0] ** 2) / s ** 2
b = (J[0][0] * J[0][1] + 2 * J[1][0] * J[1][1]) / s ** 2
d = (J[0][1] ** 2 + 2 * J[1][1] ** 2) / s ** 2
det = a * d - b * b; g = [(d * g[0] - b * g[1]) / det,
(a * g[1] - b * g[0]) / det]
t = [t[0] - eta * g[0], t[1] - eta * g[1]]
try:
m, s = to(t)
except (ValueError, OverflowError):
return None
if not (s > 0) or abs(m) > 1e6:
return None
return to(t)
def kl(a, b):
return math.log(b[1] / a[1]) + (a[1] ** 2 + (a[0] - b[0]) ** 2) / (2 * b[1] ** 2) - .5
flow = run("A (mu, sigma)", 1e-4, 20_000)
print(f"reference flow (A, eta 1e-4, 20,000 steps) at time 2: mu {flow[0]:.4f},"
f" sigma {flow[1]:.4f}")
for natural in (True, False):
print(("natural gradient" if natural else "plain gradient") +
": KL(flow || end point) per chart; spread = largest KL between charts")
print(" eta K " + " ".join(f"{c:>13s}" for c in CHARTS) + " spread")
for eta in (1.0, 0.5, 0.1, 0.02, 0.004):
K = round(2 / eta); ends = [run(c, eta, K, natural) for c in CHARTS]
cells = [f"{kl(flow, e):13.2e}" if e else f"{'diverged':>13s}" for e in ends]
ok = [e for e in ends if e]
spr = max(kl(a, b) for a in ok for b in ok) if len(ok) > 1 else float("nan")
print(f" {eta:5.3f} {K:5d} " + " ".join(cells) + f" {spr:8.1e}")
one = run("D expectation", 1.0, 1)
print(f"one natural step of size 1 in chart D lands on (mu, sigma) = ({one[0]:.4f},"
f" {one[1]:.4f}), the MLE")唯一数据源与引用
范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 4 章「自然梯度:不变性、速度与近似」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}