关于「奇异学习理论:当费希尔矩阵退化」,《信息几何与自然梯度》给出的定量结论是什么?
结论:当参数不可辨识、费希尔矩阵退化时,BIC 的 \(\tfrac d2\log n\) 不对:贝叶斯自由能的主项是 \(\lambda\log n - (m-1)\log\log n\),\(\lambda \le d/2\) 是实对数典范阈。程序直接积分得到 \(y = abx\) 的 \(\lambda \approx 1/2\)(BIC 用 1),\(y = a^2x\) 的 \(\lambda = 1/4\);\(\log\log n\) 项使有限 \(n\) 下的斜率偏低。WBIC 无须知道 \(\lambda\),它与精确自由能的误差(0.27–0.80)远小于 BIC(3.5–6.4)。奇异性解释深度学习泛化的说法尚未确立:理论描述的是贝叶斯后验的渐近行为,与随机梯度训练的联系仍在研究中。
数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库
| model d BIC d/2 fitted lambda fitted m theory (lambda, m) | |
| c x | 1 0.5 0.5000 1.00 (1/2, 1) |
| a x1 + b x2 | 2 1.0 1.0000 1.00 (1, 1) |
| a^2 x | 1 0.5 0.2500 1.00 (1/4, 1) |
| a b x | 2 1.0 0.4959 1.86 (1/2, 2) |
| a b x, ab0 | 0.25 2 1.0 0.4999 1.00 (1/2, 1) |
| a b c x | 3 1.5 0.4899 2.68 (1/2, 3) |
| naive slope dF/dlog n between n | 10^10 and 10^12 (what a plot would suggest): |
| c x | 0.5000 |
| a x1 + b x2 | 1.0000 |
| a^2 x | 0.2500 |
| a b x | 0.4623 |
| a b x, ab0 | 0.25 0.5000 |
| a b c x | 0.4244 |
核心公式
查看完整验算程序
# The learning coefficient, computed rather than assumed. For y = f(x, w) + N(0, 1)
# noise, x ~ N(0, 1), uniform prior on [-1, 1]^d, the "average" free energy is
# F(n) = -log Int exp(-n K(w)) prior(w) dw with K(w) = E (f(x,w) - f(x,w0))^2 / 2.
# Watanabe: F(n) = lambda log n - (m - 1) log log n + O(1). We integrate exactly
# (inner integrals in closed form, outer on a log grid) for n = 10^4 .. 10^12 and fit
# lambda and m by least squares; BIC would use d/2.
import math
def erf_int(s): # (1/2) Int_{-1}^{1} exp(-s t^2) dt
return 1.0 if s < 1e-12 else 0.5 * math.sqrt(math.pi / s) * math.erf(math.sqrt(s))
U = [-45 + 45 * (i + 0.5) / 3000 for i in range(3000)] # a = e^u on (0, 1]
DU = 45 / 3000
def logZ(model, n):
if model == "c x": # regular, d = 1
return math.log(erf_int(n / 2))
if model == "a x1 + b x2": # regular, d = 2
return 2 * math.log(erf_int(n / 2))
if model == "a^2 x": # d = 1, K = a^4 / 2
return math.log(sum(math.exp(-n * math.exp(4 * u) / 2) * math.exp(u)
for u in U) * DU)
if model == "a b x": # d = 2, K = (ab)^2 / 2
return math.log(sum(erf_int(n * math.exp(2 * u) / 2) * math.exp(u)
for u in U) * DU)
if model == "a b x, ab0 = 0.25": # true function nonzero
c0, tot, M = 0.25, 0.0, 4000
for i in range(M):
a = -1 + 2 * (i + 0.5) / M; r = math.sqrt(n) * abs(a)
lo, hi = (-1 - c0 / a) * r, (1 - c0 / a) * r
if a < 0:
lo, hi = -hi, -lo
p = 0.5 * (math.erf(hi / math.sqrt(2)) - math.erf(lo / math.sqrt(2)))
tot += 0.5 * math.sqrt(2 * math.pi) / r * p * (1 / M)
return math.log(tot)
if model == "a b c x": # d = 3, K = (abc)^2 / 2
V = U[::10]; dv = DU * 10
return math.log(sum(erf_int(n * math.exp(2 * (u + v)) / 2) * math.exp(u + v)
for u in V for v in V) * dv * dv)
NS = [10 ** (k / 2) for k in range(8, 25)]
print("model d BIC d/2 fitted lambda fitted m theory (lambda, m)")
THEORY = {"c x": (1, "1/2", 1), "a x1 + b x2": (2, "1", 1), "a^2 x": (1, "1/4", 1),
"a b x": (2, "1/2", 2), "a b x, ab0 = 0.25": (2, "1/2", 1),
"a b c x": (3, "1/2", 3)}
res = {}
for model, (d, lam, m) in THEORY.items():
F = [-logZ(model, n) for n in NS]
X = [(math.log(n), -math.log(math.log(n)), 1.0) for n in NS] # F = l*x1 + k*x2 + c
A = [[sum(x[i] * x[j] for x in X) for j in range(3)] for i in range(3)]
b = [sum(x[i] * f for x, f in zip(X, F)) for i in range(3)]
for c in range(3): # Gaussian elimination
for r in range(c + 1, 3):
t = A[r][c] / A[c][c]; A[r] = [u - t * v for u, v in zip(A[r], A[c])]
b[r] -= t * b[c]
sol = [0.0] * 3
for r in range(2, -1, -1):
sol[r] = (b[r] - sum(A[r][k] * sol[k] for k in range(r + 1, 3))) / A[r][r]
res[model] = F
print(f"{model:18s} {d:2d} {d / 2:5.1f} {sol[0]:8.4f} {sol[1] + 1:6.2f}"
f" ({lam}, {m})")
print("naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):")
for model in THEORY:
F = res[model]; s = (F[-1] - F[-5]) / (math.log(NS[-1]) - math.log(NS[-5]))
print(f" {model:18s} {s:.4f}")唯一数据源与引用
范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 6 章「奇异学习理论:当费希尔矩阵退化」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}