关于「奇异学习理论:当费希尔矩阵退化」,《信息几何与自然梯度》给出的定量结论是什么?

结论:当参数不可辨识、费希尔矩阵退化时,BIC 的 \(\tfrac d2\log n\) 不对:贝叶斯自由能的主项是 \(\lambda\log n - (m-1)\log\log n\),\(\lambda \le d/2\) 是实对数典范阈。程序直接积分得到 \(y = abx\) 的 \(\lambda \approx 1/2\)(BIC 用 1),\(y = a^2x\) 的 \(\lambda = 1/4\);\(\log\log n\) 项使有限 \(n\) 下的斜率偏低。WBIC 无须知道 \(\lambda\),它与精确自由能的误差(0.27–0.80)远小于 BIC(3.5–6.4)。奇异性解释深度学习泛化的说法尚未确立:理论描述的是贝叶斯后验的渐近行为,与随机梯度训练的联系仍在研究中。

数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库

model d BIC d/2 fitted lambda fitted m theory (lambda, m)
c x1 0.5 0.5000 1.00 (1/2, 1)
a x1 + b x22 1.0 1.0000 1.00 (1, 1)
a^2 x1 0.5 0.2500 1.00 (1/4, 1)
a b x2 1.0 0.4959 1.86 (1/2, 2)
a b x, ab00.25 2 1.0 0.4999 1.00 (1/2, 1)
a b c x3 1.5 0.4899 2.68 (1/2, 3)
naive slope dF/dlog n between n10^10 and 10^12 (what a plot would suggest):
c x0.5000
a x1 + b x21.0000
a^2 x0.2500
a b x0.4623
a b x, ab00.25 0.5000
a b c x0.4244

核心公式

\[F_n = nL_n(\hat w) + \frac d2\log n + O_p(1),\]
查看完整验算程序
# The learning coefficient, computed rather than assumed. For y = f(x, w) + N(0, 1)
# noise, x ~ N(0, 1), uniform prior on [-1, 1]^d, the "average" free energy is
# F(n) = -log Int exp(-n K(w)) prior(w) dw with K(w) = E (f(x,w) - f(x,w0))^2 / 2.
# Watanabe: F(n) = lambda log n - (m - 1) log log n + O(1). We integrate exactly
# (inner integrals in closed form, outer on a log grid) for n = 10^4 .. 10^12 and fit
# lambda and m by least squares; BIC would use d/2.
import math

def erf_int(s):                          # (1/2) Int_{-1}^{1} exp(-s t^2) dt
    return 1.0 if s < 1e-12 else 0.5 * math.sqrt(math.pi / s) * math.erf(math.sqrt(s))

U = [-45 + 45 * (i + 0.5) / 3000 for i in range(3000)]      # a = e^u on (0, 1]
DU = 45 / 3000

def logZ(model, n):
    if model == "c x":                                 # regular, d = 1
        return math.log(erf_int(n / 2))
    if model == "a x1 + b x2":                         # regular, d = 2
        return 2 * math.log(erf_int(n / 2))
    if model == "a^2 x":                               # d = 1, K = a^4 / 2
        return math.log(sum(math.exp(-n * math.exp(4 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x":                               # d = 2, K = (ab)^2 / 2
        return math.log(sum(erf_int(n * math.exp(2 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x, ab0 = 0.25":                   # true function nonzero
        c0, tot, M = 0.25, 0.0, 4000
        for i in range(M):
            a = -1 + 2 * (i + 0.5) / M; r = math.sqrt(n) * abs(a)
            lo, hi = (-1 - c0 / a) * r, (1 - c0 / a) * r
            if a < 0:
                lo, hi = -hi, -lo
            p = 0.5 * (math.erf(hi / math.sqrt(2)) - math.erf(lo / math.sqrt(2)))
            tot += 0.5 * math.sqrt(2 * math.pi) / r * p * (1 / M)
        return math.log(tot)
    if model == "a b c x":                             # d = 3, K = (abc)^2 / 2
        V = U[::10]; dv = DU * 10
        return math.log(sum(erf_int(n * math.exp(2 * (u + v)) / 2) * math.exp(u + v)
                            for u in V for v in V) * dv * dv)

NS = [10 ** (k / 2) for k in range(8, 25)]
print("model               d   BIC d/2   fitted lambda   fitted m   theory (lambda, m)")
THEORY = {"c x": (1, "1/2", 1), "a x1 + b x2": (2, "1", 1), "a^2 x": (1, "1/4", 1),
          "a b x": (2, "1/2", 2), "a b x, ab0 = 0.25": (2, "1/2", 1),
          "a b c x": (3, "1/2", 3)}
res = {}
for model, (d, lam, m) in THEORY.items():
    F = [-logZ(model, n) for n in NS]
    X = [(math.log(n), -math.log(math.log(n)), 1.0) for n in NS]   # F = l*x1 + k*x2 + c
    A = [[sum(x[i] * x[j] for x in X) for j in range(3)] for i in range(3)]
    b = [sum(x[i] * f for x, f in zip(X, F)) for i in range(3)]
    for c in range(3):                                   # Gaussian elimination
        for r in range(c + 1, 3):
            t = A[r][c] / A[c][c]; A[r] = [u - t * v for u, v in zip(A[r], A[c])]
            b[r] -= t * b[c]
    sol = [0.0] * 3
    for r in range(2, -1, -1):
        sol[r] = (b[r] - sum(A[r][k] * sol[k] for k in range(r + 1, 3))) / A[r][r]
    res[model] = F
    print(f"{model:18s} {d:2d}   {d / 2:5.1f}      {sol[0]:8.4f}      {sol[1] + 1:6.2f}"
          f"      ({lam}, {m})")
print("naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):")
for model in THEORY:
    F = res[model]; s = (F[-1] - F[-5]) / (math.log(NS[-1]) - math.log(NS[-5]))
    print(f"   {model:18s} {s:.4f}")

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 6 章「奇异学习理论:当费希尔矩阵退化」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

本书其他章节结论