5D Library › 信息几何、费希尔信息度量与自然梯度优化的统计流形 · English

第六章 奇异学习理论:当费希尔矩阵退化

不可辨识的模型

前五章都默认费希尔矩阵正定:不同的参数给出不同的分布,度量在每一点都是真正的黎曼度量。许多常用的模型不满足这一点。最简单的例子是 y=abx+εy = abx + \varepsilon:只有乘积 abab 影响分布,(a,b)(a, b) 与 (2a,b/2)(2a, b/2) 给出同一个模型。若真实的系数为零,所有满足 a=0a = 0 或 b=0b = 0 的参数都给出真实分布,这个「真参数集」是两条相交的直线,交点处费希尔矩阵为零矩阵。混合模型(各成分权重或位置相同时)、隐马尔可夫模型、降秩回归与多层神经网络都有这样的点;在这些点附近,第二章的克拉美—拉奥下界、第三章的杰弗里斯先验(det⁡G=0\det G = 0)、第四章的自然梯度(GG 不可逆)都失去意义。渡边澄夫把研究这类模型的理论称为奇异学习理论 (Watanabe 2009, 2018)。

自由能与实对数典范阈

贝叶斯模型比较的核心量是边际似然 Zn=∫∏ip(xi∣w)φ(w)dwZ_n = \int \prod_i p(x_i\mid w)\,\varphi(w)\,dw,其负对数 Fn=−log⁡ZnF_n = -\log Z_n 称为自由能(或随机复杂度)。在正则模型中,拉普拉斯近似给出

Fn=nLn(ŵ)+d2log⁡n+Op(1),F_n = nL_n(\hat w) + \frac d2\log n + O_p(1),

这就是施瓦茨的 BIC (Schwarz 1978)。拉普拉斯近似需要后验在最大值附近近似为正态分布,也就是需要费希尔矩阵正定。

证据

渡边的结果。 对一般(包括奇异)的模型,在真分布可由模型实现等条件下,渡边证明 (Watanabe 2009)

Fn=nLn(w0)+λlog⁡n−(m−1)log⁡log⁡n+Op(1),F_n = nL_n(w_0) + \lambda\log n – (m-1)\log\log n + O_p(1),

其中 Ln(w0)L_n(w_0) 是真参数处的经验对数损失,λ\lambda 是平均对数损失函数 K(w)K(w) 的零点集(真参数集)的实对数典范阈(RLCT,又称学习系数),mm 是它的重数。λ\lambda 是正有理数,在正则模型中等于 d/2d/2,在奇异模型中可以严格小于 d/2d/2。证明的关键是代数几何中广中平祐的奇点消解定理:在适当的坐标变换后,K(w)K(w) 局部化为单项式,从而可以算出 λ\lambda。同样的 λ\lambda 还决定贝叶斯泛化误差的期望 E[Gn]=λ/n+o(1/n)E[G_n] = \lambda/n + o(1/n)。阿尾与渡边 2005 年算出了降秩回归的 RLCT (Aoyagi and Watanabe 2005)。

作者框架中「BIC 的 d/2d/2 惩罚不对,应为 λ≤d/2\lambda \le d/2」这一点核对无误。下面的程序不用任何渐近公式,直接计算「平均」自由能 F(n)=−log⁡∫e−nK(w)φ(w)dwF(n) = -\log\int e^{-nK(w)}\varphi(w)\,dw(即上式中扣除 nLn(w0)nL_n(w_0) 与随机涨落后的部分),然后拟合 λ\lambda 与 mm。模型取 y=f(x,w)+N(0,1)y = f(x, w) + N(0,1) 噪声,x∼N(0,1)x \sim N(0,1),参数在 [−1,1]d[-1, 1]^d 上均匀,内层积分用闭式(误差函数),外层在对数网格上数值积分,nn 从 10410^4 取到 101210^{12}。

推导

# The learning coefficient, computed rather than assumed. For y = f(x, w) + N(0, 1)
# noise, x ~ N(0, 1), uniform prior on [-1, 1]^d, the "average" free energy is
# F(n) = -log Int exp(-n K(w)) prior(w) dw with K(w) = E (f(x,w) - f(x,w0))^2 / 2.
# Watanabe: F(n) = lambda log n - (m - 1) log log n + O(1). We integrate exactly
# (inner integrals in closed form, outer on a log grid) for n = 10^4 .. 10^12 and fit
# lambda and m by least squares; BIC would use d/2.
import math

def erf_int(s):                          # (1/2) Int_{-1}^{1} exp(-s t^2) dt
    return 1.0 if s < 1e-12 else 0.5 * math.sqrt(math.pi / s) * math.erf(math.sqrt(s))

U = [-45 + 45 * (i + 0.5) / 3000 for i in range(3000)]      # a = e^u on (0, 1]
DU = 45 / 3000

def logZ(model, n):
    if model == "c x":                                 # regular, d = 1
        return math.log(erf_int(n / 2))
    if model == "a x1 + b x2":                         # regular, d = 2
        return 2 * math.log(erf_int(n / 2))
    if model == "a^2 x":                               # d = 1, K = a^4 / 2
        return math.log(sum(math.exp(-n * math.exp(4 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x":                               # d = 2, K = (ab)^2 / 2
        return math.log(sum(erf_int(n * math.exp(2 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x, ab0 = 0.25":                   # true function nonzero
        c0, tot, M = 0.25, 0.0, 4000
        for i in range(M):
            a = -1 + 2 * (i + 0.5) / M; r = math.sqrt(n) * abs(a)
            lo, hi = (-1 - c0 / a) * r, (1 - c0 / a) * r
            if a < 0:
                lo, hi = -hi, -lo
            p = 0.5 * (math.erf(hi / math.sqrt(2)) - math.erf(lo / math.sqrt(2)))
            tot += 0.5 * math.sqrt(2 * math.pi) / r * p * (1 / M)
        return math.log(tot)
    if model == "a b c x":                             # d = 3, K = (abc)^2 / 2
        V = U[::10]; dv = DU * 10
        return math.log(sum(erf_int(n * math.exp(2 * (u + v)) / 2) * math.exp(u + v)
                            for u in V for v in V) * dv * dv)

NS = [10 ** (k / 2) for k in range(8, 25)]
print("model               d   BIC d/2   fitted lambda   fitted m   theory (lambda, m)")
THEORY = {"c x": (1, "1/2", 1), "a x1 + b x2": (2, "1", 1), "a^2 x": (1, "1/4", 1),
          "a b x": (2, "1/2", 2), "a b x, ab0 = 0.25": (2, "1/2", 1),
          "a b c x": (3, "1/2", 3)}
res = {}
for model, (d, lam, m) in THEORY.items():
    F = [-logZ(model, n) for n in NS]
    X = [(math.log(n), -math.log(math.log(n)), 1.0) for n in NS]   # F = l*x1 + k*x2 + c
    A = [[sum(x[i] * x[j] for x in X) for j in range(3)] for i in range(3)]
    b = [sum(x[i] * f for x, f in zip(X, F)) for i in range(3)]
    for c in range(3):                                   # Gaussian elimination
        for r in range(c + 1, 3):
            t = A[r][c] / A[c][c]; A[r] = [u - t * v for u, v in zip(A[r], A[c])]
            b[r] -= t * b[c]
    sol = [0.0] * 3
    for r in range(2, -1, -1):
        sol[r] = (b[r] - sum(A[r][k] * sol[k] for k in range(r + 1, 3))) / A[r][r]
    res[model] = F
    print(f"{model:18s} {d:2d}   {d / 2:5.1f}      {sol[0]:8.4f}      {sol[1] + 1:6.2f}"
          f"      ({lam}, {m})")
print("naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):")
for model in THEORY:
    F = res[model]; s = (F[-1] - F[-5]) / (math.log(NS[-1]) - math.log(NS[-5]))
    print(f"   {model:18s} {s:.4f}")
# model               d   BIC d/2   fitted lambda   fitted m   theory (lambda, m)
# c x                 1     0.5        0.5000        1.00      (1/2, 1)
# a x1 + b x2         2     1.0        1.0000        1.00      (1, 1)
# a^2 x               1     0.5        0.2500        1.00      (1/4, 1)
# a b x               2     1.0        0.4959        1.86      (1/2, 2)
# a b x, ab0 = 0.25   2     1.0        0.4999        1.00      (1/2, 1)
# a b c x             3     1.5        0.4899        2.68      (1/2, 3)
# naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):
#    c x                0.5000
#    a x1 + b x2        1.0000
#    a^2 x              0.2500
#    a b x              0.4623
#    a b x, ab0 = 0.25  0.5000
#    a b c x            0.4244

两个正则模型给出 λ=0.5000\lambda = 0.5000(y=cxy = cx,d=1d = 1)与 1.00001.0000(y=ax1+bx2y = ax_1 + bx_2,d=2d = 2),与 BIC 一致。y=abxy = abx 在真值为零时,BIC 用 d/2=1d/2 = 1,拟合得到 λ=0.4959\lambda = 0.4959、m=1.86m = 1.86,理论值为 (1/2,2)(1/2, 2);三个因子的 y=abcxy = abcx 拟合为 0.48990.4899 与 2.682.68,理论值 (1/2,3)(1/2, 3),BIC 则用 1.5。真值 ab=0.25ab = 0.25 不为零时,真参数集是一条光滑曲线,λ=1/2\lambda = 1/2、m=1m = 1(拟合 0.4999、1.00):即使没有奇点,不可辨识也使 λ\lambda 小于 d/2d/2。最后,y=a2xy = a^2x 只有一个参数,看起来是「正则」的,但 K=a4/2K = a^4/2 在零点是四次的,λ=1/4\lambda = 1/4(拟合 0.2500),只有 BIC 所用值的一半。

程序最后一部分提醒一个实际困难。log⁡log⁡n\log\log n 项变化极慢,在一张 FF 对 log⁡n\log n 的图上,即使 nn 已到 101210^{12},y=abxy = abx 的局部斜率也只有 0.4623,y=abcxy = abcx 为 0.4244,都低于真正的 λ=0.5\lambda = 0.5;重数 mm 的拟合值也偏离理论值。从有限的 nn 中估计 λ\lambda 与 mm 需要知道展开的形式,否则会把重数项误读为更小的 λ\lambda。

左:五个模型的平均自由能 F(n)(只含先验部分)随 log₁₀ n 的变化。右:局部斜率 dF/d log n;正则模型的斜率是常数 d/2(1 与 0.5),y = a²x 为 0.25,奇异模型 y = abx 与 y = abcx 的斜率从下方缓慢趋近 0.5(点线为 0.25、0.5 与 1)。

WAIC 与 WBIC

在实际问题中 λ\lambda 未知(它依赖未知的真分布),所以渡边提出了两个无须知道 λ\lambda 的量。WAIC(广泛适用信息准则)由后验预测的对数损失加上逐样本对数似然的后验方差构成,在奇异模型中与贝叶斯留一交叉验证渐近等价,并以 1/n1/n 的精度估计贝叶斯泛化损失 (Watanabe 2010)。WBIC(广泛适用贝叶斯信息准则)是在逆温度 β=1/log⁡n\beta = 1/\log n 下的后验中对 nLn(w)nL_n(w) 取平均,它与自由能有相同的主项 nLn(w0)+λlog⁡nnL_n(w_0) + \lambda\log n,因而可以用一次后验抽样估计自由能 (Watanabe 2013)。德尔顿与普卢默 2017 年提出了另一种「奇异 BIC」,在候选模型的 λ\lambda 已知时进行模型选择 (Drton and Plummer 2017)。

推导

# Free energy, BIC and WBIC on data from a singular model. Model y = a b x + N(0, 1),
# a, b uniform on [-1, 1]; data from a b = 0 (the singular point). The likelihood depends
# on w only through t = ab, whose prior density is -log|t| / 2, so the Bayes free energy
# F = -log Int exp(-n L_n) prior is an exact one-dimensional integral. WBIC (Watanabe
# 2013) = posterior mean of n L_n at inverse temperature 1/log n. 200 data sets per n;
# all quantities reported relative to n L_n(w0). Theory: lambda = 1/2, m = 2.
import math, random, statistics

T = sorted([s * math.exp(-30 + 30 * (i + 0.5) / 2500) for i in range(2500)
            for s in (-1, 1)])
W = [abs(t) * 30 / 2500 * (-math.log(abs(t)) / 2) for t in T]   # prior mass per node

def quantities(sxx, sxy, n):
    nl = [(t * t * sxx - 2 * t * sxy) / 2 for t in T]           # n L_n(t) - n L_n(0)
    out = {}
    for beta in (1.0, 1 / math.log(n)):
        e = [-beta * v for v in nl]; mx = max(e)
        z = [w * math.exp(x - mx) for w, x in zip(W, e)]; Z = sum(z)
        out[beta] = (mx + math.log(Z), sum(v * q for v, q in zip(nl, z)) / Z)
    F = -out[1.0][0]
    wbic = out[1 / math.log(n)][1]
    that = max(-1.0, min(1.0, sxy / sxx))                       # MLE of t on [-1, 1]
    nlhat = (that * that * sxx - 2 * that * sxy) / 2
    return F, wbic, nlhat + math.log(n), nlhat + 0.5 * math.log(n) - math.log(math.log(n))

rng = random.Random(2013)
print("     n  theory   mean F  mean WBIC  mean BIC   mean |error| vs F:"
      "  WBIC   BIC  BIC_lam")
for n in (100, 1000, 10_000):
    rows = []
    for _ in range(200):
        sxx = sxy = 0.0
        for _ in range(n):
            x = rng.gauss(0, 1); y = rng.gauss(0, 1); sxx += x * x; sxy += x * y
        rows.append(quantities(sxx, sxy, n))
    th = 0.5 * math.log(n) - math.log(math.log(n))
    F, Wb, B, Bs = (statistics.fmean(r[i] for r in rows) for i in range(4))
    ew = statistics.fmean(abs(r[1] - r[0]) for r in rows)
    eb = statistics.fmean(abs(r[2] - r[0]) for r in rows)
    es = statistics.fmean(abs(r[3] - r[0]) for r in rows)
    print(f"  {n:6,d}  {th:6.3f}  {F:6.3f}   {Wb:6.3f}    {B:6.3f}"
          f"                     {ew:5.3f} {eb:5.3f}  {es:5.3f}")
print("theory = lambda log n - (m - 1) log log n with lambda = 1/2, m = 2, up to O(1);")
print("BIC_lam = n L_n(w_hat) + lambda log n - (m - 1) log log n (needs lambda and m)")
print("BIC(d=2) adds (d/2) log n = log n, so it overstates F by roughly")
print("  (1/2) log n + log log n; WBIC needs no lambda and matches the leading term")
#      n  theory   mean F  mean WBIC  mean BIC   mean |error| vs F:  WBIC   BIC  BIC_lam
#      100   0.775   0.652    0.918     4.131                     0.265 3.479  0.351
#    1,000   1.521   1.376    1.937     6.366                     0.561 4.990  0.397
#   10,000   2.385   2.276    3.073     8.692                     0.797 6.416  0.410
# theory = lambda log n - (m - 1) log log n with lambda = 1/2, m = 2, up to O(1);
# BIC_lam = n L_n(w_hat) + lambda log n - (m - 1) log log n (needs lambda and m)
# BIC(d=2) adds (d/2) log n = log n, so it overstates F by roughly
#   (1/2) log n + log log n; WBIC needs no lambda and matches the leading term

在 y=abxy = abx 模型中,似然只通过 t=abt = ab 依赖参数,而两个均匀变量乘积的密度是 −log⁡|t|/2-\log|t|/2,所以自由能可以精确地化为一维积分,WBIC 也一样。每个 nn 模拟 200 个数据集,全部量都相对于 nLn(w0)nL_n(w_0) 报告。精确的自由能均值为 0.652、1.376 与 2.276,理论主项 12log⁡n−log⁡log⁡n\tfrac12\log n – \log\log n 为 0.775、1.521 与 2.385,差一个大致不变的常数。BIC(d=2d = 2)的均值为 4.131、6.366 与 8.692,与自由能的平均绝对误差为 3.479、4.990 与 6.416,并随 nn 增大,因为它多算了约 12log⁡n+log⁡log⁡n\tfrac12\log n + \log\log n。WBIC 的平均绝对误差为 0.265、0.561 与 0.797,远小于 BIC,但也随 nn 缓慢增长:渡边的定理保证的是主项一致,不保证低阶项一致。若已知 λ\lambda 与 mm,修正后的 BIC 误差约为 0.35–0.41。

奇异性能说明深度学习什么

神经网络当然是奇异模型:隐藏单元可以交换,输出权重为零的单元的输入权重不影响输出,激活函数的对称性造成更多的不可辨识。由此产生一个流行的说法:「奇异学习理论已经解释了深度学习为什么在参数远多于样本时仍能泛化」——因为有效复杂度是 λ\lambda 而不是 d/2d/2。

这个说法需要严格限定。第一,渡边的结果描述的是贝叶斯后验的自由能与贝叶斯泛化误差;实际训练用随机梯度下降得到一个点估计,它的泛化性质是另一个问题。第二,结果是 n→∞n \to \infty、模型固定时的渐近展开,而深度学习中参数个数常常与样本量同阶甚至更大,渐近区间是否适用并不清楚。第三,λ\lambda 依赖真分布与模型,对一个实际网络无法精确计算。张驰原等人 2017 年的实验显示,同样的网络既能拟合真实标签,也能拟合完全随机的标签,说明仅凭模型类的性质(包括它的奇异性)不足以解释在真实数据上的泛化 (Zhang et al. 2017)。魏(Wei)等人 2023 年系统论证了深度网络的奇异性及其对贝叶斯泛化的意义 (Wei et al. 2023);劳(Lau)等人 2023 年提出「局部学习系数」,试图用随机梯度朗之万动力学在训练得到的点附近估计一个局部的 λ\lambda (Lau et al. 2025)。这些是活跃的研究方向,而不是已经完成的解释。本书的表述是:奇异学习理论正确地指出 BIC 与基于费希尔矩阵正定的近似在神经网络中不适用,并给出了贝叶斯意义下正确的复杂度度量;它对随机梯度训练的泛化能说明多少,尚未确立。

假说 · 尚未验证 · 局部学习系数与泛化差距

作者的假说(尚未确立): 在同一架构、同一数据集上用不同随机种子与学习率训练的一组小型网络中,事先登记的局部学习系数估计值与测试损失减训练损失的差距正相关。什么样的结果能证明它是错的: 在事先登记的至少 50 次训练运行上,二者的斯皮尔曼相关系数的 95% 置信区间包含零或全部为负。本书不做这一检验;即使相关存在,也不说明因果方向。

本章小结

当参数不可辨识、费希尔矩阵退化时,BIC 的 d2log⁡n\tfrac d2\log n 不对:贝叶斯自由能的主项是 λlog⁡n−(m−1)log⁡log⁡n\lambda\log n – (m-1)\log\log n,λ≤d/2\lambda \le d/2 是实对数典范阈。程序直接积分得到 y=abxy = abx 的 λ≈1/2\lambda \approx 1/2(BIC 用 1),y=a2xy = a^2x 的 λ=1/4\lambda = 1/4;log⁡log⁡n\log\log n 项使有限 nn 下的斜率偏低。WBIC 无须知道 λ\lambda,它与精确自由能的误差(0.27–0.80)远小于 BIC(3.5–6.4)。奇异性解释深度学习泛化的说法尚未确立:理论描述的是贝叶斯后验的渐近行为,与随机梯度训练的联系仍在研究中。

参考文献

Aoyagi, Miki, and Sumio Watanabe. 2005. “Stochastic Complexities of Reduced Rank Regression in Bayesian Estimation.” Neural Networks 18 (7): 924–33.
Drton, Mathias, and Martyn Plummer. 2017. “A Bayesian Information Criterion for Singular Models.” Journal of the Royal Statistical Society. Series B 79 (2): 323–80.
Lau, Edmund, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. 2025. “The Local Learning Coefficient: A Singularity-Aware Complexity Measure.” Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS 2025), Proceedings of machine learning research, vol. 258: 244–52. https://arxiv.org/abs/2308.12108.
Schwarz, Gideon. 1978. “Estimating the Dimension of a Model.” Annals of Statistics 6 (2): 461–64.
Watanabe, Sumio. 2009. Algebraic Geometry and Statistical Learning Theory. Vol. 25. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press.
Watanabe, Sumio. 2010. “Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory.” Journal of Machine Learning Research 11: 3571–94.
Watanabe, Sumio. 2013. “A Widely Applicable Bayesian Information Criterion.” Journal of Machine Learning Research 14: 867–97.
Watanabe, Sumio. 2018. Mathematical Theory of Bayesian Statistics. CRC Press.
Wei, Susan, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. 2023. “Deep Learning Is Singular, and That’s Good.” IEEE Transactions on Neural Networks and Learning Systems 34 (12): 10473–86. https://doi.org/10.1109/TNNLS.2022.3167409.
Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. “Understanding Deep Learning Requires Rethinking Generalization.” 5th International Conference on Learning Representations (ICLR).

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).