5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文

Chapter 6 Singular Learning Theory: When the Fisher Matrix Degenerates

Models that are not identifiable

The first five chapters assumed that the Fisher matrix is positive definite: different parameters give different distributions, and the metric is a genuine Riemannian metric at every point. Many common models do not satisfy this. The simplest example is y=abx+εy = abx + \varepsilon: only the product abab affects the distribution, and (a,b)(a, b) and (2a,b/2)(2a, b/2) give the same model. If the true coefficient is zero, every parameter with a=0a = 0 or b=0b = 0 gives the true distribution; this “set of true parameters” is two intersecting lines, and at their crossing the Fisher matrix is the zero matrix. Mixture models (when component weights or locations coincide), hidden Markov models, reduced-rank regression and multilayer neural networks all have such points; near them, the Cramér–Rao bound of Chapter 2, Jeffreys’s prior of Chapter 3 (det⁡G=0\det G = 0) and the natural gradient of Chapter 4 (GG not invertible) all lose their meaning. Sumio Watanabe calls the theory of such models singular learning theory (Watanabe 2009, 2018).

The free energy and the real log canonical threshold

The central quantity of Bayesian model comparison is the marginal likelihood Zn=∫∏ip(xi∣w)φ(w)dwZ_n = \int \prod_i p(x_i\mid w)\,\varphi(w)\,dw, whose negative logarithm Fn=−log⁡ZnF_n = -\log Z_n is called the free energy (or stochastic complexity). In regular models the Laplace approximation gives

Fn=nLn(ŵ)+d2log⁡n+Op(1),F_n = nL_n(\hat w) + \frac d2\log n + O_p(1),

which is Schwarz’s BIC (Schwarz 1978). The Laplace approximation requires the posterior to be approximately normal around its maximum, i.e. the Fisher matrix to be positive definite.

Evidence

Watanabe’s result. For general (including singular) models, under conditions such as the true distribution being realisable by the model, Watanabe proved (Watanabe 2009)

Fn=nLn(w0)+λlog⁡n−(m−1)log⁡log⁡n+Op(1),F_n = nL_n(w_0) + \lambda\log n – (m-1)\log\log n + O_p(1),

where Ln(w0)L_n(w_0) is the empirical log loss at the true parameter, λ\lambda is the real log canonical threshold (RLCT, also called the learning coefficient) of the zero set (the set of true parameters) of the average log-loss function K(w)K(w), and mm is its multiplicity. λ\lambda is a positive rational number, equal to d/2d/2 in regular models and possibly strictly smaller than d/2d/2 in singular ones. The key to the proof is Hironaka’s resolution of singularities from algebraic geometry: after a suitable change of coordinates, K(w)K(w) becomes locally a monomial, from which λ\lambda can be computed. The same λ\lambda also determines the expected Bayes generalisation error, E[Gn]=λ/n+o(1/n)E[G_n] = \lambda/n + o(1/n). Aoyagi and Watanabe computed the RLCT of reduced-rank regression in 2005 (Aoyagi and Watanabe 2005).

The framework’s point that “BIC’s d/2d/2 penalty is wrong and should be λ≤d/2\lambda \le d/2” checks out. The program below uses no asymptotic formula; it directly computes the “average” free energy F(n)=−log⁡∫e−nK(w)φ(w)dwF(n) = -\log\int e^{-nK(w)}\varphi(w)\,dw (the part of the expansion left after removing nLn(w0)nL_n(w_0) and random fluctuation) and then fits λ\lambda and mm. The model is y=f(x,w)+N(0,1)y = f(x, w) + N(0,1) noise with x∼N(0,1)x \sim N(0,1) and parameters uniform on [−1,1]d[-1, 1]^d; the inner integrals are done in closed form (error functions) and the outer ones numerically on a logarithmic grid, for nn from 10410^4 to 101210^{12}.

Calculation

# The learning coefficient, computed rather than assumed. For y = f(x, w) + N(0, 1)
# noise, x ~ N(0, 1), uniform prior on [-1, 1]^d, the "average" free energy is
# F(n) = -log Int exp(-n K(w)) prior(w) dw with K(w) = E (f(x,w) - f(x,w0))^2 / 2.
# Watanabe: F(n) = lambda log n - (m - 1) log log n + O(1). We integrate exactly
# (inner integrals in closed form, outer on a log grid) for n = 10^4 .. 10^12 and fit
# lambda and m by least squares; BIC would use d/2.
import math

def erf_int(s):                          # (1/2) Int_{-1}^{1} exp(-s t^2) dt
    return 1.0 if s < 1e-12 else 0.5 * math.sqrt(math.pi / s) * math.erf(math.sqrt(s))

U = [-45 + 45 * (i + 0.5) / 3000 for i in range(3000)]      # a = e^u on (0, 1]
DU = 45 / 3000

def logZ(model, n):
    if model == "c x":                                 # regular, d = 1
        return math.log(erf_int(n / 2))
    if model == "a x1 + b x2":                         # regular, d = 2
        return 2 * math.log(erf_int(n / 2))
    if model == "a^2 x":                               # d = 1, K = a^4 / 2
        return math.log(sum(math.exp(-n * math.exp(4 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x":                               # d = 2, K = (ab)^2 / 2
        return math.log(sum(erf_int(n * math.exp(2 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x, ab0 = 0.25":                   # true function nonzero
        c0, tot, M = 0.25, 0.0, 4000
        for i in range(M):
            a = -1 + 2 * (i + 0.5) / M; r = math.sqrt(n) * abs(a)
            lo, hi = (-1 - c0 / a) * r, (1 - c0 / a) * r
            if a < 0:
                lo, hi = -hi, -lo
            p = 0.5 * (math.erf(hi / math.sqrt(2)) - math.erf(lo / math.sqrt(2)))
            tot += 0.5 * math.sqrt(2 * math.pi) / r * p * (1 / M)
        return math.log(tot)
    if model == "a b c x":                             # d = 3, K = (abc)^2 / 2
        V = U[::10]; dv = DU * 10
        return math.log(sum(erf_int(n * math.exp(2 * (u + v)) / 2) * math.exp(u + v)
                            for u in V for v in V) * dv * dv)

NS = [10 ** (k / 2) for k in range(8, 25)]
print("model               d   BIC d/2   fitted lambda   fitted m   theory (lambda, m)")
THEORY = {"c x": (1, "1/2", 1), "a x1 + b x2": (2, "1", 1), "a^2 x": (1, "1/4", 1),
          "a b x": (2, "1/2", 2), "a b x, ab0 = 0.25": (2, "1/2", 1),
          "a b c x": (3, "1/2", 3)}
res = {}
for model, (d, lam, m) in THEORY.items():
    F = [-logZ(model, n) for n in NS]
    X = [(math.log(n), -math.log(math.log(n)), 1.0) for n in NS]   # F = l*x1 + k*x2 + c
    A = [[sum(x[i] * x[j] for x in X) for j in range(3)] for i in range(3)]
    b = [sum(x[i] * f for x, f in zip(X, F)) for i in range(3)]
    for c in range(3):                                   # Gaussian elimination
        for r in range(c + 1, 3):
            t = A[r][c] / A[c][c]; A[r] = [u - t * v for u, v in zip(A[r], A[c])]
            b[r] -= t * b[c]
    sol = [0.0] * 3
    for r in range(2, -1, -1):
        sol[r] = (b[r] - sum(A[r][k] * sol[k] for k in range(r + 1, 3))) / A[r][r]
    res[model] = F
    print(f"{model:18s} {d:2d}   {d / 2:5.1f}      {sol[0]:8.4f}      {sol[1] + 1:6.2f}"
          f"      ({lam}, {m})")
print("naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):")
for model in THEORY:
    F = res[model]; s = (F[-1] - F[-5]) / (math.log(NS[-1]) - math.log(NS[-5]))
    print(f"   {model:18s} {s:.4f}")
# model               d   BIC d/2   fitted lambda   fitted m   theory (lambda, m)
# c x                 1     0.5        0.5000        1.00      (1/2, 1)
# a x1 + b x2         2     1.0        1.0000        1.00      (1, 1)
# a^2 x               1     0.5        0.2500        1.00      (1/4, 1)
# a b x               2     1.0        0.4959        1.86      (1/2, 2)
# a b x, ab0 = 0.25   2     1.0        0.4999        1.00      (1/2, 1)
# a b c x             3     1.5        0.4899        2.68      (1/2, 3)
# naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):
#    c x                0.5000
#    a x1 + b x2        1.0000
#    a^2 x              0.2500
#    a b x              0.4623
#    a b x, ab0 = 0.25  0.5000
#    a b c x            0.4244

The two regular models give λ=0.5000\lambda = 0.5000 (y=cxy = cx, d=1d = 1) and 1.00001.0000 (y=ax1+bx2y = ax_1 + bx_2, d=2d = 2), agreeing with BIC. For y=abxy = abx with true value zero, BIC uses d/2=1d/2 = 1, while the fit gives λ=0.4959\lambda = 0.4959 and m=1.86m = 1.86, against the theoretical (1/2,2)(1/2, 2); the three-factor y=abcxy = abcx fits as 0.48990.4899 and 2.682.68, against (1/2,3)(1/2, 3), while BIC uses 1.5. When the true value ab=0.25ab = 0.25 is not zero, the set of true parameters is a smooth curve, with λ=1/2\lambda = 1/2 and m=1m = 1 (fitted 0.4999 and 1.00): even without a singular point, non-identifiability makes λ\lambda smaller than d/2d/2. Finally, y=a2xy = a^2x has a single parameter and looks “regular”, but K=a4/2K = a^4/2 vanishes to fourth order at zero, so λ=1/4\lambda = 1/4 (fitted 0.2500), half the value BIC uses.

The last part of the program points to a practical difficulty. The log⁡log⁡n\log\log n term changes extremely slowly, so on a plot of FF against log⁡n\log n the local slope for y=abxy = abx is only 0.4623 even at n=1012n = 10^{12}, and 0.4244 for y=abcxy = abcx, both below the true λ=0.5\lambda = 0.5; the fitted multiplicities mm also deviate from theory. Estimating λ\lambda and mm from finite nn requires knowing the form of the expansion; otherwise the multiplicity term is misread as a smaller λ\lambda.

Left: the average free energy F(n) (prior part only) of five models against log₁₀ n. Right: the local slope dF/d log n; for the regular models it is the constant d/2 (1 and 0.5), for y = a²x it is 0.25, and for the singular models y = abx and y = abcx it approaches 0.5 slowly from below (dotted lines at 0.25, 0.5 and 1).

WAIC and WBIC

In real problems λ\lambda is unknown (it depends on the unknown true distribution), so Watanabe proposed two quantities that do not require it. WAIC (widely applicable information criterion) combines the log loss of the posterior predictive with the posterior variance of the pointwise log-likelihood; in singular models it is asymptotically equivalent to Bayesian leave-one-out cross-validation and estimates the Bayes generalisation loss to order 1/n1/n (Watanabe 2010). WBIC (widely applicable Bayesian information criterion) is the average of nLn(w)nL_n(w) under the posterior at inverse temperature β=1/log⁡n\beta = 1/\log n; it has the same leading terms as the free energy, nLn(w0)+λlog⁡nnL_n(w_0) + \lambda\log n, so the free energy can be estimated from a single posterior sample (Watanabe 2013). Drton and Plummer in 2017 proposed a different “singular BIC” for model selection when the λ\lambda of the candidate models are known (Drton and Plummer 2017).

Calculation

# Free energy, BIC and WBIC on data from a singular model. Model y = a b x + N(0, 1),
# a, b uniform on [-1, 1]; data from a b = 0 (the singular point). The likelihood depends
# on w only through t = ab, whose prior density is -log|t| / 2, so the Bayes free energy
# F = -log Int exp(-n L_n) prior is an exact one-dimensional integral. WBIC (Watanabe
# 2013) = posterior mean of n L_n at inverse temperature 1/log n. 200 data sets per n;
# all quantities reported relative to n L_n(w0). Theory: lambda = 1/2, m = 2.
import math, random, statistics

T = sorted([s * math.exp(-30 + 30 * (i + 0.5) / 2500) for i in range(2500)
            for s in (-1, 1)])
W = [abs(t) * 30 / 2500 * (-math.log(abs(t)) / 2) for t in T]   # prior mass per node

def quantities(sxx, sxy, n):
    nl = [(t * t * sxx - 2 * t * sxy) / 2 for t in T]           # n L_n(t) - n L_n(0)
    out = {}
    for beta in (1.0, 1 / math.log(n)):
        e = [-beta * v for v in nl]; mx = max(e)
        z = [w * math.exp(x - mx) for w, x in zip(W, e)]; Z = sum(z)
        out[beta] = (mx + math.log(Z), sum(v * q for v, q in zip(nl, z)) / Z)
    F = -out[1.0][0]
    wbic = out[1 / math.log(n)][1]
    that = max(-1.0, min(1.0, sxy / sxx))                       # MLE of t on [-1, 1]
    nlhat = (that * that * sxx - 2 * that * sxy) / 2
    return F, wbic, nlhat + math.log(n), nlhat + 0.5 * math.log(n) - math.log(math.log(n))

rng = random.Random(2013)
print("     n  theory   mean F  mean WBIC  mean BIC   mean |error| vs F:"
      "  WBIC   BIC  BIC_lam")
for n in (100, 1000, 10_000):
    rows = []
    for _ in range(200):
        sxx = sxy = 0.0
        for _ in range(n):
            x = rng.gauss(0, 1); y = rng.gauss(0, 1); sxx += x * x; sxy += x * y
        rows.append(quantities(sxx, sxy, n))
    th = 0.5 * math.log(n) - math.log(math.log(n))
    F, Wb, B, Bs = (statistics.fmean(r[i] for r in rows) for i in range(4))
    ew = statistics.fmean(abs(r[1] - r[0]) for r in rows)
    eb = statistics.fmean(abs(r[2] - r[0]) for r in rows)
    es = statistics.fmean(abs(r[3] - r[0]) for r in rows)
    print(f"  {n:6,d}  {th:6.3f}  {F:6.3f}   {Wb:6.3f}    {B:6.3f}"
          f"                     {ew:5.3f} {eb:5.3f}  {es:5.3f}")
print("theory = lambda log n - (m - 1) log log n with lambda = 1/2, m = 2, up to O(1);")
print("BIC_lam = n L_n(w_hat) + lambda log n - (m - 1) log log n (needs lambda and m)")
print("BIC(d=2) adds (d/2) log n = log n, so it overstates F by roughly")
print("  (1/2) log n + log log n; WBIC needs no lambda and matches the leading term")
#      n  theory   mean F  mean WBIC  mean BIC   mean |error| vs F:  WBIC   BIC  BIC_lam
#      100   0.775   0.652    0.918     4.131                     0.265 3.479  0.351
#    1,000   1.521   1.376    1.937     6.366                     0.561 4.990  0.397
#   10,000   2.385   2.276    3.073     8.692                     0.797 6.416  0.410
# theory = lambda log n - (m - 1) log log n with lambda = 1/2, m = 2, up to O(1);
# BIC_lam = n L_n(w_hat) + lambda log n - (m - 1) log log n (needs lambda and m)
# BIC(d=2) adds (d/2) log n = log n, so it overstates F by roughly
#   (1/2) log n + log log n; WBIC needs no lambda and matches the leading term

In the y=abxy = abx model the likelihood depends on the parameters only through t=abt = ab, and the product of two uniform variables has density −log⁡|t|/2-\log|t|/2, so the free energy reduces exactly to a one-dimensional integral, and so does WBIC. For each nn, 200 data sets are simulated, and all quantities are reported relative to nLn(w0)nL_n(w_0). The mean exact free energy is 0.652, 1.376 and 2.276, while the theoretical leading part 12log⁡n−log⁡log⁡n\tfrac12\log n – \log\log n is 0.775, 1.521 and 2.385, differing by a roughly constant amount. BIC (d=2d = 2) averages 4.131, 6.366 and 8.692, with mean absolute errors against the free energy of 3.479, 4.990 and 6.416, growing with nn because it over-counts by about 12log⁡n+log⁡log⁡n\tfrac12\log n + \log\log n. WBIC has mean absolute errors of 0.265, 0.561 and 0.797, far smaller than BIC’s, though also growing slowly with nn: Watanabe’s theorem guarantees agreement of the leading terms, not of the lower-order ones. With λ\lambda and mm known, the corrected BIC errs by about 0.35–0.41.

What singularity can tell us about deep learning

Neural networks are certainly singular models: hidden units can be permuted, the input weights of a unit whose output weight is zero do not affect the output, and symmetries of the activation function create more non-identifiability. This gives rise to a popular claim: “singular learning theory has explained why deep learning generalises even with far more parameters than samples” — because the effective complexity is λ\lambda, not d/2d/2.

The claim needs strict qualification. First, Watanabe’s results describe the free energy and Bayes generalisation error of the Bayesian posterior; practical training produces a point estimate by stochastic gradient descent, whose generalisation is a different question. Second, the results are asymptotic expansions as n→∞n \to \infty with the model fixed, whereas in deep learning the number of parameters is often comparable to or larger than the sample size, and it is not clear whether the asymptotic regime applies. Third, λ\lambda depends on the true distribution and the model and cannot be computed exactly for a real network. Zhang and co-authors showed in 2017 that the same networks can fit real labels and also completely random labels, indicating that properties of the model class alone (including its singularity) are not enough to explain generalisation on real data (Zhang et al. 2017). Wei and co-authors in 2023 argued systematically for the singularity of deep networks and its significance for Bayesian generalisation (Wei et al. 2023); Lau and co-authors in 2023 proposed the “local learning coefficient”, which tries to estimate a local λ\lambda near a trained point using stochastic-gradient Langevin dynamics (Lau et al. 2025). These are active research directions, not a completed explanation. The book’s position is: singular learning theory correctly shows that BIC and approximations relying on a positive-definite Fisher matrix do not apply to neural networks, and gives the correct Bayesian measure of complexity; how much it says about the generalisation of stochastic-gradient training is not established.

Hypothesis · not established · Local learning coefficient and the generalisation gap

The author’s hypothesis (not established): among a set of small networks of the same architecture trained on the same data set with different random seeds and learning rates, a pre-registered estimate of the local learning coefficient correlates positively with the gap between test loss and training loss. What would show it wrong: over at least 50 pre-registered training runs, the 95% confidence interval of the Spearman correlation between the two includes zero or lies entirely below it. The book does not carry out this test; even if a correlation exists, it would not establish the direction of causation.

Summary

When parameters are not identifiable and the Fisher matrix degenerates, BIC’s d2log⁡n\tfrac d2\log n is wrong: the leading terms of the Bayes free energy are λlog⁡n−(m−1)log⁡log⁡n\lambda\log n – (m-1)\log\log n, with λ≤d/2\lambda \le d/2 the real log canonical threshold. Direct integration in the program gives λ≈1/2\lambda \approx 1/2 for y=abxy = abx (BIC uses 1) and λ=1/4\lambda = 1/4 for y=a2xy = a^2x; the log⁡log⁡n\log\log n term depresses the slope at finite nn. WBIC does not need λ\lambda, and its error against the exact free energy (0.27–0.80) is far smaller than BIC’s (3.5–6.4). The claim that singularity explains generalisation in deep learning is not established: the theory describes the asymptotics of the Bayesian posterior, and its connection to stochastic-gradient training is still being studied.

References

Aoyagi, Miki, and Sumio Watanabe. 2005. “Stochastic Complexities of Reduced Rank Regression in Bayesian Estimation.” Neural Networks 18 (7): 924–33.
Drton, Mathias, and Martyn Plummer. 2017. “A Bayesian Information Criterion for Singular Models.” Journal of the Royal Statistical Society. Series B 79 (2): 323–80.
Lau, Edmund, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. 2025. “The Local Learning Coefficient: A Singularity-Aware Complexity Measure.” Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS 2025), Proceedings of machine learning research, vol. 258: 244–52. https://arxiv.org/abs/2308.12108.
Schwarz, Gideon. 1978. “Estimating the Dimension of a Model.” Annals of Statistics 6 (2): 461–64.
Watanabe, Sumio. 2009. Algebraic Geometry and Statistical Learning Theory. Vol. 25. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press.
Watanabe, Sumio. 2010. “Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory.” Journal of Machine Learning Research 11: 3571–94.
Watanabe, Sumio. 2013. “A Widely Applicable Bayesian Information Criterion.” Journal of Machine Learning Research 14: 867–97.
Watanabe, Sumio. 2018. Mathematical Theory of Bayesian Statistics. CRC Press.
Wei, Susan, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. 2023. “Deep Learning Is Singular, and That’s Good.” IEEE Transactions on Neural Networks and Learning Systems 34 (12): 10473–86. https://doi.org/10.1109/TNNLS.2022.3167409.
Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. “Understanding Deep Learning Requires Rethinking Generalization.” 5th International Conference on Learning Representations (ICLR).

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).