What does Information Geometry and the Natural Gradient conclude, in numbers, about “Singular Learning Theory: When the Fisher Matrix Degenerates”?

Answer: When parameters are not identifiable and the Fisher matrix degenerates, BIC's \(\tfrac d2\log n\) is wrong: the leading terms of the Bayes free energy are \(\lambda\log n - (m-1)\log\log n\), with \(\lambda \le d/2\) the real log canonical threshold. Direct integration in the program gives \(\lambda \approx 1/2\) for \(y = abx\) (BIC uses 1) and \(\lambda = 1/4\) for \(y = a^2x\); the \(\log\log n\) term depresses the slope at finite \(n\). WBIC does not need \(\lambda\), and its error against the exact free energy (0.27–0.80) is far smaller than BIC's (3.5–6.4). The claim that singularity explains generalisation in deep learning is not established: the theory describes the asymptotics of the Bayesian posterior, and its connection to stochastic-gradient training is still being studied.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

model d BIC d/2 fitted lambda fitted m theory (lambda, m)
c x1 0.5 0.5000 1.00 (1/2, 1)
a x1 + b x22 1.0 1.0000 1.00 (1, 1)
a^2 x1 0.5 0.2500 1.00 (1/4, 1)
a b x2 1.0 0.4959 1.86 (1/2, 2)
a b x, ab00.25 2 1.0 0.4999 1.00 (1/2, 1)
a b c x3 1.5 0.4899 2.68 (1/2, 3)
naive slope dF/dlog n between n10^10 and 10^12 (what a plot would suggest):
c x0.5000
a x1 + b x21.0000
a^2 x0.2500
a b x0.4623
a b x, ab00.25 0.5000
a b c x0.4244

Key formula

\[F_n = nL_n(\hat w) + \frac d2\log n + O_p(1),\]
Show the full program
# The learning coefficient, computed rather than assumed. For y = f(x, w) + N(0, 1)
# noise, x ~ N(0, 1), uniform prior on [-1, 1]^d, the "average" free energy is
# F(n) = -log Int exp(-n K(w)) prior(w) dw with K(w) = E (f(x,w) - f(x,w0))^2 / 2.
# Watanabe: F(n) = lambda log n - (m - 1) log log n + O(1). We integrate exactly
# (inner integrals in closed form, outer on a log grid) for n = 10^4 .. 10^12 and fit
# lambda and m by least squares; BIC would use d/2.
import math

def erf_int(s):                          # (1/2) Int_{-1}^{1} exp(-s t^2) dt
    return 1.0 if s < 1e-12 else 0.5 * math.sqrt(math.pi / s) * math.erf(math.sqrt(s))

U = [-45 + 45 * (i + 0.5) / 3000 for i in range(3000)]      # a = e^u on (0, 1]
DU = 45 / 3000

def logZ(model, n):
    if model == "c x":                                 # regular, d = 1
        return math.log(erf_int(n / 2))
    if model == "a x1 + b x2":                         # regular, d = 2
        return 2 * math.log(erf_int(n / 2))
    if model == "a^2 x":                               # d = 1, K = a^4 / 2
        return math.log(sum(math.exp(-n * math.exp(4 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x":                               # d = 2, K = (ab)^2 / 2
        return math.log(sum(erf_int(n * math.exp(2 * u) / 2) * math.exp(u)
                            for u in U) * DU)
    if model == "a b x, ab0 = 0.25":                   # true function nonzero
        c0, tot, M = 0.25, 0.0, 4000
        for i in range(M):
            a = -1 + 2 * (i + 0.5) / M; r = math.sqrt(n) * abs(a)
            lo, hi = (-1 - c0 / a) * r, (1 - c0 / a) * r
            if a < 0:
                lo, hi = -hi, -lo
            p = 0.5 * (math.erf(hi / math.sqrt(2)) - math.erf(lo / math.sqrt(2)))
            tot += 0.5 * math.sqrt(2 * math.pi) / r * p * (1 / M)
        return math.log(tot)
    if model == "a b c x":                             # d = 3, K = (abc)^2 / 2
        V = U[::10]; dv = DU * 10
        return math.log(sum(erf_int(n * math.exp(2 * (u + v)) / 2) * math.exp(u + v)
                            for u in V for v in V) * dv * dv)

NS = [10 ** (k / 2) for k in range(8, 25)]
print("model               d   BIC d/2   fitted lambda   fitted m   theory (lambda, m)")
THEORY = {"c x": (1, "1/2", 1), "a x1 + b x2": (2, "1", 1), "a^2 x": (1, "1/4", 1),
          "a b x": (2, "1/2", 2), "a b x, ab0 = 0.25": (2, "1/2", 1),
          "a b c x": (3, "1/2", 3)}
res = {}
for model, (d, lam, m) in THEORY.items():
    F = [-logZ(model, n) for n in NS]
    X = [(math.log(n), -math.log(math.log(n)), 1.0) for n in NS]   # F = l*x1 + k*x2 + c
    A = [[sum(x[i] * x[j] for x in X) for j in range(3)] for i in range(3)]
    b = [sum(x[i] * f for x, f in zip(X, F)) for i in range(3)]
    for c in range(3):                                   # Gaussian elimination
        for r in range(c + 1, 3):
            t = A[r][c] / A[c][c]; A[r] = [u - t * v for u, v in zip(A[r], A[c])]
            b[r] -= t * b[c]
    sol = [0.0] * 3
    for r in range(2, -1, -1):
        sol[r] = (b[r] - sum(A[r][k] * sol[k] for k in range(r + 1, 3))) / A[r][r]
    res[model] = F
    print(f"{model:18s} {d:2d}   {d / 2:5.1f}      {sol[0]:8.4f}      {sol[1] + 1:6.2f}"
          f"      ({lam}, {m})")
print("naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):")
for model in THEORY:
    F = res[model]; s = (F[-1] - F[-5]) / (math.log(NS[-1]) - math.log(NS[-5]))
    print(f"   {model:18s} {s:.4f}")

Single source and citation

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 6: Singular Learning Theory: When the Fisher Matrix Degenerates. 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

Other chapter answers in this book