What does Information Geometry and the Natural Gradient conclude, in numbers, about “Singular Learning Theory: When the Fisher Matrix Degenerates”?
Answer: When parameters are not identifiable and the Fisher matrix degenerates, BIC's \(\tfrac d2\log n\) is wrong: the leading terms of the Bayes free energy are \(\lambda\log n - (m-1)\log\log n\), with \(\lambda \le d/2\) the real log canonical threshold. Direct integration in the program gives \(\lambda \approx 1/2\) for \(y = abx\) (BIC uses 1) and \(\lambda = 1/4\) for \(y = a^2x\); the \(\log\log n\) term depresses the slope at finite \(n\). WBIC does not need \(\lambda\), and its error against the exact free energy (0.27–0.80) is far smaller than BIC's (3.5–6.4). The claim that singularity explains generalisation in deep learning is not established: the theory describes the asymptotics of the Bayesian posterior, and its connection to stochastic-gradient training is still being studied.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
| model d BIC d/2 fitted lambda fitted m theory (lambda, m) | |
| c x | 1 0.5 0.5000 1.00 (1/2, 1) |
| a x1 + b x2 | 2 1.0 1.0000 1.00 (1, 1) |
| a^2 x | 1 0.5 0.2500 1.00 (1/4, 1) |
| a b x | 2 1.0 0.4959 1.86 (1/2, 2) |
| a b x, ab0 | 0.25 2 1.0 0.4999 1.00 (1/2, 1) |
| a b c x | 3 1.5 0.4899 2.68 (1/2, 3) |
| naive slope dF/dlog n between n | 10^10 and 10^12 (what a plot would suggest): |
| c x | 0.5000 |
| a x1 + b x2 | 1.0000 |
| a^2 x | 0.2500 |
| a b x | 0.4623 |
| a b x, ab0 | 0.25 0.5000 |
| a b c x | 0.4244 |
Key formula
Show the full program
# The learning coefficient, computed rather than assumed. For y = f(x, w) + N(0, 1)
# noise, x ~ N(0, 1), uniform prior on [-1, 1]^d, the "average" free energy is
# F(n) = -log Int exp(-n K(w)) prior(w) dw with K(w) = E (f(x,w) - f(x,w0))^2 / 2.
# Watanabe: F(n) = lambda log n - (m - 1) log log n + O(1). We integrate exactly
# (inner integrals in closed form, outer on a log grid) for n = 10^4 .. 10^12 and fit
# lambda and m by least squares; BIC would use d/2.
import math
def erf_int(s): # (1/2) Int_{-1}^{1} exp(-s t^2) dt
return 1.0 if s < 1e-12 else 0.5 * math.sqrt(math.pi / s) * math.erf(math.sqrt(s))
U = [-45 + 45 * (i + 0.5) / 3000 for i in range(3000)] # a = e^u on (0, 1]
DU = 45 / 3000
def logZ(model, n):
if model == "c x": # regular, d = 1
return math.log(erf_int(n / 2))
if model == "a x1 + b x2": # regular, d = 2
return 2 * math.log(erf_int(n / 2))
if model == "a^2 x": # d = 1, K = a^4 / 2
return math.log(sum(math.exp(-n * math.exp(4 * u) / 2) * math.exp(u)
for u in U) * DU)
if model == "a b x": # d = 2, K = (ab)^2 / 2
return math.log(sum(erf_int(n * math.exp(2 * u) / 2) * math.exp(u)
for u in U) * DU)
if model == "a b x, ab0 = 0.25": # true function nonzero
c0, tot, M = 0.25, 0.0, 4000
for i in range(M):
a = -1 + 2 * (i + 0.5) / M; r = math.sqrt(n) * abs(a)
lo, hi = (-1 - c0 / a) * r, (1 - c0 / a) * r
if a < 0:
lo, hi = -hi, -lo
p = 0.5 * (math.erf(hi / math.sqrt(2)) - math.erf(lo / math.sqrt(2)))
tot += 0.5 * math.sqrt(2 * math.pi) / r * p * (1 / M)
return math.log(tot)
if model == "a b c x": # d = 3, K = (abc)^2 / 2
V = U[::10]; dv = DU * 10
return math.log(sum(erf_int(n * math.exp(2 * (u + v)) / 2) * math.exp(u + v)
for u in V for v in V) * dv * dv)
NS = [10 ** (k / 2) for k in range(8, 25)]
print("model d BIC d/2 fitted lambda fitted m theory (lambda, m)")
THEORY = {"c x": (1, "1/2", 1), "a x1 + b x2": (2, "1", 1), "a^2 x": (1, "1/4", 1),
"a b x": (2, "1/2", 2), "a b x, ab0 = 0.25": (2, "1/2", 1),
"a b c x": (3, "1/2", 3)}
res = {}
for model, (d, lam, m) in THEORY.items():
F = [-logZ(model, n) for n in NS]
X = [(math.log(n), -math.log(math.log(n)), 1.0) for n in NS] # F = l*x1 + k*x2 + c
A = [[sum(x[i] * x[j] for x in X) for j in range(3)] for i in range(3)]
b = [sum(x[i] * f for x, f in zip(X, F)) for i in range(3)]
for c in range(3): # Gaussian elimination
for r in range(c + 1, 3):
t = A[r][c] / A[c][c]; A[r] = [u - t * v for u, v in zip(A[r], A[c])]
b[r] -= t * b[c]
sol = [0.0] * 3
for r in range(2, -1, -1):
sol[r] = (b[r] - sum(A[r][k] * sol[k] for k in range(r + 1, 3))) / A[r][r]
res[model] = F
print(f"{model:18s} {d:2d} {d / 2:5.1f} {sol[0]:8.4f} {sol[1] + 1:6.2f}"
f" ({lam}, {m})")
print("naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):")
for model in THEORY:
F = res[model]; s = (F[-1] - F[-5]) / (math.log(NS[-1]) - math.log(NS[-5]))
print(f" {model:18s} {s:.4f}")Single source and citation
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 6: Singular Learning Theory: When the Fisher Matrix Degenerates. 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}Other chapter answers in this book
- Chapter 1 · Fisher Information: a Metric on Statistical Manifolds
- Chapter 2 · The Cramér–Rao Bound: Attained, Not Attained and "Beaten"
- Chapter 3 · Jeffreys's Prior: Invariance and Its Price
- Chapter 4 · The Natural Gradient: Invariance, Speed and Approximation
- Chapter 5 · Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem
- Chapter 8 · Practice and a Pre-Registered Test