5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文
Chapter 6 Singular Learning Theory: When the Fisher Matrix Degenerates
Models that are not identifiable
The first five chapters assumed that the Fisher matrix is positive definite: different parameters give different distributions, and the metric is a genuine Riemannian metric at every point. Many common models do not satisfy this. The simplest example is : only the product affects the distribution, and and give the same model. If the true coefficient is zero, every parameter with or gives the true distribution; this “set of true parameters” is two intersecting lines, and at their crossing the Fisher matrix is the zero matrix. Mixture models (when component weights or locations coincide), hidden Markov models, reduced-rank regression and multilayer neural networks all have such points; near them, the Cramér–Rao bound of Chapter 2, Jeffreys’s prior of Chapter 3 () and the natural gradient of Chapter 4 ( not invertible) all lose their meaning. Sumio Watanabe calls the theory of such models singular learning theory (Watanabe 2009, 2018).
The free energy and the real log canonical threshold
The central quantity of Bayesian model comparison is the marginal likelihood , whose negative logarithm is called the free energy (or stochastic complexity). In regular models the Laplace approximation gives
which is Schwarz’s BIC (Schwarz 1978). The Laplace approximation requires the posterior to be approximately normal around its maximum, i.e. the Fisher matrix to be positive definite.
Evidence
Watanabe’s result. For general (including singular) models, under conditions such as the true distribution being realisable by the model, Watanabe proved (Watanabe 2009)
where is the empirical log loss at the true parameter, is the real log canonical threshold (RLCT, also called the learning coefficient) of the zero set (the set of true parameters) of the average log-loss function , and is its multiplicity. is a positive rational number, equal to in regular models and possibly strictly smaller than in singular ones. The key to the proof is Hironaka’s resolution of singularities from algebraic geometry: after a suitable change of coordinates, becomes locally a monomial, from which can be computed. The same also determines the expected Bayes generalisation error, . Aoyagi and Watanabe computed the RLCT of reduced-rank regression in 2005 (Aoyagi and Watanabe 2005).
The framework’s point that “BIC’s penalty is wrong and should be ” checks out. The program below uses no asymptotic formula; it directly computes the “average” free energy (the part of the expansion left after removing and random fluctuation) and then fits and . The model is noise with and parameters uniform on ; the inner integrals are done in closed form (error functions) and the outer ones numerically on a logarithmic grid, for from to .
Calculation
# The learning coefficient, computed rather than assumed. For y = f(x, w) + N(0, 1)
# noise, x ~ N(0, 1), uniform prior on [-1, 1]^d, the "average" free energy is
# F(n) = -log Int exp(-n K(w)) prior(w) dw with K(w) = E (f(x,w) - f(x,w0))^2 / 2.
# Watanabe: F(n) = lambda log n - (m - 1) log log n + O(1). We integrate exactly
# (inner integrals in closed form, outer on a log grid) for n = 10^4 .. 10^12 and fit
# lambda and m by least squares; BIC would use d/2.
import math
def erf_int(s): # (1/2) Int_{-1}^{1} exp(-s t^2) dt
return 1.0 if s < 1e-12 else 0.5 * math.sqrt(math.pi / s) * math.erf(math.sqrt(s))
U = [-45 + 45 * (i + 0.5) / 3000 for i in range(3000)] # a = e^u on (0, 1]
DU = 45 / 3000
def logZ(model, n):
if model == "c x": # regular, d = 1
return math.log(erf_int(n / 2))
if model == "a x1 + b x2": # regular, d = 2
return 2 * math.log(erf_int(n / 2))
if model == "a^2 x": # d = 1, K = a^4 / 2
return math.log(sum(math.exp(-n * math.exp(4 * u) / 2) * math.exp(u)
for u in U) * DU)
if model == "a b x": # d = 2, K = (ab)^2 / 2
return math.log(sum(erf_int(n * math.exp(2 * u) / 2) * math.exp(u)
for u in U) * DU)
if model == "a b x, ab0 = 0.25": # true function nonzero
c0, tot, M = 0.25, 0.0, 4000
for i in range(M):
a = -1 + 2 * (i + 0.5) / M; r = math.sqrt(n) * abs(a)
lo, hi = (-1 - c0 / a) * r, (1 - c0 / a) * r
if a < 0:
lo, hi = -hi, -lo
p = 0.5 * (math.erf(hi / math.sqrt(2)) - math.erf(lo / math.sqrt(2)))
tot += 0.5 * math.sqrt(2 * math.pi) / r * p * (1 / M)
return math.log(tot)
if model == "a b c x": # d = 3, K = (abc)^2 / 2
V = U[::10]; dv = DU * 10
return math.log(sum(erf_int(n * math.exp(2 * (u + v)) / 2) * math.exp(u + v)
for u in V for v in V) * dv * dv)
NS = [10 ** (k / 2) for k in range(8, 25)]
print("model d BIC d/2 fitted lambda fitted m theory (lambda, m)")
THEORY = {"c x": (1, "1/2", 1), "a x1 + b x2": (2, "1", 1), "a^2 x": (1, "1/4", 1),
"a b x": (2, "1/2", 2), "a b x, ab0 = 0.25": (2, "1/2", 1),
"a b c x": (3, "1/2", 3)}
res = {}
for model, (d, lam, m) in THEORY.items():
F = [-logZ(model, n) for n in NS]
X = [(math.log(n), -math.log(math.log(n)), 1.0) for n in NS] # F = l*x1 + k*x2 + c
A = [[sum(x[i] * x[j] for x in X) for j in range(3)] for i in range(3)]
b = [sum(x[i] * f for x, f in zip(X, F)) for i in range(3)]
for c in range(3): # Gaussian elimination
for r in range(c + 1, 3):
t = A[r][c] / A[c][c]; A[r] = [u - t * v for u, v in zip(A[r], A[c])]
b[r] -= t * b[c]
sol = [0.0] * 3
for r in range(2, -1, -1):
sol[r] = (b[r] - sum(A[r][k] * sol[k] for k in range(r + 1, 3))) / A[r][r]
res[model] = F
print(f"{model:18s} {d:2d} {d / 2:5.1f} {sol[0]:8.4f} {sol[1] + 1:6.2f}"
f" ({lam}, {m})")
print("naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):")
for model in THEORY:
F = res[model]; s = (F[-1] - F[-5]) / (math.log(NS[-1]) - math.log(NS[-5]))
print(f" {model:18s} {s:.4f}")
# model d BIC d/2 fitted lambda fitted m theory (lambda, m)
# c x 1 0.5 0.5000 1.00 (1/2, 1)
# a x1 + b x2 2 1.0 1.0000 1.00 (1, 1)
# a^2 x 1 0.5 0.2500 1.00 (1/4, 1)
# a b x 2 1.0 0.4959 1.86 (1/2, 2)
# a b x, ab0 = 0.25 2 1.0 0.4999 1.00 (1/2, 1)
# a b c x 3 1.5 0.4899 2.68 (1/2, 3)
# naive slope dF/dlog n between n = 10^10 and 10^12 (what a plot would suggest):
# c x 0.5000
# a x1 + b x2 1.0000
# a^2 x 0.2500
# a b x 0.4623
# a b x, ab0 = 0.25 0.5000
# a b c x 0.4244The two regular models give (, ) and (, ), agreeing with BIC. For with true value zero, BIC uses , while the fit gives and , against the theoretical ; the three-factor fits as and , against , while BIC uses 1.5. When the true value is not zero, the set of true parameters is a smooth curve, with and (fitted 0.4999 and 1.00): even without a singular point, non-identifiability makes smaller than . Finally, has a single parameter and looks “regular”, but vanishes to fourth order at zero, so (fitted 0.2500), half the value BIC uses.
The last part of the program points to a practical difficulty. The term changes extremely slowly, so on a plot of against the local slope for is only 0.4623 even at , and 0.4244 for , both below the true ; the fitted multiplicities also deviate from theory. Estimating and from finite requires knowing the form of the expansion; otherwise the multiplicity term is misread as a smaller .
WAIC and WBIC
In real problems is unknown (it depends on the unknown true distribution), so Watanabe proposed two quantities that do not require it. WAIC (widely applicable information criterion) combines the log loss of the posterior predictive with the posterior variance of the pointwise log-likelihood; in singular models it is asymptotically equivalent to Bayesian leave-one-out cross-validation and estimates the Bayes generalisation loss to order (Watanabe 2010). WBIC (widely applicable Bayesian information criterion) is the average of under the posterior at inverse temperature ; it has the same leading terms as the free energy, , so the free energy can be estimated from a single posterior sample (Watanabe 2013). Drton and Plummer in 2017 proposed a different “singular BIC” for model selection when the of the candidate models are known (Drton and Plummer 2017).
Calculation
# Free energy, BIC and WBIC on data from a singular model. Model y = a b x + N(0, 1),
# a, b uniform on [-1, 1]; data from a b = 0 (the singular point). The likelihood depends
# on w only through t = ab, whose prior density is -log|t| / 2, so the Bayes free energy
# F = -log Int exp(-n L_n) prior is an exact one-dimensional integral. WBIC (Watanabe
# 2013) = posterior mean of n L_n at inverse temperature 1/log n. 200 data sets per n;
# all quantities reported relative to n L_n(w0). Theory: lambda = 1/2, m = 2.
import math, random, statistics
T = sorted([s * math.exp(-30 + 30 * (i + 0.5) / 2500) for i in range(2500)
for s in (-1, 1)])
W = [abs(t) * 30 / 2500 * (-math.log(abs(t)) / 2) for t in T] # prior mass per node
def quantities(sxx, sxy, n):
nl = [(t * t * sxx - 2 * t * sxy) / 2 for t in T] # n L_n(t) - n L_n(0)
out = {}
for beta in (1.0, 1 / math.log(n)):
e = [-beta * v for v in nl]; mx = max(e)
z = [w * math.exp(x - mx) for w, x in zip(W, e)]; Z = sum(z)
out[beta] = (mx + math.log(Z), sum(v * q for v, q in zip(nl, z)) / Z)
F = -out[1.0][0]
wbic = out[1 / math.log(n)][1]
that = max(-1.0, min(1.0, sxy / sxx)) # MLE of t on [-1, 1]
nlhat = (that * that * sxx - 2 * that * sxy) / 2
return F, wbic, nlhat + math.log(n), nlhat + 0.5 * math.log(n) - math.log(math.log(n))
rng = random.Random(2013)
print(" n theory mean F mean WBIC mean BIC mean |error| vs F:"
" WBIC BIC BIC_lam")
for n in (100, 1000, 10_000):
rows = []
for _ in range(200):
sxx = sxy = 0.0
for _ in range(n):
x = rng.gauss(0, 1); y = rng.gauss(0, 1); sxx += x * x; sxy += x * y
rows.append(quantities(sxx, sxy, n))
th = 0.5 * math.log(n) - math.log(math.log(n))
F, Wb, B, Bs = (statistics.fmean(r[i] for r in rows) for i in range(4))
ew = statistics.fmean(abs(r[1] - r[0]) for r in rows)
eb = statistics.fmean(abs(r[2] - r[0]) for r in rows)
es = statistics.fmean(abs(r[3] - r[0]) for r in rows)
print(f" {n:6,d} {th:6.3f} {F:6.3f} {Wb:6.3f} {B:6.3f}"
f" {ew:5.3f} {eb:5.3f} {es:5.3f}")
print("theory = lambda log n - (m - 1) log log n with lambda = 1/2, m = 2, up to O(1);")
print("BIC_lam = n L_n(w_hat) + lambda log n - (m - 1) log log n (needs lambda and m)")
print("BIC(d=2) adds (d/2) log n = log n, so it overstates F by roughly")
print(" (1/2) log n + log log n; WBIC needs no lambda and matches the leading term")
# n theory mean F mean WBIC mean BIC mean |error| vs F: WBIC BIC BIC_lam
# 100 0.775 0.652 0.918 4.131 0.265 3.479 0.351
# 1,000 1.521 1.376 1.937 6.366 0.561 4.990 0.397
# 10,000 2.385 2.276 3.073 8.692 0.797 6.416 0.410
# theory = lambda log n - (m - 1) log log n with lambda = 1/2, m = 2, up to O(1);
# BIC_lam = n L_n(w_hat) + lambda log n - (m - 1) log log n (needs lambda and m)
# BIC(d=2) adds (d/2) log n = log n, so it overstates F by roughly
# (1/2) log n + log log n; WBIC needs no lambda and matches the leading termIn the model the likelihood depends on the parameters only through , and the product of two uniform variables has density , so the free energy reduces exactly to a one-dimensional integral, and so does WBIC. For each , 200 data sets are simulated, and all quantities are reported relative to . The mean exact free energy is 0.652, 1.376 and 2.276, while the theoretical leading part is 0.775, 1.521 and 2.385, differing by a roughly constant amount. BIC () averages 4.131, 6.366 and 8.692, with mean absolute errors against the free energy of 3.479, 4.990 and 6.416, growing with because it over-counts by about . WBIC has mean absolute errors of 0.265, 0.561 and 0.797, far smaller than BIC’s, though also growing slowly with : Watanabe’s theorem guarantees agreement of the leading terms, not of the lower-order ones. With and known, the corrected BIC errs by about 0.35–0.41.
What singularity can tell us about deep learning
Neural networks are certainly singular models: hidden units can be permuted, the input weights of a unit whose output weight is zero do not affect the output, and symmetries of the activation function create more non-identifiability. This gives rise to a popular claim: “singular learning theory has explained why deep learning generalises even with far more parameters than samples” — because the effective complexity is , not .
The claim needs strict qualification. First, Watanabe’s results describe the free energy and Bayes generalisation error of the Bayesian posterior; practical training produces a point estimate by stochastic gradient descent, whose generalisation is a different question. Second, the results are asymptotic expansions as with the model fixed, whereas in deep learning the number of parameters is often comparable to or larger than the sample size, and it is not clear whether the asymptotic regime applies. Third, depends on the true distribution and the model and cannot be computed exactly for a real network. Zhang and co-authors showed in 2017 that the same networks can fit real labels and also completely random labels, indicating that properties of the model class alone (including its singularity) are not enough to explain generalisation on real data (Zhang et al. 2017). Wei and co-authors in 2023 argued systematically for the singularity of deep networks and its significance for Bayesian generalisation (Wei et al. 2023); Lau and co-authors in 2023 proposed the “local learning coefficient”, which tries to estimate a local near a trained point using stochastic-gradient Langevin dynamics (Lau et al. 2025). These are active research directions, not a completed explanation. The book’s position is: singular learning theory correctly shows that BIC and approximations relying on a positive-definite Fisher matrix do not apply to neural networks, and gives the correct Bayesian measure of complexity; how much it says about the generalisation of stochastic-gradient training is not established.
Hypothesis · not established · Local learning coefficient and the generalisation gap
The author’s hypothesis (not established): among a set of small networks of the same architecture trained on the same data set with different random seeds and learning rates, a pre-registered estimate of the local learning coefficient correlates positively with the gap between test loss and training loss. What would show it wrong: over at least 50 pre-registered training runs, the 95% confidence interval of the Spearman correlation between the two includes zero or lies entirely below it. The book does not carry out this test; even if a correlation exists, it would not establish the direction of causation.
Summary
When parameters are not identifiable and the Fisher matrix degenerates, BIC’s is wrong: the leading terms of the Bayes free energy are , with the real log canonical threshold. Direct integration in the program gives for (BIC uses 1) and for ; the term depresses the slope at finite . WBIC does not need , and its error against the exact free energy (0.27–0.80) is far smaller than BIC’s (3.5–6.4). The claim that singularity explains generalisation in deep learning is not established: the theory describes the asymptotics of the Bayesian posterior, and its connection to stochastic-gradient training is still being studied.
References
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).