Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: chapter answers
A family of probability distributions can be treated as a space. How "far apart" are two nearby distributions? Does that distance change if the same family is described with different parameters? In which direction should an optimiser descend so that the choice of coordinates does not matter? This book answers these questions with mathematical statistics, information theory and optimisation theory, and checks, qualifies or corrects the popular claims in the author's framework one by one. It covers the Fisher information as a Riemannian metric and as the second-order term of the KL divergence…
DOI 10.5281/zenodo.23836268 · Read the monograph · 中文版
- Chapter 1 · Fisher Information: a Metric on Statistical Manifolds
The Fisher information $g_{ij} = E[\partial_i\ell\,\partial_j\ell]$ transforms as a tensor; it is a coordinate-free metric on parameter space and the second-order term of the KL divergence (a ratio of 0.9987 at $dp = 0.… - Chapter 2 · The Cramér–Rao Bound: Attained, Not Attained and "Beaten"
The Cramér–Rao bound (Rao 1945, Cramér 1946, found independently by Fréchet and Darmois) constrains the variance of unbiased estimators under regularity conditions. "Always attained" is false: exact attainment occurs on… - Chapter 3 · Jeffreys's Prior: Invariance and Its Price
Jeffreys's prior $\sqrt{\det G}$ is the Fisher–Rao volume element, so applying it in any coordinates gives the same distribution (an interval probability of 0.164177 in both charts in the program). But it is not "non-in… - Chapter 4 · The Natural Gradient: Invariance, Speed and Approximation
The natural gradient $G^{-1}\nabla L$ is the steepest-descent direction under the Fisher metric. Its invariance holds exactly for the continuous-time flow and only to first order for discrete steps: in the program the g… - Chapter 5 · Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem
An exponential family has two coordinate systems, the natural parameters $\theta$ and the expectation parameters $\eta$, linked by the convex function $\psi$ and its Legendre dual $\phi$ (the negative entropy); the Fish… - Chapter 6 · Singular Learning Theory: When the Fisher Matrix Degenerates
When parameters are not identifiable and the Fisher matrix degenerates, BIC's $\tfrac d2\log n$ is wrong: the leading terms of the Bayes free energy are $\lambda\log n - (m-1)\log\log n$, with $\lambda \le d/2$ the real… - Chapter 8 · Practice and a Pre-Registered Test
Chapter 8 gives a four-step checklist and a pre-registered computational benchmark: on a stratified suite of synthetic tasks frozen by an independent third party, natural-gradient optimisation (exact or K-FAC) is compar…
Illustrative model calculation, not a live quote