5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文
Preface: When the Coordinates Change, What Stays the Same?
Note
Scope: please read this first. This book is a work of mathematical statistics and optimisation theory for teaching and research. It is no guarantee of the performance of any particular machine-learning system, model or product, and it is not engineering, investment or career advice. All the data in the book (samples, tasks, loss curves, timings) are synthetic or simulated by the programs and come from no real system; the “cost” in the programs is counted in units defined by a stated rule, not measured time on any hardware. The results on which optimiser is faster hold only for the stated models, dimensions and parameters, and may reverse on another problem. The book offers no method that is “guaranteed to converge faster”, “certain to generalise better” or “proven optimal”.
The same family of probability distributions can be described with different parameters. The probability that a coin lands heads can be written as or as the log-odds ; the spread of a normal distribution can be given by its standard deviation , its variance or . Changing the parameters does not change the distributions. Yet many habitual procedures depend on the choice: the “straight-line distance” in parameter space, a “uniform” prior over the parameters, a step down the gradient of the parameters. Change the coordinates and all of these change. The starting point of information geometry is that, since coordinates are a human choice, we should look for quantities that do not change with them. The Fisher information provides such a metric and turns a family of distributions into a Riemannian manifold; from it one can define an invariant prior (Jeffreys), an invariant direction of descent (the natural gradient), and the limit on the precision of estimation (Cramér–Rao).
The author’s framework strings these results together and, in several places, states them more strongly than the mathematics allows: the bound is “always attained”, Jeffreys’s prior is “non-informative and always proper”, the natural gradient comes with a “proof” that it removes coordinate-dependent oscillation in ill-conditioned valleys and is “always faster”, the empirical Fisher can stand in for the Fisher, and singular learning theory “explains” why deep learning generalises. This book first checks these claims, refuting those that can be refuted and stating conditions for those that need them, then gives the boundaries that do hold mathematically, and checks every number with a program that can be run.
What the book covers
Chapter 1 treats the Fisher information as a metric: the score, the second-order expansion of the KL divergence, the exact conditions of Čencov’s uniqueness theorem, the monotonicity of information under statistics, and why KL is not a distance. Chapter 2 treats the Cramér–Rao bound: when it is attained and when it is not, how biased estimators, James–Stein shrinkage and superefficiency go “below” it, and the cases where its regularity conditions fail. Chapter 3 treats the invariance of Jeffreys’s prior, its improperness, the Neyman–Scott problem, its dependence on the sampling design and reference priors. Chapter 4 treats Amari’s natural gradient: in what sense it is invariant, its relation to Newton and Gauss–Newton methods, and comparisons with gradient descent, Adam, the empirical Fisher and K-FAC on ill-conditioned logistic regression and a small neural network. Chapter 5 treats the dually flat structure of exponential families, Legendre duality, Bregman divergences, α-connections and the generalised Pythagorean theorem, with iterative proportional fitting and the EM algorithm read as projections. Chapter 6 treats Watanabe’s singular learning theory: why BIC’s is wrong when the Fisher matrix degenerates, how the real log canonical threshold is computed, WAIC and WBIC, and what this theory can and cannot say about deep learning. Chapter 7 reads these beside “changing and moving without rest, flowing through the six empty places” of the Great Commentary on the Book of Changes and its commentaries. Chapter 8 gives a four-step checklist and a protocol for a pre-registered computational benchmark.
The book neighbours several others in 5D Library; overlaps are only briefly recalled with a reference, and their programs are not repeated. Book 11, on the geometry of self-attention, requisite variety and cognitive projection, discussed conditional entropy and mutual information (the KL divergence and mutual information of Chapters 1 and 5 add only the geometric view). Book 15, on Markov blankets, variational free energy and active inference, discussed variational free energy and the KL divergence in active inference (the “free energy” of Chapter 6 here is the negative log marginal likelihood of Bayesian statistics, which shares the name but not the meaning). Book 26, on probability judgement, Brier-score calibration and the wisdom of crowds, discussed strictly proper scoring rules (the link between the logarithmic score and KL is mentioned once in Chapter 1). Book 33, on scientific method, falsifiability and the replication crisis, discussed pre-registration, power and decision rules (Chapter 8 follows it). Book 42, on Bayesian search, the secretary problem and Gittins indices, discussed Bayesian updating and the choice of priors (Chapter 3 only recalls it). Book 23 discussed stochastic gradient descent and learning rates, which Chapter 4 does not repeat.
Claims corrected after checking
Several points in the author’s framework needed correction, qualification or supplement after checking. Each chapter states them where they arise, and the accompanying editorial notes collect them.
- The “uniqueness” of the Fisher metric. Čencov’s theorem (Russian original 1972, English translation 1982) says that on a finite sample space the only Riemannian metric invariant under all Markov embeddings (think of them as “refinements” that lose no information) is the Fisher metric, up to a constant multiple (Čencov 1982). Campbell extended this to unnormalised positive measures (Campbell 1986), Ay, Jost, Lê and Schwachhöfer to parametrised measure models on general sample spaces (Ay et al. 2015), and Bauer, Bruveris and Michor gave another version on the space of smooth densities (Bauer et al. 2016). “Unique” is therefore relative to an explicit invariance requirement; it does not mean that a statistical manifold can carry no other metric. A program shows the Euclidean metric changing as soon as a category is split (Chapter 1).
- The second-order expansion of KL, and “distance”. checks out and is verified step by step in Bernoulli and normal models; but the KL divergence is asymmetric and fails the triangle inequality, as does its square root, and should not be called a “distance”. The true distance is the Fisher–Rao geodesic distance obtained by integrating the metric, already proposed in Rao’s 1945 paper (Rao 1945) (Chapter 1).
- The Cramér–Rao bound “is always attained”. The bound was obtained by Rao in 1945 (Rao 1945) and Cramér in 1946 (Cramér 1946), and independently by Fréchet (Fréchet 1943) and Darmois (Darmois 1945); Aitken and Silverstone had a related result in 1942 (Aitken and Silverstone 1942). “Always attained” is false. At finite sample size an unbiased estimator attains the bound exactly if and only if the model is an exponential family and the target is the expectation of its sufficient statistic (the mean parameter) or a linear function of it; the rate of an exponential distribution has no unbiased estimator that attains it; biased estimators can have mean squared error below the unbiased bound (SS/(n+1) for a normal variance, the James–Stein estimator (James and Stein 1961)); models whose support depends on the parameter, such as the endpoint of a uniform distribution, violate the regularity conditions, and the bound does not apply; and Hodges’s superefficient estimator shows that the bound holds only in an asymptotic, “almost everywhere” sense (Le Cam 1953) (Chapter 2).
- Jeffreys’s prior is “non-informative and always proper”. Jeffreys’s 1946 paper appears in Proceedings of the Royal Society A, volume 186, issue 1007, pages 453–461 (Jeffreys 1946); this checks out. But the rule gives improper priors for a normal mean and a Poisson rate; with several parameters Jeffreys himself recommended the “independent” form for location–scale families rather than the given by (Jeffreys 1961); in the Neyman–Scott problem the joint rule makes the posterior concentrate on a wrong value; and it depends on the sampling design (binomial and negative-binomial sampling give different priors), so it violates the likelihood principle (Kass and Wasserman 1996; Berger and Wolpert 1988). “Non-informative” is only a label; Bernardo’s reference priors are a more systematic generalisation (Bernardo 1979; Berger et al. 2009) (Chapter 3).
- The natural gradient’s “proof” and “always faster”. Amari’s 1998 paper appears in Neural Computation, volume 10, issue 2, pages 251–276 (Amari 1998). Its main results are that online natural-gradient learning is Fisher-efficient asymptotically and that the natural gradient may alleviate “plateaus” in multilayer networks. The framework’s “proof that it removes coordinate-dependent oscillation in ill-conditioned valleys” should be restated as a weaker fact that does hold: when the Fisher matrix equals the Hessian of the loss (for instance a quadratic loss, or a generalised linear model with canonical link), a natural-gradient step is a Newton step and so is unaffected by the condition number. Invariance holds exactly only for the continuous-time gradient flow; with discrete steps, results in different coordinates differ by , so invariance is only first-order (Martens 2020). The programs show that the natural gradient usually takes the fewest steps but is not always the cheapest in computation: gradient descent is cheaper when conditioning is good, and Adam is cheaper on a small neural network (Kingma and Ba 2015). The “empirical Fisher” is not the Fisher (Kunstner et al. 2019), and K-FAC is an approximation (Martens and Grosse 2015) (Chapter 4).
- The conditions of the generalised Pythagorean theorem. holds when is the m-projection of onto an e-flat submanifold (for example an exponential family), i.e. the point minimising , and is any point of that submanifold; equivalently, the m-geodesic joining and is orthogonal to the e-geodesic joining and . The order and direction cannot be swapped; a program shows the equality failing when they are (Amari and Nagaoka 2000; Csiszár 1975) (Chapter 5).
- Singular learning theory and deep learning. In models whose parameters are not identifiable and whose Fisher matrix degenerates, BIC’s penalty is wrong: the leading terms of the Bayes free energy are , where is the real log canonical threshold (Watanabe 2009), and WAIC (Watanabe 2010) and WBIC (Watanabe 2013) remain valid. A program computes for the model (BIC uses 1). But “the singularity of neural networks has explained why deep learning generalises” does not hold: the theory describes the asymptotic free energy and generalisation error of the Bayesian posterior, whereas practical training produces a point estimate by stochastic gradient descent; work connecting the two is ongoing (Wei et al. 2023; Lau et al. 2025) (Chapter 6).
- The classical text. “The Yi is a book that must not be kept at a distance; its way often shifts, changing and moving without rest, flowing through the six empty places; above and below without constancy, firm and yielding exchanging; it cannot be made a fixed rule, only following where change goes” (Xici, part II) was checked, with its commentaries, against the online transcription of the Siku Quanshu text (Zhouyi zhengyi (Siku quanshu edition), juan 12, on Wikisource) and agrees; the printed Ruan Yuan edition of the Commentaries and Subcommentaries on the Thirteen Classics (阮元 校刻 1980) remains to be checked by the author (especially 因/困 in Kong’s subcommentary). The framework cites “the commentaries of Wang Bi and Han Kangbo”: in the Correct Meaning of the Zhouyi, the commentary on the two parts of the classic is Wang Bi’s and that on the Xici onwards is Han Kangbo’s, so this passage has only Han’s gloss “the six empty places are the six positions”; Kong Yingda’s subcommentary explains them as the positions of the six lines. The comparison with “representation invariance” is an analogy; the book does not claim that the Book of Changes contains differential geometry and does not support treating hexagrams as statistical models (Chapter 7).
- The outcome of Chapter 8. The framework suggests comparing “the number of steps or the wall-clock time to reach a target loss”. With a single primary outcome, the book chooses the logarithm of the wall-clock time ratio: step counts favour methods whose steps are expensive (Chapter 4 shows the natural gradient taking the fewest steps without always costing the least). Gradient-evaluation counts are a secondary outcome (Chapter 8).
How to read the boxes
As in the other books of 5D Library, different kinds of statement are marked separately.
Evidence
Evidence boxes report published research findings, with their sources.
Calculation
Calculation boxes give short Python programs. Every program runs as printed in a standard Python 3 environment (3.8 or later), uses only the standard library, fixes its random seeds, and is followed by its actual output. The samples, tasks and loss curves in the book are synthetic; parameters are illustrative, not estimates for any real system.
Hypothesis · not established
Hypothesis boxes contain the author’s ideas, marked “not established”, together with the results that would show them wrong. The author’s own models and definitions are marked as such in the text.
Source text
Source boxes quote classical texts with a plain-language explanation.
Note
Note boxes give reminders and explain choices.
About this edition
This is version 1.0.0, the first complete edition: a preface and eight chapters, in Chinese and English. It contains 16 Python programs and 5 vector figures. The text, boxes and code were drafted with an AI writing tool from the author’s title, framework and notes; the author is responsible for reviewing and revising them and for checking sources. AI-drafted text can get bibliographic details wrong; corrections are welcome through the 5D Library website. The “gradient-evaluation unit” cost accounting in the Chapter 4 programs, the positions–lines / coordinates–objects table of Chapter 7 and the pilot environment of Chapter 8 are the author’s models; their mathematical properties are not predictions about any real training system.
Disclosure
The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check. The businesses the author runs use machine-learning and AI tools in content production and in the knowledge base, but there is no direct sales relationship between the methods discussed in this book and their commercial products.
References
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).