5D Library › Books › 5DL-B-049

信息几何、费希尔信息度量与自然梯度优化的统计流形

从克拉美—拉奥下界、杰弗里斯不变先验到奇异学习理论及《周易·系辞》象数与义理变易观考辨

Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates

  • 范雅福 Fan Yafu
  • Version 1.0.0 · 2026-10-31
  • CC-BY-NC-ND-4.0
  • 5DL-B-049
DOI10.5281/zenodo.23836268

摘要

一族概率分布可以看作一个空间:两个相近的分布之间有多「远」?换一套参数来描述同一族分布,这个距离会不会变?优化算法沿着哪个方向下降才不依赖坐标的选择?本书用数理统计、信息论与优化理论回答这些问题,并把作者框架中的流行说法逐条核对、限定或改正。依次讨论:费希尔信息作为黎曼度量与 KL 散度的二阶项(程序逐级验证二阶展开;KL 不对称,也不满足三角不等式,连它的平方根也不满足),以及钦佐夫(Čencov)唯一性定理的准确条件(有限样本空间、在马尔可夫嵌入下不变、至多差一个常数;艾等人 2015 年推广到一般样本空间),统计量只能丢失而不能增加费希尔信息;克拉美—拉奥下界(拉奥 1945、克拉美 1946,弗雷歇与达尔穆瓦也独立得到)及「下界总能达到」之误:有限样本中只有指数族的均值参数可以被无偏估计恰好达到,有偏的 SS/(n+1) 与詹姆斯—斯坦估计量的均方误差低于下界,均匀分布端点的估计误差按 1/n² 下降,霍奇斯超有效估计量在局部付出无界的代价;杰弗里斯先验(1946)的重参数化不变性、它在正态均值与泊松率上不可积、在内曼—斯科特问题中后验集中在真值的一半、二项与负二项抽样下给出不同的后验(违反似然原理),以及它的区间在二项比例上有良好的频率覆盖;甘利俊一(Amari)1998 年的自然梯度:不变性只对连续时间的流严格成立,离散步长下不同坐标的结果相差 O(η),步长为 1 时差别很大;在条件数高的逻辑回归上自然梯度(此时即牛顿法)步数最少,但按计算量在条件良好时不如梯度下降,在一个小型神经网络上按计算量不如 Adam,经验费希尔在噪声估计不对时步长偏差百倍,K-FAC 近似的相对误差可达 0.67;指数族的对偶平坦结构、勒让德对偶与布雷格曼散度、广义勾股定理(方向不能颠倒)与迭代比例拟合的交替投影;渡边澄夫的奇异学习理论:在 y = abx 这样的奇异模型中自由能的系数是实对数典范阈 λ = 1/2 而不是 BIC 的 d/2 = 1,WBIC 无须知道 λ 就能跟上自由能,但奇异性并未「解释」深度学习的泛化。

第七章与《周易·系辞下》「变动不居,周流六虚」及韩康伯注「六虚,六位也」、孔颖达疏,以及《系辞上》「神无方而易无体」并读,明确这只是「位与爻」与「坐标与对象」之间的类比,不主张《周易》含有微分几何,也不把卦象当作统计模型。第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。

关键词:信息几何;费希尔信息;费希尔—拉奥度量;KL 散度;钦佐夫定理;克拉美—拉奥下界;有效估计量;詹姆斯—斯坦估计量;杰弗里斯先验;参考先验;似然原理;自然梯度;经验费希尔;K-FAC;Adam;指数族;对偶平坦流形;布雷格曼散度;广义勾股定理;奇异学习理论;实对数典范阈;WBIC

Abstract

A family of probability distributions can be treated as a space. How "far apart" are two nearby distributions? Does that distance change if the same family is described with different parameters? In which direction should an optimiser descend so that the choice of coordinates does not matter? This book answers these questions with mathematical statistics, information theory and optimisation theory, and checks, qualifies or corrects the popular claims in the author's framework one by one. It covers the Fisher information as a Riemannian metric and as the second-order term of the KL divergence (verified numerically; KL is asymmetric and fails the triangle inequality, as does its square root), and the exact conditions of Čencov's uniqueness theorem (finite sample spaces, invariance under Markov embeddings, uniqueness up to a constant; extended to general sample spaces by Ay and co-authors in 2015), with statistics able only to lose Fisher information; the Cramér–Rao bound (Rao 1945, Cramér 1946, found independently by Fréchet and Darmois) and the mistaken idea that it is always attained: at finite n only the mean parameter of an exponential family is attained by an unbiased estimator, the biased SS/(n+1) and the James–Stein estimator have mean squared error below the bound, the error of the uniform-endpoint estimator falls like 1/n², and Hodges's superefficient estimator pays an unbounded local price; Jeffreys's 1946 prior, its invariance under reparametrisation, its improperness for a normal mean and a Poisson rate, its posterior concentrating at half the true variance in the Neyman–Scott problem, its different posteriors under binomial and negative-binomial sampling (a breach of the likelihood principle), and the good frequentist coverage of its intervals for a binomial proportion; Amari's 1998 natural gradient, whose invariance holds exactly only for the continuous-time flow, discrete steps in different coordinates differing by O(η) and greatly at step size 1; on ill-conditioned logistic regression the natural gradient (here Newton's method) takes the fewest steps but costs more than gradient descent when conditioning is good, costs more than Adam on a small neural network, the empirical Fisher mis-scales steps a hundredfold when the noise level is wrong, and the K-FAC approximation has relative error up to 0.67; the dually flat structure of exponential families, Legendre duality and Bregman divergences, the generalised Pythagorean theorem (whose direction cannot be reversed) and iterative proportional fitting as alternating projections; and Watanabe's singular learning theory: in a singular model such as y = abx the free-energy coefficient is the real log canonical threshold λ = 1/2, not BIC's d/2 = 1, and WBIC tracks the free energy without knowing λ, but singularity does not "explain" generalisation in deep learning.

Chapter 7 reads these beside "changing and moving without rest, flowing through the six empty places" of the Great Commentary (Xici, part II) of the Book of Changes, Han Kangbo's gloss "the six empty places are the six positions" and Kong Yingda's subcommentary, and "the spirit has no fixed place and change has no fixed body" of part I, stating plainly that the parallel between positions and lines and between coordinates and objects is an analogy: the book does not claim that the Book of Changes contains differential geometry, nor treat hexagrams as statistical models. Chapter 8 gives a four-step checklist and a pre-registered computational benchmark: on a stratified suite of synthetic tasks frozen by an independent third party, natural-gradient optimisation (exact or K-FAC) is compared with Adam on wall-clock time to a target loss; the single primary outcome is the mean over tasks of the base-2 logarithm of the time ratio, the smallest effect of interest log₂1.25, 393 paired tasks for 90 % power and 470 planned after allowing for invalid tasks, with the decision rule written as code. Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's models and hypotheses; all data in the book are synthetic or simulated and are no guarantee of the performance of any particular machine-learning system.

Keywords: information geometry; Fisher information; Fisher-Rao metric; Kullback-Leibler divergence; Chentsov theorem; Cramer-Rao bound; efficient estimator; James-Stein estimator; Jeffreys prior; reference prior; likelihood principle; natural gradient; empirical Fisher; K-FAC; Adam; exponential family; dually flat manifold; Bregman divergence; generalised Pythagorean theorem; singular learning theory; real log canonical threshold; WBIC

目录 · Contents

  • 序:换了坐标,什么不变?English 中文Preface: When the Coordinates Change, What Stays the Same?
  • 第一章 费希尔信息:统计流形上的度量English 中文Chapter 1 Fisher Information: a Metric on Statistical Manifolds
  • 第二章 克拉美—拉奥下界:达到、达不到与「低于」English 中文Chapter 2 The Cramér–Rao Bound: Attained, Not Attained and "Beaten"
  • 第三章 杰弗里斯先验:不变性及其代价English 中文Chapter 3 Jeffreys's Prior: Invariance and Its Price
  • 第四章 自然梯度:不变性、速度与近似English 中文Chapter 4 The Natural Gradient: Invariance, Speed and Approximation
  • 第五章 对偶平坦:指数族、勒让德对偶与广义勾股定理English 中文Chapter 5 Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem
  • 第六章 奇异学习理论:当费希尔矩阵退化English 中文Chapter 6 Singular Learning Theory: When the Fisher Matrix Degenerates
  • 第七章 原典并读:《周易·系辞》的变易与「位」English 中文Chapter 7 Reading the Classics: Change and "Position" in the Great Commentary of the *Book of Changes*
  • 第八章 实践与预注册检验English 中文Chapter 8 Practice and a Pre-Registered Test

How to cite · 引用

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268

BibTeX
@book{fan2026information,
  title = {{Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds}: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates},
  author = {Fan, Yafu},
  year = {2026},
  publisher = {5D Library},
  edition = {Version 1.0.0},
  doi = {10.5281/zenodo.23836268},
  url = {https://doi.org/10.5281/zenodo.23836268},
  note = {Chinese title: 信息几何、费希尔信息度量与自然梯度优化的统计流形}
}
DOIhttps://doi.org/10.5281/zenodo.23836268
DOI(全部版本 all versions)https://doi.org/10.5281/zenodo.23836267
ORCIDhttps://orcid.org/0009-0004-3519-1203
5D Library ID5DL-B-049
版本 Version1.0.0 · 中英双语版 Bilingual edition
出版日期 Date2026-10-31
网址 Onlinehttps://5dlibrary.com/books/information-geometry-fisher-natural-gradient/
SHA-256 · information-geometry-fisher-natural-gradient_bilingual_v1.0.0.pdf214f8c26c912b84c036672cf3f7cfeccd4f8eb538d6cfe6be085fb2e32d84498
SHA-256 · information-geometry-fisher-natural-gradient_en_v1.0.0.pdfb2006f4bd7947ba946cc9f053fe7e6497e5f4fdfaa55a25c3489d58ed36ab9e3
SHA-256 · information-geometry-fisher-natural-gradient_zh_v1.0.0.pdfb03c75ef39959ba0998c51fc8c9330e9fcc76bc597e65990ff3087cabc0dd8cc
Licence · 许可Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0). https://creativecommons.org/licenses/by-nc-nd/4.0/

Disclosure. The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author's business and does not recruit from the author's community, students or customers; all the code and the study protocol are published openly for anyone to check. The businesses the author runs use machine-learning and AI tools in content production and in the knowledge base, but there is no direct sales relationship between the methods discussed in this book and their commercial products.