关于「克拉美—拉奥下界:达到、达不到与「低于」」,《信息几何与自然梯度》给出的定量结论是什么?
结论:克拉美—拉奥下界(拉奥 1945、克拉美 1946,弗雷歇、达尔穆瓦独立得到)在正则条件下约束无偏估计的方差。「总能达到」是错的:恰好达到只发生在指数族的均值参数上(伯努利比值 1.000);指数率参数的最好无偏估计在 \(n = 10\) 时是下界的 1.250 倍;有偏的 \(SS/(n+1)\) 为 0.909 倍,詹姆斯—斯坦在 \(d = 10\)、\(\theta = 0\) 时把总风险从约 10 降到约 2;均匀分布端点的「下界」不适用,误差按 \(1/n^2\) 下降;霍奇斯估计在一点上超有效,却在邻近处付出无界的代价。
数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.909, 1.000, 1.250, 10
| estimator n bias MSE (exact) MSE (MC) CRB MSE/CRB | |||||
| Bernoulli p: sample mean | 10 +0.0000 | 2.100e-02 | 2.067e-02 | 2.100e-02 | 1.000 |
| exp rate: (n-1)/sum x (unbiased) | 10 +0.0000 | 5.000e-01 | 5.145e-01 | 4.000e-01 | 1.250 |
| exp rate: n/sum x (MLE) | 10 +0.2222 | 6.667e-01 | 6.683e-01 | 4.000e-01 | 1.667 |
| normal var: SS/(n-1) (unbiased) | 10 +0.0000 | 2.222e-01 | 2.215e-01 | 2.000e-01 | 1.111 |
| normal var: SS/n (MLE) | 10 -0.1000 | 1.900e-01 | 1.901e-01 | 2.000e-01 | 0.950 |
| normal var: SS/(n+1) | 10 -0.1818 | 1.818e-01 | 1.788e-01 | 2.000e-01 | 0.909 |
| uniform(0,t): (n+1)/n max | 10 +0.0000 | 8.333e-03 | 8.608e-03 | 1.000e-01 | 0.083 |
| Bernoulli p: sample mean | 100 +0.0000 | 2.100e-03 | 2.050e-03 | 2.100e-03 | 1.000 |
| exp rate: (n-1)/sum x (unbiased) | 100 +0.0000 | 4.082e-02 | 4.030e-02 | 4.000e-02 | 1.020 |
| exp rate: n/sum x (MLE) | 100 +0.0202 | 4.205e-02 | 4.307e-02 | 4.000e-02 | 1.051 |
| normal var: SS/(n-1) (unbiased) | 100 +0.0000 | 2.020e-02 | 2.045e-02 | 2.000e-02 | 1.010 |
| normal var: SS/n (MLE) | 100 -0.0100 | 1.990e-02 | 1.988e-02 | 2.000e-02 | 0.995 |
| normal var: SS/(n+1) | 100 -0.0198 | 1.980e-02 | 1.932e-02 | 2.000e-02 | 0.990 |
| uniform(0,t): (n+1)/n max | 100 +0.0000 | 9.804e-05 | 9.888e-05 | 1.000e-02 | 0.010 |
| bias column: exact; MC bias agrees to within simulation error (not shown) | |||||
| uniform row: the 'CRB' uses the formal score -1/theta; the support depends on | |||||
| theta, the regularity conditions fail and the bound does not apply | |||||
| no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial | |||||
核心公式
查看完整验算程序
# When is the Cramer-Rao bound Var >= 1/(n I(theta)) attained, and when is it beaten?
# Exact formulas (with a seeded Monte Carlo check, 10,000 samples each) for: Bernoulli
# mean (efficient: exponential family in its mean parameter); exponential rate (no
# unbiased estimator attains it at finite n); normal variance (unbiased s^2 misses it,
# the biased SS/(n+1) has MSE BELOW it); uniform(0, theta) endpoint (regularity fails,
# error variance falls like 1/n^2).
import math, random, statistics
R = 10_000
def mc(rng, draw, est, truth):
e = [est(draw(rng)) for _ in range(R)]
return statistics.fmean(e) - truth, statistics.fmean((x - truth) ** 2 for x in e)
print("estimator n bias MSE (exact) MSE (MC) CRB"
" MSE/CRB")
for n in (10, 100):
rng = random.Random(1945 + n)
rows = []
p = 0.3 # Bernoulli mean
crb = p * (1 - p) / n
b, m = mc(rng, lambda r: [r.random() < p for _ in range(n)], statistics.fmean, p)
rows.append(("Bernoulli p: sample mean", 0.0, crb, b, m, crb))
lam = 2.0; crb = lam ** 2 / n # exponential rate
draw = lambda r: [r.expovariate(lam) for _ in range(n)]
b, m = mc(rng, draw, lambda x: (n - 1) / sum(x), lam)
rows.append(("exp rate: (n-1)/sum x (unbiased)", 0.0, lam ** 2 / (n - 2), b, m, crb))
b, m = mc(rng, draw, lambda x: n / sum(x), lam)
ex = lam ** 2 * (n + 2) / ((n - 1) * (n - 2))
rows.append(("exp rate: n/sum x (MLE)", lam / (n - 1), ex, b, m, crb))
s2 = 1.0; crb = 2 * s2 ** 2 / n # normal variance
draw = lambda r: [r.gauss(0, 1) for _ in range(n)]
ss = lambda x: sum((v - statistics.fmean(x)) ** 2 for v in x)
for lab, k in (("normal var: SS/(n-1) (unbiased)", n - 1),
("normal var: SS/n (MLE)", n), ("normal var: SS/(n+1)", n + 1)):
b, m = mc(rng, draw, lambda x: ss(x) / k, s2)
exact = (2 * (n - 1) + (n - 1 - k) ** 2) / k ** 2
rows.append((lab, (n - 1) / k - 1, exact, b, m, crb))
th = 1.0; crb = th ** 2 / n # formal "bound"
draw = lambda r: [r.random() for _ in range(n)]
b, m = mc(rng, draw, lambda x: (n + 1) / n * max(x), th)
rows.append(("uniform(0,t): (n+1)/n max", 0.0, th ** 2 / (n * (n + 2)), b, m, crb))
for lab, bias, ex, b, m, c in rows:
print(f"{lab:33s}{n:4d} {bias:+.4f} {ex:.3e} {m:.3e} {c:.3e} {ex / c:6.3f}")
print("bias column: exact; MC bias agrees to within simulation error (not shown)")
print("uniform row: the 'CRB' uses the formal score -1/theta; the support depends on")
print(" theta, the regularity conditions fail and the bound does not apply")
print("no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial")唯一数据源与引用
范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 2 章「克拉美—拉奥下界:达到、达不到与「低于」」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}