关于「克拉美—拉奥下界:达到、达不到与「低于」」,《信息几何与自然梯度》给出的定量结论是什么?

结论:克拉美—拉奥下界(拉奥 1945、克拉美 1946,弗雷歇、达尔穆瓦独立得到)在正则条件下约束无偏估计的方差。「总能达到」是错的:恰好达到只发生在指数族的均值参数上(伯努利比值 1.000);指数率参数的最好无偏估计在 \(n = 10\) 时是下界的 1.250 倍;有偏的 \(SS/(n+1)\) 为 0.909 倍,詹姆斯—斯坦在 \(d = 10\)、\(\theta = 0\) 时把总风险从约 10 降到约 2;均匀分布端点的「下界」不适用,误差按 \(1/n^2\) 下降;霍奇斯估计在一点上超有效,却在邻近处付出无界的代价。

数据口径:第八章给出四步核查表,以及一项预注册的计算基准试验:由独立第三方冻结一组分层的合成任务,比较自然梯度(精确或 K-FAC)与 Adam 达到目标损失的墙钟时间,唯一主要结局为二者时间比的以 2 为底的对数在任务上的均值,最小关注效应 log₂1.25,90% 把握度需 393 个配对任务、计入无效任务后计划 470 个,判定规则写成代码。 书中以标注框区分已发表的证据、可运行的 Python 计算(16 段程序)、原典与作者的模型和假说;书中数据均为合成或模拟,不是对任何具体机器学习系统性能的保证。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.909, 1.000, 1.250, 10

estimator n bias MSE (exact) MSE (MC) CRB MSE/CRB
Bernoulli p: sample mean10 +0.00002.100e-022.067e-022.100e-021.000
exp rate: (n-1)/sum x (unbiased)10 +0.00005.000e-015.145e-014.000e-011.250
exp rate: n/sum x (MLE)10 +0.22226.667e-016.683e-014.000e-011.667
normal var: SS/(n-1) (unbiased)10 +0.00002.222e-012.215e-012.000e-011.111
normal var: SS/n (MLE)10 -0.10001.900e-011.901e-012.000e-010.950
normal var: SS/(n+1)10 -0.18181.818e-011.788e-012.000e-010.909
uniform(0,t): (n+1)/n max10 +0.00008.333e-038.608e-031.000e-010.083
Bernoulli p: sample mean100 +0.00002.100e-032.050e-032.100e-031.000
exp rate: (n-1)/sum x (unbiased)100 +0.00004.082e-024.030e-024.000e-021.020
exp rate: n/sum x (MLE)100 +0.02024.205e-024.307e-024.000e-021.051
normal var: SS/(n-1) (unbiased)100 +0.00002.020e-022.045e-022.000e-021.010
normal var: SS/n (MLE)100 -0.01001.990e-021.988e-022.000e-020.995
normal var: SS/(n+1)100 -0.01981.980e-021.932e-022.000e-020.990
uniform(0,t): (n+1)/n max100 +0.00009.804e-059.888e-051.000e-020.010
bias column: exact; MC bias agrees to within simulation error (not shown)
uniform row: the 'CRB' uses the formal score -1/theta; the support depends on
theta, the regularity conditions fail and the bound does not apply
no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial

核心公式

\[\mathrm{Var}_\theta(T) \;\ge\; \frac{\tau'(\theta)^2}{n\,I(\theta)} ,\]
查看完整验算程序
# When is the Cramer-Rao bound Var >= 1/(n I(theta)) attained, and when is it beaten?
# Exact formulas (with a seeded Monte Carlo check, 10,000 samples each) for: Bernoulli
# mean (efficient: exponential family in its mean parameter); exponential rate (no
# unbiased estimator attains it at finite n); normal variance (unbiased s^2 misses it,
# the biased SS/(n+1) has MSE BELOW it); uniform(0, theta) endpoint (regularity fails,
# error variance falls like 1/n^2).
import math, random, statistics

R = 10_000

def mc(rng, draw, est, truth):
    e = [est(draw(rng)) for _ in range(R)]
    return statistics.fmean(e) - truth, statistics.fmean((x - truth) ** 2 for x in e)

print("estimator                         n   bias      MSE (exact)  MSE (MC)   CRB"
      "      MSE/CRB")
for n in (10, 100):
    rng = random.Random(1945 + n)
    rows = []
    p = 0.3                                                     # Bernoulli mean
    crb = p * (1 - p) / n
    b, m = mc(rng, lambda r: [r.random() < p for _ in range(n)], statistics.fmean, p)
    rows.append(("Bernoulli p: sample mean", 0.0, crb, b, m, crb))
    lam = 2.0; crb = lam ** 2 / n                               # exponential rate
    draw = lambda r: [r.expovariate(lam) for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n - 1) / sum(x), lam)
    rows.append(("exp rate: (n-1)/sum x (unbiased)", 0.0, lam ** 2 / (n - 2), b, m, crb))
    b, m = mc(rng, draw, lambda x: n / sum(x), lam)
    ex = lam ** 2 * (n + 2) / ((n - 1) * (n - 2))
    rows.append(("exp rate: n/sum x (MLE)", lam / (n - 1), ex, b, m, crb))
    s2 = 1.0; crb = 2 * s2 ** 2 / n                             # normal variance
    draw = lambda r: [r.gauss(0, 1) for _ in range(n)]
    ss = lambda x: sum((v - statistics.fmean(x)) ** 2 for v in x)
    for lab, k in (("normal var: SS/(n-1) (unbiased)", n - 1),
                   ("normal var: SS/n (MLE)", n), ("normal var: SS/(n+1)", n + 1)):
        b, m = mc(rng, draw, lambda x: ss(x) / k, s2)
        exact = (2 * (n - 1) + (n - 1 - k) ** 2) / k ** 2
        rows.append((lab, (n - 1) / k - 1, exact, b, m, crb))
    th = 1.0; crb = th ** 2 / n                                 # formal "bound"
    draw = lambda r: [r.random() for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n + 1) / n * max(x), th)
    rows.append(("uniform(0,t): (n+1)/n max", 0.0, th ** 2 / (n * (n + 2)), b, m, crb))
    for lab, bias, ex, b, m, c in rows:
        print(f"{lab:33s}{n:4d} {bias:+.4f}  {ex:.3e}   {m:.3e}  {c:.3e}  {ex / c:6.3f}")
print("bias column: exact; MC bias agrees to within simulation error (not shown)")
print("uniform row: the 'CRB' uses the formal score -1/theta; the support depends on")
print("  theta, the regularity conditions fail and the bound does not apply")
print("no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial")

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《信息几何、费希尔信息度量与自然梯度优化的统计流形》第 2 章「克拉美—拉奥下界:达到、达不到与「低于」」(版本 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · 阅读本章所在专著 · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

本书其他章节结论