5D Library › 信息几何、费希尔信息度量与自然梯度优化的统计流形 · English

第二章 克拉美—拉奥下界:达到、达不到与「低于」

下界与它的来历

估计一个参数能准到什么程度?设 T(X)T(X) 是 τ(θ)\tau(\theta) 的无偏估计量,XX 由 nn 个独立观测组成,每个观测的费希尔信息为 I(θ)I(\theta)。在正则条件下,

Varθ(T)≥τ′(θ)2nI(θ),\mathrm{Var}_\theta(T) \;\ge\; \frac{\tau'(\theta)^2}{n\,I(\theta)} ,

多参数时为矩阵不等式 Cov(T)≽∇τ⊤G−1∇τ\mathrm{Cov}(T) \succeq \nabla\tau^\top G^{-1}\nabla\tau(GG 为全部样本的信息矩阵)。证明只需一行:τ′(θ)=Cov(T,∂θℓ)\tau'(\theta) = \mathrm{Cov}(T, \partial_\theta \ell),再用柯西—施瓦茨不等式。

证据

谁先得到。 这一不等式通称克拉美—拉奥下界。拉奥 1945 年发表于《加尔各答数学会通报》第 37 卷第 81–91 页 (Rao 1945),同一篇论文还提出了第一章的测地距离;克拉美在 1946 年的专著《统计学的数学方法》中独立给出 (Cramér 1946)。弗雷歇 1943 年 (Fréchet 1943) 与达尔穆瓦 1945 年 (Darmois 1945) 也独立得到了这一不等式,所以法文文献中常称「弗雷歇—达尔穆瓦—克拉美—拉奥不等式」;艾特肯与西尔弗斯通 1942 年在讨论最小方差估计时已有相近的结果 (Aitken and Silverstone 1942)。系统的现代表述见莱曼与卡塞拉 (Lehmann and Casella 1998)。

下界需要的正则条件主要有两条:分布的支撑不随参数变化;可以在积分号下对参数求导。第一条不满足时,后面会看到,下界可以被远远「打破」——不是因为估计量特别好,而是因为不等式的前提不成立。

什么时候恰好达到

柯西—施瓦茨不等式取等号,当且仅当 T−τ(θ)T – \tau(\theta) 与得分成比例:∂θℓ=a(θ)(T−τ(θ))\partial_\theta\ell = a(\theta)\,(T – \tau(\theta))。积分这个关系可知,模型必须是以 TT 为充分统计量的指数族,并且被估计的必须是 TT 的期望(均值参数)或它的线性函数。这就是「下界总能达到」之误的准确改正:在有限样本中,只有这一种情形下存在恰好达到下界的无偏估计量。在大样本中,极大似然估计在正则条件下是渐近有效的,n(θ̂−θ)\sqrt n(\hat\theta – \theta) 的方差趋于 1/I(θ)1/I(\theta);埃夫隆用统计曲率刻画了偏离指数族时二阶效率的损失 (Efron 1975)。

下面的程序对四种模型给出精确的均方误差公式,并用 10,000 次模拟核对:伯努利均值、指数分布的率参数、正态分布的方差、均匀分布 (0,θ)(0,\theta) 的端点。最后一列是均方误差与下界之比。

推导

# When is the Cramer-Rao bound Var >= 1/(n I(theta)) attained, and when is it beaten?
# Exact formulas (with a seeded Monte Carlo check, 10,000 samples each) for: Bernoulli
# mean (efficient: exponential family in its mean parameter); exponential rate (no
# unbiased estimator attains it at finite n); normal variance (unbiased s^2 misses it,
# the biased SS/(n+1) has MSE BELOW it); uniform(0, theta) endpoint (regularity fails,
# error variance falls like 1/n^2).
import math, random, statistics

R = 10_000

def mc(rng, draw, est, truth):
    e = [est(draw(rng)) for _ in range(R)]
    return statistics.fmean(e) - truth, statistics.fmean((x - truth) ** 2 for x in e)

print("estimator                         n   bias      MSE (exact)  MSE (MC)   CRB"
      "      MSE/CRB")
for n in (10, 100):
    rng = random.Random(1945 + n)
    rows = []
    p = 0.3                                                     # Bernoulli mean
    crb = p * (1 - p) / n
    b, m = mc(rng, lambda r: [r.random() < p for _ in range(n)], statistics.fmean, p)
    rows.append(("Bernoulli p: sample mean", 0.0, crb, b, m, crb))
    lam = 2.0; crb = lam ** 2 / n                               # exponential rate
    draw = lambda r: [r.expovariate(lam) for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n - 1) / sum(x), lam)
    rows.append(("exp rate: (n-1)/sum x (unbiased)", 0.0, lam ** 2 / (n - 2), b, m, crb))
    b, m = mc(rng, draw, lambda x: n / sum(x), lam)
    ex = lam ** 2 * (n + 2) / ((n - 1) * (n - 2))
    rows.append(("exp rate: n/sum x (MLE)", lam / (n - 1), ex, b, m, crb))
    s2 = 1.0; crb = 2 * s2 ** 2 / n                             # normal variance
    draw = lambda r: [r.gauss(0, 1) for _ in range(n)]
    ss = lambda x: sum((v - statistics.fmean(x)) ** 2 for v in x)
    for lab, k in (("normal var: SS/(n-1) (unbiased)", n - 1),
                   ("normal var: SS/n (MLE)", n), ("normal var: SS/(n+1)", n + 1)):
        b, m = mc(rng, draw, lambda x: ss(x) / k, s2)
        exact = (2 * (n - 1) + (n - 1 - k) ** 2) / k ** 2
        rows.append((lab, (n - 1) / k - 1, exact, b, m, crb))
    th = 1.0; crb = th ** 2 / n                                 # formal "bound"
    draw = lambda r: [r.random() for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n + 1) / n * max(x), th)
    rows.append(("uniform(0,t): (n+1)/n max", 0.0, th ** 2 / (n * (n + 2)), b, m, crb))
    for lab, bias, ex, b, m, c in rows:
        print(f"{lab:33s}{n:4d} {bias:+.4f}  {ex:.3e}   {m:.3e}  {c:.3e}  {ex / c:6.3f}")
print("bias column: exact; MC bias agrees to within simulation error (not shown)")
print("uniform row: the 'CRB' uses the formal score -1/theta; the support depends on")
print("  theta, the regularity conditions fail and the bound does not apply")
print("no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial")
# estimator                         n   bias      MSE (exact)  MSE (MC)   CRB      MSE/CRB
# Bernoulli p: sample mean           10 +0.0000  2.100e-02   2.067e-02  2.100e-02   1.000
# exp rate: (n-1)/sum x (unbiased)   10 +0.0000  5.000e-01   5.145e-01  4.000e-01   1.250
# exp rate: n/sum x (MLE)            10 +0.2222  6.667e-01   6.683e-01  4.000e-01   1.667
# normal var: SS/(n-1) (unbiased)    10 +0.0000  2.222e-01   2.215e-01  2.000e-01   1.111
# normal var: SS/n (MLE)             10 -0.1000  1.900e-01   1.901e-01  2.000e-01   0.950
# normal var: SS/(n+1)               10 -0.1818  1.818e-01   1.788e-01  2.000e-01   0.909
# uniform(0,t): (n+1)/n max          10 +0.0000  8.333e-03   8.608e-03  1.000e-01   0.083
# Bernoulli p: sample mean          100 +0.0000  2.100e-03   2.050e-03  2.100e-03   1.000
# exp rate: (n-1)/sum x (unbiased)  100 +0.0000  4.082e-02   4.030e-02  4.000e-02   1.020
# exp rate: n/sum x (MLE)           100 +0.0202  4.205e-02   4.307e-02  4.000e-02   1.051
# normal var: SS/(n-1) (unbiased)   100 +0.0000  2.020e-02   2.045e-02  2.000e-02   1.010
# normal var: SS/n (MLE)            100 -0.0100  1.990e-02   1.988e-02  2.000e-02   0.995
# normal var: SS/(n+1)              100 -0.0198  1.980e-02   1.932e-02  2.000e-02   0.990
# uniform(0,t): (n+1)/n max         100 +0.0000  9.804e-05   9.888e-05  1.000e-02   0.010
# bias column: exact; MC bias agrees to within simulation error (not shown)
# uniform row: the 'CRB' uses the formal score -1/theta; the support depends on
#   theta, the regularity conditions fail and the bound does not apply
# no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial

伯努利的样本均值恰好达到下界(比值 1.000),因为成功比例正是二项分布的均值参数。指数分布的率参数 λ\lambda 不是均值参数(均值是 1/λ1/\lambda),所以没有无偏估计量能达到下界:最好的无偏估计 (n−1)/∑x(n-1)/\sum x 的方差为 λ2/(n−2)\lambda^2/(n-2),n=10n = 10 时是下界的 1.250 倍,n=100n = 100 时为 1.020 倍;极大似然估计 n/∑xn/\sum x 有偏差 λ/(n−1)\lambda/(n-1),均方误差是下界的 1.667 倍与 1.051 倍。正态方差的无偏估计 SS/(n−1)SS/(n-1)(SSSS 为离差平方和)的方差为 2σ4/(n−1)2\sigma^4/(n-1),是下界 2σ4/n2\sigma^4/n 的 n/(n−1)n/(n-1) 倍,n=10n = 10 时为 1.111。这些比值都随 nn 增大而趋于 1,与渐近有效性一致;但在有限样本中,「达到下界」是例外,不是常态。

另外,程序最后一行指出:对数几率 log⁡p1−p\log\frac{p}{1-p} 在有限样本中根本没有无偏估计量,因为任何统计量 g(X)g(X) 的期望都是 pp 的多项式,而对数几率不是。所以对这样的参数,「无偏估计的下界」是空的。

有偏估计可以「低于」下界

下界只约束无偏估计量。有偏估计的均方误差满足另一个不等式 MSE≥(1+b′(θ))2/(nI)+b(θ)2\mathrm{MSE} \ge (1 + b'(\theta))^2/(nI) + b(\theta)^2,其中 bb 是偏差;当 b′<0b' < 0(向某处收缩)时,右边可以小于无偏下界。程序中的正态方差给出了最简单的例子:把 SSSS 除以 nn(极大似然)而不是 n−1n-1,n=10n = 10 时均方误差是下界的 0.950 倍;除以 n+1n + 1 时为 0.909 倍,这是形如 SS/kSS/k 的估计中均方误差最小的一个。它们都有偏差(−0.1000-0.1000 与 −0.1818-0.1818),但用偏差换来了更小的方差。

均匀分布的端点。 X∼U(0,θ)X \sim U(0, \theta) 的支撑依赖 θ\theta,正则条件不成立。若形式地把 −1/θ-1/\theta 当作「得分」代入公式,得到的「下界」是 θ2/n\theta^2/n;而无偏估计 n+1nmax⁡Xi\frac{n+1}{n}\max X_i 的方差为 θ2/(n(n+2))\theta^2/(n(n+2)),n=10n = 10 时只有「下界」的 0.083,n=100n = 100 时为 0.010,误差按 1/n21/n^2 而不是 1/n1/n 下降。这不是什么悖论:不等式的前提不满足,结论就不适用。

詹姆斯—斯坦:每一点都更好

更深刻的例子来自多维。设 X∼N(θ,Id)X \sim N(\theta, I_d),d≥3d \ge 3,用 XX 本身估计 θ\theta,总均方误差为 dd,这也是无偏估计的下界。斯坦 1956 年证明,在 d≥3d \ge 3 时 XX 是「不可容许」的:存在另一个估计量,对每一个 θ\theta 均方误差都更小 (Stein 1956)。詹姆斯与斯坦 1961 年给出了显式的收缩估计 (James and Stein 1961):

θ̂JS=(1−d−2|X|2)X.\hat\theta_{JS} = \Big(1 – \frac{d-2}{|X|^2}\Big)X .

推导

# Two ways to "beat" the information bound, and what each costs. (1) James-Stein: for
# X ~ N(theta, I_d), d = 10, the shrinkage estimator (1 - (d-2)/|X|^2) X has total MSE
# below d (the bound for unbiased estimators) for EVERY theta, most at theta = 0; it is
# biased. (2) Hodges: T = 0 if |mean| < n^(-1/4), else the mean. At theta = 0 its
# n * MSE tends to 0 (superefficiency), but at theta = c / sqrt(n) its n * MSE is large:
# the bound holds asymptotically almost everywhere (Le Cam), not pointwise in finite n.
import math, random, statistics

d, R = 10, 4000
rng = random.Random(1961)
print(f"James-Stein, d = {d}, {R:,} draws per row; risk = E|est - theta|^2")
print("   |theta|   MLE risk   JS risk   JS+ risk   JS / MLE")
for norm in (0.0, 1.0, 2.0, 4.0, 8.0):
    th = [norm / math.sqrt(d)] * d
    r_ml = r_js = r_jp = 0.0
    for _ in range(R):
        x = [t + rng.gauss(0, 1) for t in th]
        s = sum(v * v for v in x); f = 1 - (d - 2) / s
        r_ml += sum((a - b) ** 2 for a, b in zip(x, th))
        r_js += sum((f * a - b) ** 2 for a, b in zip(x, th))
        r_jp += sum((max(f, 0) * a - b) ** 2 for a, b in zip(x, th))
    print(f"   {norm:6.1f}   {r_ml / R:8.3f}  {r_js / R:8.3f}  {r_jp / R:8.3f}"
          f"   {r_js / r_ml:7.3f}")
print("   exact JS risk at theta = 0: d - (d-2)^2 E[1/chi2_d] = d - (d-2) ="
      f" {d - (d - 2):.1f}")

def hodges_nmse(n, theta, reps=20000, seed=0):
    r = random.Random(seed); cut = n ** -0.25; acc = 0.0
    for _ in range(reps):
        m = theta + r.gauss(0, 1) / math.sqrt(n)        # the sample mean, exactly
        t = 0.0 if abs(m) < cut else m
        acc += (t - theta) ** 2
    return n * acc / reps

print("Hodges estimator, N(theta, 1); n * MSE (the bound is 1 for unbiased estimators)")
print("        n   theta=0   theta=1/sqrt(n)   theta=2/sqrt(n)   theta=n^(-1/4)  theta=1")
for n in (100, 10_000, 1_000_000):
    row = [hodges_nmse(n, t, seed=n + i) for i, t in
           enumerate((0.0, 1 / math.sqrt(n), 2 / math.sqrt(n), n ** -0.25, 1.0))]
    print(f"  {n:9,d} {row[0]:8.3f}  {row[1]:12.3f}  {row[2]:16.3f}  {row[3]:14.3f}"
          f"  {row[4]:8.3f}")
print("max over theta of n * MSE grows without bound as n increases (sqrt(n) scale)")
# James-Stein, d = 10, 4,000 draws per row; risk = E|est - theta|^2
#    |theta|   MLE risk   JS risk   JS+ risk   JS / MLE
#       0.0      9.914     2.037     1.233     0.205
#       1.0     10.012     2.797     2.095     0.279
#       2.0     10.049     4.350     3.985     0.433
#       4.0     10.001     7.238     7.209     0.724
#       8.0      9.867     8.976     8.976     0.910
#    exact JS risk at theta = 0: d - (d-2)^2 E[1/chi2_d] = d - (d-2) = 2.0
# Hodges estimator, N(theta, 1); n * MSE (the bound is 1 for unbiased estimators)
#         n   theta=0   theta=1/sqrt(n)   theta=2/sqrt(n)   theta=n^(-1/4)  theta=1
#         100    0.013         1.078             3.867           5.495     0.983
#      10,000    0.000         1.000             4.000          50.195     1.000
#   1,000,000    0.000         1.000             4.000         497.859     1.005
# max over theta of n * MSE grows without bound as n increases (sqrt(n) scale)

程序取 d=10d = 10。在 θ=0\theta = 0 处,詹姆斯—斯坦估计的风险为 2.037(精确值 d−(d−2)=2d – (d-2) = 2),只有 XX 的约五分之一;取正部(把负的收缩系数截为零)后为 1.233。随着 |θ||\theta| 增大,优势逐渐变小:|θ|=2|\theta| = 2 时比值为 0.433,|θ|=8|\theta| = 8 时为 0.910,但始终小于 1。代价是什么?詹姆斯—斯坦估计有偏,而且它降低的是总均方误差:对某一个分量,它可能比 XX 更差。它说明下界描述的是无偏估计这一类,而不是所有估计。

霍奇斯的超有效估计。 一维情形还有另一种「低于」下界的方式。霍奇斯的估计量在样本均值的绝对值小于 n−1/4n^{-1/4} 时报告 0,否则报告样本均值。程序显示,在 θ=0\theta = 0 处 n⋅MSEn\cdot\mathrm{MSE} 趋于零(n=100n = 100 时 0.013,n=10,000n = 10{,}000 时已为 0.000),似乎打破了下界 1;在 θ=1\theta = 1 处与样本均值一样(1.000)。但在 θ=n−1/4\theta = n^{-1/4} 附近,n⋅MSEn\cdot\mathrm{MSE} 为 5.495、50.195 与 497.859,随 nn 无界增长。勒卡姆 1953 年证明,这样的超有效点只能构成勒贝格测度为零的集合 (Le Cam 1953);更强的说法是哈耶克的局部渐近极小极大定理 (Hájek 1972):在真值的任何 1/n1/\sqrt n 邻域内取最坏情况,没有估计量能好于 1/I(θ)1/I(\theta)。所以下界的准确含义是一个渐近的、在邻域上取最坏情况的极限,而不是「每一个点上每一个估计量都不可能更好」。

左:正态方差的三种估计 SS/(n−1)、SS/n 与 SS/(n+1) 的 n × 均方误差随样本量的变化,以及 n × 克拉美—拉奥下界(= 2);两个有偏估计在下界之下。右:d = 10 时詹姆斯—斯坦估计与正部詹姆斯—斯坦估计的总均方误差随 |θ| 的变化(每点 1,500 次模拟),虚线为极大似然估计的风险 d。

小结:下界说了什么

把本章的结果放在一起,克拉美—拉奥下界的准确表述是:在正则条件下,任何无偏估计量的方差不小于信息的逆;有限样本中恰好达到它的只有指数族的均值参数;极大似然在正则模型中渐近达到它;有偏估计可以有更小的均方误差,但不能在真值的邻域内一致地更好(渐近极小极大意义下);正则条件不满足时它根本不适用。第一章的几何语言给出一个直观的读法:G−1G^{-1} 描述的是统计流形上一个点周围「可以区分的最小邻域」的形状。

假说 · 尚未验证 · 报告下界之比

作者的假说(尚未确立): 在应用研究报告估计精度时,同时报告「实际标准误与克拉美—拉奥下界之比」,能帮助读者识别模型设定或正则条件出了问题的情形(比值远大于 1 或远小于 1)。什么样的结果能证明它是错的: 在一组事先登记的已知有问题与无问题的模拟研究中,让盲评者分别看有无这一比值的报告,若有比值时识别问题研究的准确率没有提高(差的置信区间包含零),则假说不成立。本书不做这一检验。

本章小结

克拉美—拉奥下界(拉奥 1945、克拉美 1946,弗雷歇、达尔穆瓦独立得到)在正则条件下约束无偏估计的方差。「总能达到」是错的:恰好达到只发生在指数族的均值参数上(伯努利比值 1.000);指数率参数的最好无偏估计在 n=10n = 10 时是下界的 1.250 倍;有偏的 SS/(n+1)SS/(n+1) 为 0.909 倍,詹姆斯—斯坦在 d=10d = 10、θ=0\theta = 0 时把总风险从约 10 降到约 2;均匀分布端点的「下界」不适用,误差按 1/n21/n^2 下降;霍奇斯估计在一点上超有效,却在邻近处付出无界的代价。

参考文献

Aitken, A. C., and H. Silverstone. 1942. “On the Estimation of Statistical Parameters.” Proceedings of the Royal Society of Edinburgh, Section A 61 (2): 186–94. https://doi.org/10.1017/S008045410000618X.
Cramér, Harald. 1946. Mathematical Methods of Statistics. Princeton University Press.
Darmois, Georges. 1945. “Sur Les Limites de La Dispersion de Certaines Estimations.” Revue de l’Institut International de Statistique 13 (1/4): 9–15.
Efron, Bradley. 1975. “Defining the Curvature of a Statistical Problem (with Applications to Second Order Efficiency).” Annals of Statistics 3 (6): 1189–242.
Fréchet, Maurice. 1943. “Sur l’extension de Certaines évaluations Statistiques Au Cas de Petits échantillons.” Revue de l’Institut International de Statistique 11 (3): 182–205.
Hájek, Jaroslav. 1972. “Local Asymptotic Minimax and Admissibility in Estimation.” Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability 1: 175–94.
James, W., and Charles Stein. 1961. “Estimation with Quadratic Loss.” Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability 1: 361–79.
Le Cam, Lucien. 1953. “On Some Asymptotic Properties of Maximum Likelihood Estimates and Related Bayes’ Estimates.” University of California Publications in Statistics 1: 277–330.
Lehmann, E. L., and George Casella. 1998. Theory of Point Estimation. 2nd ed. Springer.
Rao, C. Radhakrishna. 1945. “Information and the Accuracy Attainable in the Estimation of Statistical Parameters.” Bulletin of the Calcutta Mathematical Society 37 (3): 81–91.
Stein, Charles. 1956. “Inadmissibility of the Usual Estimator for the Mean of a Multivariate Normal Distribution.” Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability 1: 197–206.

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).