5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文

Chapter 2 The Cramér–Rao Bound: Attained, Not Attained and “Beaten”

The bound and its history

How precisely can a parameter be estimated? Let T(X)T(X) be an unbiased estimator of τ(θ)\tau(\theta), where XX consists of nn independent observations each carrying Fisher information I(θ)I(\theta). Under regularity conditions,

Varθ(T)≥τ′(θ)2nI(θ),\mathrm{Var}_\theta(T) \;\ge\; \frac{\tau'(\theta)^2}{n\,I(\theta)} ,

and with several parameters the matrix inequality Cov(T)≽∇τ⊤G−1∇τ\mathrm{Cov}(T) \succeq \nabla\tau^\top G^{-1}\nabla\tau holds (GG being the information matrix of the whole sample). The proof takes one line: τ′(θ)=Cov(T,∂θℓ)\tau'(\theta) = \mathrm{Cov}(T, \partial_\theta \ell), followed by the Cauchy–Schwarz inequality.

Evidence

Who found it. The inequality is generally called the Cramér–Rao bound. Rao published it in 1945 in the Bulletin of the Calcutta Mathematical Society, volume 37, pages 81–91 (Rao 1945), in the same paper that proposed the geodesic distance of Chapter 1; Cramér gave it independently in his 1946 monograph Mathematical Methods of Statistics (Cramér 1946). Fréchet in 1943 (Fréchet 1943) and Darmois in 1945 (Darmois 1945) also obtained the inequality independently, which is why French literature often calls it the Fréchet–Darmois–Cramér–Rao inequality; Aitken and Silverstone had a closely related result in 1942 while discussing minimum-variance estimation (Aitken and Silverstone 1942). A systematic modern account is Lehmann and Casella (Lehmann and Casella 1998).

The bound requires two main regularity conditions: the support of the distribution does not vary with the parameter, and differentiation with respect to the parameter can be carried out under the integral sign. When the first fails, as we shall see, the bound can be “beaten” by a wide margin — not because the estimator is especially good, but because the premises of the inequality do not hold.

When it is attained exactly

The Cauchy–Schwarz inequality is an equality if and only if T−τ(θ)T – \tau(\theta) is proportional to the score: ∂θℓ=a(θ)(T−τ(θ))\partial_\theta\ell = a(\theta)\,(T – \tau(\theta)). Integrating this relation shows that the model must be an exponential family with TT as its sufficient statistic, and the target must be the expectation of TT (the mean parameter) or a linear function of it. That is the precise correction to “the bound is always attained”: at finite sample size, an unbiased estimator attaining the bound exists only in this case. In large samples the maximum-likelihood estimator is asymptotically efficient under regularity conditions, the variance of n(θ̂−θ)\sqrt n(\hat\theta – \theta) tending to 1/I(θ)1/I(\theta); Efron used statistical curvature to describe the loss of second-order efficiency away from exponential families (Efron 1975).

The program below gives exact mean-squared-error formulas for four models, checked with 10,000 simulations each: a Bernoulli mean, the rate of an exponential distribution, the variance of a normal distribution and the endpoint of a uniform distribution on (0,θ)(0,\theta). The last column is the ratio of the mean squared error to the bound.

Calculation

# When is the Cramer-Rao bound Var >= 1/(n I(theta)) attained, and when is it beaten?
# Exact formulas (with a seeded Monte Carlo check, 10,000 samples each) for: Bernoulli
# mean (efficient: exponential family in its mean parameter); exponential rate (no
# unbiased estimator attains it at finite n); normal variance (unbiased s^2 misses it,
# the biased SS/(n+1) has MSE BELOW it); uniform(0, theta) endpoint (regularity fails,
# error variance falls like 1/n^2).
import math, random, statistics

R = 10_000

def mc(rng, draw, est, truth):
    e = [est(draw(rng)) for _ in range(R)]
    return statistics.fmean(e) - truth, statistics.fmean((x - truth) ** 2 for x in e)

print("estimator                         n   bias      MSE (exact)  MSE (MC)   CRB"
      "      MSE/CRB")
for n in (10, 100):
    rng = random.Random(1945 + n)
    rows = []
    p = 0.3                                                     # Bernoulli mean
    crb = p * (1 - p) / n
    b, m = mc(rng, lambda r: [r.random() < p for _ in range(n)], statistics.fmean, p)
    rows.append(("Bernoulli p: sample mean", 0.0, crb, b, m, crb))
    lam = 2.0; crb = lam ** 2 / n                               # exponential rate
    draw = lambda r: [r.expovariate(lam) for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n - 1) / sum(x), lam)
    rows.append(("exp rate: (n-1)/sum x (unbiased)", 0.0, lam ** 2 / (n - 2), b, m, crb))
    b, m = mc(rng, draw, lambda x: n / sum(x), lam)
    ex = lam ** 2 * (n + 2) / ((n - 1) * (n - 2))
    rows.append(("exp rate: n/sum x (MLE)", lam / (n - 1), ex, b, m, crb))
    s2 = 1.0; crb = 2 * s2 ** 2 / n                             # normal variance
    draw = lambda r: [r.gauss(0, 1) for _ in range(n)]
    ss = lambda x: sum((v - statistics.fmean(x)) ** 2 for v in x)
    for lab, k in (("normal var: SS/(n-1) (unbiased)", n - 1),
                   ("normal var: SS/n (MLE)", n), ("normal var: SS/(n+1)", n + 1)):
        b, m = mc(rng, draw, lambda x: ss(x) / k, s2)
        exact = (2 * (n - 1) + (n - 1 - k) ** 2) / k ** 2
        rows.append((lab, (n - 1) / k - 1, exact, b, m, crb))
    th = 1.0; crb = th ** 2 / n                                 # formal "bound"
    draw = lambda r: [r.random() for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n + 1) / n * max(x), th)
    rows.append(("uniform(0,t): (n+1)/n max", 0.0, th ** 2 / (n * (n + 2)), b, m, crb))
    for lab, bias, ex, b, m, c in rows:
        print(f"{lab:33s}{n:4d} {bias:+.4f}  {ex:.3e}   {m:.3e}  {c:.3e}  {ex / c:6.3f}")
print("bias column: exact; MC bias agrees to within simulation error (not shown)")
print("uniform row: the 'CRB' uses the formal score -1/theta; the support depends on")
print("  theta, the regularity conditions fail and the bound does not apply")
print("no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial")
# estimator                         n   bias      MSE (exact)  MSE (MC)   CRB      MSE/CRB
# Bernoulli p: sample mean           10 +0.0000  2.100e-02   2.067e-02  2.100e-02   1.000
# exp rate: (n-1)/sum x (unbiased)   10 +0.0000  5.000e-01   5.145e-01  4.000e-01   1.250
# exp rate: n/sum x (MLE)            10 +0.2222  6.667e-01   6.683e-01  4.000e-01   1.667
# normal var: SS/(n-1) (unbiased)    10 +0.0000  2.222e-01   2.215e-01  2.000e-01   1.111
# normal var: SS/n (MLE)             10 -0.1000  1.900e-01   1.901e-01  2.000e-01   0.950
# normal var: SS/(n+1)               10 -0.1818  1.818e-01   1.788e-01  2.000e-01   0.909
# uniform(0,t): (n+1)/n max          10 +0.0000  8.333e-03   8.608e-03  1.000e-01   0.083
# Bernoulli p: sample mean          100 +0.0000  2.100e-03   2.050e-03  2.100e-03   1.000
# exp rate: (n-1)/sum x (unbiased)  100 +0.0000  4.082e-02   4.030e-02  4.000e-02   1.020
# exp rate: n/sum x (MLE)           100 +0.0202  4.205e-02   4.307e-02  4.000e-02   1.051
# normal var: SS/(n-1) (unbiased)   100 +0.0000  2.020e-02   2.045e-02  2.000e-02   1.010
# normal var: SS/n (MLE)            100 -0.0100  1.990e-02   1.988e-02  2.000e-02   0.995
# normal var: SS/(n+1)              100 -0.0198  1.980e-02   1.932e-02  2.000e-02   0.990
# uniform(0,t): (n+1)/n max         100 +0.0000  9.804e-05   9.888e-05  1.000e-02   0.010
# bias column: exact; MC bias agrees to within simulation error (not shown)
# uniform row: the 'CRB' uses the formal score -1/theta; the support depends on
#   theta, the regularity conditions fail and the bound does not apply
# no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial

The Bernoulli sample mean attains the bound exactly (ratio 1.000), because the proportion of successes is precisely the mean parameter of the binomial distribution. The exponential rate λ\lambda is not a mean parameter (the mean is 1/λ1/\lambda), so no unbiased estimator attains the bound: the best unbiased estimator (n−1)/∑x(n-1)/\sum x has variance λ2/(n−2)\lambda^2/(n-2), 1.250 times the bound at n=10n = 10 and 1.020 times at n=100n = 100; the maximum-likelihood estimator n/∑xn/\sum x has bias λ/(n−1)\lambda/(n-1) and mean squared error 1.667 and 1.051 times the bound. The unbiased estimator of a normal variance, SS/(n−1)SS/(n-1) (with SSSS the sum of squared deviations), has variance 2σ4/(n−1)2\sigma^4/(n-1), n/(n−1)n/(n-1) times the bound 2σ4/n2\sigma^4/n, i.e. 1.111 at n=10n = 10. All these ratios tend to 1 as nn grows, in line with asymptotic efficiency; but in finite samples attaining the bound is the exception, not the rule.

The last line of the program also notes that the log-odds log⁡p1−p\log\frac{p}{1-p} has no unbiased estimator at all in finite samples, because the expectation of any statistic g(X)g(X) is a polynomial in pp and the log-odds is not. For such parameters the “bound for unbiased estimators” is vacuous.

Biased estimators can go “below” the bound

The bound constrains only unbiased estimators. A biased estimator satisfies a different inequality, MSE≥(1+b′(θ))2/(nI)+b(θ)2\mathrm{MSE} \ge (1 + b'(\theta))^2/(nI) + b(\theta)^2 with bb the bias; when b′<0b' < 0 (shrinkage towards some point) the right-hand side can be smaller than the unbiased bound. The normal variance in the program gives the simplest example: dividing SSSS by nn (maximum likelihood) instead of n−1n-1 gives mean squared error 0.950 times the bound at n=10n = 10; dividing by n+1n + 1 gives 0.909 times, the smallest mean squared error among estimators of the form SS/kSS/k. Both are biased (−0.1000-0.1000 and −0.1818-0.1818), trading bias for smaller variance.

The endpoint of a uniform distribution. For X∼U(0,θ)X \sim U(0, \theta) the support depends on θ\theta, and the regularity conditions fail. If one formally plugs −1/θ-1/\theta into the formula as the “score”, the resulting “bound” is θ2/n\theta^2/n; yet the unbiased estimator n+1nmax⁡Xi\frac{n+1}{n}\max X_i has variance θ2/(n(n+2))\theta^2/(n(n+2)), only 0.083 of the “bound” at n=10n = 10 and 0.010 at n=100n = 100, its error falling like 1/n21/n^2 rather than 1/n1/n. There is no paradox: when the premises of an inequality fail, its conclusion does not apply.

James–Stein: better at every point

A deeper example comes from several dimensions. Let X∼N(θ,Id)X \sim N(\theta, I_d) with d≥3d \ge 3 and estimate θ\theta by XX itself; the total mean squared error is dd, which is also the bound for unbiased estimators. In 1956 Stein proved that for d≥3d \ge 3, XX is “inadmissible”: there is another estimator whose mean squared error is smaller for every θ\theta (Stein 1956). In 1961 James and Stein gave an explicit shrinkage estimator (James and Stein 1961):

θ̂JS=(1−d−2|X|2)X.\hat\theta_{JS} = \Big(1 – \frac{d-2}{|X|^2}\Big)X .

Calculation

# Two ways to "beat" the information bound, and what each costs. (1) James-Stein: for
# X ~ N(theta, I_d), d = 10, the shrinkage estimator (1 - (d-2)/|X|^2) X has total MSE
# below d (the bound for unbiased estimators) for EVERY theta, most at theta = 0; it is
# biased. (2) Hodges: T = 0 if |mean| < n^(-1/4), else the mean. At theta = 0 its
# n * MSE tends to 0 (superefficiency), but at theta = c / sqrt(n) its n * MSE is large:
# the bound holds asymptotically almost everywhere (Le Cam), not pointwise in finite n.
import math, random, statistics

d, R = 10, 4000
rng = random.Random(1961)
print(f"James-Stein, d = {d}, {R:,} draws per row; risk = E|est - theta|^2")
print("   |theta|   MLE risk   JS risk   JS+ risk   JS / MLE")
for norm in (0.0, 1.0, 2.0, 4.0, 8.0):
    th = [norm / math.sqrt(d)] * d
    r_ml = r_js = r_jp = 0.0
    for _ in range(R):
        x = [t + rng.gauss(0, 1) for t in th]
        s = sum(v * v for v in x); f = 1 - (d - 2) / s
        r_ml += sum((a - b) ** 2 for a, b in zip(x, th))
        r_js += sum((f * a - b) ** 2 for a, b in zip(x, th))
        r_jp += sum((max(f, 0) * a - b) ** 2 for a, b in zip(x, th))
    print(f"   {norm:6.1f}   {r_ml / R:8.3f}  {r_js / R:8.3f}  {r_jp / R:8.3f}"
          f"   {r_js / r_ml:7.3f}")
print("   exact JS risk at theta = 0: d - (d-2)^2 E[1/chi2_d] = d - (d-2) ="
      f" {d - (d - 2):.1f}")

def hodges_nmse(n, theta, reps=20000, seed=0):
    r = random.Random(seed); cut = n ** -0.25; acc = 0.0
    for _ in range(reps):
        m = theta + r.gauss(0, 1) / math.sqrt(n)        # the sample mean, exactly
        t = 0.0 if abs(m) < cut else m
        acc += (t - theta) ** 2
    return n * acc / reps

print("Hodges estimator, N(theta, 1); n * MSE (the bound is 1 for unbiased estimators)")
print("        n   theta=0   theta=1/sqrt(n)   theta=2/sqrt(n)   theta=n^(-1/4)  theta=1")
for n in (100, 10_000, 1_000_000):
    row = [hodges_nmse(n, t, seed=n + i) for i, t in
           enumerate((0.0, 1 / math.sqrt(n), 2 / math.sqrt(n), n ** -0.25, 1.0))]
    print(f"  {n:9,d} {row[0]:8.3f}  {row[1]:12.3f}  {row[2]:16.3f}  {row[3]:14.3f}"
          f"  {row[4]:8.3f}")
print("max over theta of n * MSE grows without bound as n increases (sqrt(n) scale)")
# James-Stein, d = 10, 4,000 draws per row; risk = E|est - theta|^2
#    |theta|   MLE risk   JS risk   JS+ risk   JS / MLE
#       0.0      9.914     2.037     1.233     0.205
#       1.0     10.012     2.797     2.095     0.279
#       2.0     10.049     4.350     3.985     0.433
#       4.0     10.001     7.238     7.209     0.724
#       8.0      9.867     8.976     8.976     0.910
#    exact JS risk at theta = 0: d - (d-2)^2 E[1/chi2_d] = d - (d-2) = 2.0
# Hodges estimator, N(theta, 1); n * MSE (the bound is 1 for unbiased estimators)
#         n   theta=0   theta=1/sqrt(n)   theta=2/sqrt(n)   theta=n^(-1/4)  theta=1
#         100    0.013         1.078             3.867           5.495     0.983
#      10,000    0.000         1.000             4.000          50.195     1.000
#   1,000,000    0.000         1.000             4.000         497.859     1.005
# max over theta of n * MSE grows without bound as n increases (sqrt(n) scale)

The program takes d=10d = 10. At θ=0\theta = 0 the risk of the James–Stein estimator is 2.037 (exactly d−(d−2)=2d – (d-2) = 2), about a fifth of that of XX; taking the positive part (truncating a negative shrinkage factor at zero) gives 1.233. As |θ||\theta| grows the advantage shrinks: the ratio is 0.433 at |θ|=2|\theta| = 2 and 0.910 at |θ|=8|\theta| = 8, but always below 1. What is the price? The James–Stein estimator is biased, and what it reduces is the total mean squared error: for an individual component it can be worse than XX. It shows that the bound describes the class of unbiased estimators, not all estimators.

Hodges’s superefficient estimator. In one dimension there is another way to go “below” the bound. Hodges’s estimator reports 0 when the absolute value of the sample mean is below n−1/4n^{-1/4} and the sample mean otherwise. The program shows that at θ=0\theta = 0, n⋅MSEn\cdot\mathrm{MSE} tends to zero (0.013 at n=100n = 100, already 0.000 at n=10,000n = 10{,}000), apparently beating the bound of 1; at θ=1\theta = 1 it behaves like the sample mean (1.000). But near θ=n−1/4\theta = n^{-1/4}, n⋅MSEn\cdot\mathrm{MSE} is 5.495, 50.195 and 497.859, growing without bound in nn. In 1953 Le Cam proved that such points of superefficiency can form only a set of Lebesgue measure zero (Le Cam 1953); the stronger statement is Hájek’s local asymptotic minimax theorem (Hájek 1972): taking the worst case over any 1/n1/\sqrt n neighbourhood of the true value, no estimator does better than 1/I(θ)1/I(\theta). So the precise meaning of the bound is an asymptotic limit taken as the worst case over a neighbourhood, not “at every point no estimator can do better”.

Left: n × mean squared error of three estimators of a normal variance, SS/(n−1), SS/n and SS/(n+1), against sample size, with n × the Cramér–Rao bound (= 2); the two biased estimators lie below the bound. Right: total mean squared error of the James–Stein and positive-part James–Stein estimators against |θ| for d = 10 (1,500 simulations per point); the dotted line is the risk d of the maximum-likelihood estimator.

What the bound says

Putting the chapter’s results together, the precise statement of the Cramér–Rao bound is: under regularity conditions, the variance of any unbiased estimator is at least the inverse of the information; at finite sample size only the mean parameter of an exponential family attains it exactly; maximum likelihood attains it asymptotically in regular models; biased estimators can have smaller mean squared error, but cannot be uniformly better over a neighbourhood of the true value (in the asymptotic minimax sense); and when the regularity conditions fail it does not apply at all. The geometric language of Chapter 1 gives an intuitive reading: G−1G^{-1} describes the shape of the “smallest distinguishable neighbourhood” around a point of the statistical manifold.

Hypothesis · not established · Reporting the ratio to the bound

The author’s hypothesis (not established): when applied studies report the precision of estimates, also reporting “the ratio of the actual standard error to the Cramér–Rao bound” helps readers spot cases where the model specification or the regularity conditions have gone wrong (ratios far above or far below 1). What would show it wrong: in a pre-registered set of simulated studies, some with known problems and some without, blind assessors shown reports with and without the ratio do not identify the problematic studies more accurately when the ratio is shown (the confidence interval for the difference includes zero). The book does not carry out this test.

Summary

The Cramér–Rao bound (Rao 1945, Cramér 1946, found independently by Fréchet and Darmois) constrains the variance of unbiased estimators under regularity conditions. “Always attained” is false: exact attainment occurs only for the mean parameter of an exponential family (ratio 1.000 for the Bernoulli); the best unbiased estimator of an exponential rate is 1.250 times the bound at n=10n = 10; the biased SS/(n+1)SS/(n+1) is 0.909 times; James–Stein with d=10d = 10 and θ=0\theta = 0 cuts total risk from about 10 to about 2; the “bound” for a uniform endpoint does not apply and the error falls like 1/n21/n^2; Hodges’s estimator is superefficient at one point and pays an unbounded price nearby.

References

Aitken, A. C., and H. Silverstone. 1942. “On the Estimation of Statistical Parameters.” Proceedings of the Royal Society of Edinburgh, Section A 61 (2): 186–94. https://doi.org/10.1017/S008045410000618X.
Cramér, Harald. 1946. Mathematical Methods of Statistics. Princeton University Press.
Darmois, Georges. 1945. “Sur Les Limites de La Dispersion de Certaines Estimations.” Revue de l’Institut International de Statistique 13 (1/4): 9–15.
Efron, Bradley. 1975. “Defining the Curvature of a Statistical Problem (with Applications to Second Order Efficiency).” Annals of Statistics 3 (6): 1189–242.
Fréchet, Maurice. 1943. “Sur l’extension de Certaines évaluations Statistiques Au Cas de Petits échantillons.” Revue de l’Institut International de Statistique 11 (3): 182–205.
Hájek, Jaroslav. 1972. “Local Asymptotic Minimax and Admissibility in Estimation.” Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability 1: 175–94.
James, W., and Charles Stein. 1961. “Estimation with Quadratic Loss.” Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability 1: 361–79.
Le Cam, Lucien. 1953. “On Some Asymptotic Properties of Maximum Likelihood Estimates and Related Bayes’ Estimates.” University of California Publications in Statistics 1: 277–330.
Lehmann, E. L., and George Casella. 1998. Theory of Point Estimation. 2nd ed. Springer.
Rao, C. Radhakrishna. 1945. “Information and the Accuracy Attainable in the Estimation of Statistical Parameters.” Bulletin of the Calcutta Mathematical Society 37 (3): 81–91.
Stein, Charles. 1956. “Inadmissibility of the Usual Estimator for the Mean of a Multivariate Normal Distribution.” Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability 1: 197–206.

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).