5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文
Chapter 2 The Cramér–Rao Bound: Attained, Not Attained and “Beaten”
The bound and its history
How precisely can a parameter be estimated? Let be an unbiased estimator of , where consists of independent observations each carrying Fisher information . Under regularity conditions,
and with several parameters the matrix inequality holds ( being the information matrix of the whole sample). The proof takes one line: , followed by the Cauchy–Schwarz inequality.
Evidence
Who found it. The inequality is generally called the Cramér–Rao bound. Rao published it in 1945 in the Bulletin of the Calcutta Mathematical Society, volume 37, pages 81–91 (Rao 1945), in the same paper that proposed the geodesic distance of Chapter 1; Cramér gave it independently in his 1946 monograph Mathematical Methods of Statistics (Cramér 1946). Fréchet in 1943 (Fréchet 1943) and Darmois in 1945 (Darmois 1945) also obtained the inequality independently, which is why French literature often calls it the Fréchet–Darmois–Cramér–Rao inequality; Aitken and Silverstone had a closely related result in 1942 while discussing minimum-variance estimation (Aitken and Silverstone 1942). A systematic modern account is Lehmann and Casella (Lehmann and Casella 1998).
The bound requires two main regularity conditions: the support of the distribution does not vary with the parameter, and differentiation with respect to the parameter can be carried out under the integral sign. When the first fails, as we shall see, the bound can be “beaten” by a wide margin — not because the estimator is especially good, but because the premises of the inequality do not hold.
When it is attained exactly
The Cauchy–Schwarz inequality is an equality if and only if is proportional to the score: . Integrating this relation shows that the model must be an exponential family with as its sufficient statistic, and the target must be the expectation of (the mean parameter) or a linear function of it. That is the precise correction to “the bound is always attained”: at finite sample size, an unbiased estimator attaining the bound exists only in this case. In large samples the maximum-likelihood estimator is asymptotically efficient under regularity conditions, the variance of tending to ; Efron used statistical curvature to describe the loss of second-order efficiency away from exponential families (Efron 1975).
The program below gives exact mean-squared-error formulas for four models, checked with 10,000 simulations each: a Bernoulli mean, the rate of an exponential distribution, the variance of a normal distribution and the endpoint of a uniform distribution on . The last column is the ratio of the mean squared error to the bound.
Calculation
# When is the Cramer-Rao bound Var >= 1/(n I(theta)) attained, and when is it beaten?
# Exact formulas (with a seeded Monte Carlo check, 10,000 samples each) for: Bernoulli
# mean (efficient: exponential family in its mean parameter); exponential rate (no
# unbiased estimator attains it at finite n); normal variance (unbiased s^2 misses it,
# the biased SS/(n+1) has MSE BELOW it); uniform(0, theta) endpoint (regularity fails,
# error variance falls like 1/n^2).
import math, random, statistics
R = 10_000
def mc(rng, draw, est, truth):
e = [est(draw(rng)) for _ in range(R)]
return statistics.fmean(e) - truth, statistics.fmean((x - truth) ** 2 for x in e)
print("estimator n bias MSE (exact) MSE (MC) CRB"
" MSE/CRB")
for n in (10, 100):
rng = random.Random(1945 + n)
rows = []
p = 0.3 # Bernoulli mean
crb = p * (1 - p) / n
b, m = mc(rng, lambda r: [r.random() < p for _ in range(n)], statistics.fmean, p)
rows.append(("Bernoulli p: sample mean", 0.0, crb, b, m, crb))
lam = 2.0; crb = lam ** 2 / n # exponential rate
draw = lambda r: [r.expovariate(lam) for _ in range(n)]
b, m = mc(rng, draw, lambda x: (n - 1) / sum(x), lam)
rows.append(("exp rate: (n-1)/sum x (unbiased)", 0.0, lam ** 2 / (n - 2), b, m, crb))
b, m = mc(rng, draw, lambda x: n / sum(x), lam)
ex = lam ** 2 * (n + 2) / ((n - 1) * (n - 2))
rows.append(("exp rate: n/sum x (MLE)", lam / (n - 1), ex, b, m, crb))
s2 = 1.0; crb = 2 * s2 ** 2 / n # normal variance
draw = lambda r: [r.gauss(0, 1) for _ in range(n)]
ss = lambda x: sum((v - statistics.fmean(x)) ** 2 for v in x)
for lab, k in (("normal var: SS/(n-1) (unbiased)", n - 1),
("normal var: SS/n (MLE)", n), ("normal var: SS/(n+1)", n + 1)):
b, m = mc(rng, draw, lambda x: ss(x) / k, s2)
exact = (2 * (n - 1) + (n - 1 - k) ** 2) / k ** 2
rows.append((lab, (n - 1) / k - 1, exact, b, m, crb))
th = 1.0; crb = th ** 2 / n # formal "bound"
draw = lambda r: [r.random() for _ in range(n)]
b, m = mc(rng, draw, lambda x: (n + 1) / n * max(x), th)
rows.append(("uniform(0,t): (n+1)/n max", 0.0, th ** 2 / (n * (n + 2)), b, m, crb))
for lab, bias, ex, b, m, c in rows:
print(f"{lab:33s}{n:4d} {bias:+.4f} {ex:.3e} {m:.3e} {c:.3e} {ex / c:6.3f}")
print("bias column: exact; MC bias agrees to within simulation error (not shown)")
print("uniform row: the 'CRB' uses the formal score -1/theta; the support depends on")
print(" theta, the regularity conditions fail and the bound does not apply")
print("no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial")
# estimator n bias MSE (exact) MSE (MC) CRB MSE/CRB
# Bernoulli p: sample mean 10 +0.0000 2.100e-02 2.067e-02 2.100e-02 1.000
# exp rate: (n-1)/sum x (unbiased) 10 +0.0000 5.000e-01 5.145e-01 4.000e-01 1.250
# exp rate: n/sum x (MLE) 10 +0.2222 6.667e-01 6.683e-01 4.000e-01 1.667
# normal var: SS/(n-1) (unbiased) 10 +0.0000 2.222e-01 2.215e-01 2.000e-01 1.111
# normal var: SS/n (MLE) 10 -0.1000 1.900e-01 1.901e-01 2.000e-01 0.950
# normal var: SS/(n+1) 10 -0.1818 1.818e-01 1.788e-01 2.000e-01 0.909
# uniform(0,t): (n+1)/n max 10 +0.0000 8.333e-03 8.608e-03 1.000e-01 0.083
# Bernoulli p: sample mean 100 +0.0000 2.100e-03 2.050e-03 2.100e-03 1.000
# exp rate: (n-1)/sum x (unbiased) 100 +0.0000 4.082e-02 4.030e-02 4.000e-02 1.020
# exp rate: n/sum x (MLE) 100 +0.0202 4.205e-02 4.307e-02 4.000e-02 1.051
# normal var: SS/(n-1) (unbiased) 100 +0.0000 2.020e-02 2.045e-02 2.000e-02 1.010
# normal var: SS/n (MLE) 100 -0.0100 1.990e-02 1.988e-02 2.000e-02 0.995
# normal var: SS/(n+1) 100 -0.0198 1.980e-02 1.932e-02 2.000e-02 0.990
# uniform(0,t): (n+1)/n max 100 +0.0000 9.804e-05 9.888e-05 1.000e-02 0.010
# bias column: exact; MC bias agrees to within simulation error (not shown)
# uniform row: the 'CRB' uses the formal score -1/theta; the support depends on
# theta, the regularity conditions fail and the bound does not apply
# no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomialThe Bernoulli sample mean attains the bound exactly (ratio 1.000), because the proportion of successes is precisely the mean parameter of the binomial distribution. The exponential rate is not a mean parameter (the mean is ), so no unbiased estimator attains the bound: the best unbiased estimator has variance , 1.250 times the bound at and 1.020 times at ; the maximum-likelihood estimator has bias and mean squared error 1.667 and 1.051 times the bound. The unbiased estimator of a normal variance, (with the sum of squared deviations), has variance , times the bound , i.e. 1.111 at . All these ratios tend to 1 as grows, in line with asymptotic efficiency; but in finite samples attaining the bound is the exception, not the rule.
The last line of the program also notes that the log-odds has no unbiased estimator at all in finite samples, because the expectation of any statistic is a polynomial in and the log-odds is not. For such parameters the “bound for unbiased estimators” is vacuous.
Biased estimators can go “below” the bound
The bound constrains only unbiased estimators. A biased estimator satisfies a different inequality, with the bias; when (shrinkage towards some point) the right-hand side can be smaller than the unbiased bound. The normal variance in the program gives the simplest example: dividing by (maximum likelihood) instead of gives mean squared error 0.950 times the bound at ; dividing by gives 0.909 times, the smallest mean squared error among estimators of the form . Both are biased ( and ), trading bias for smaller variance.
The endpoint of a uniform distribution. For the support depends on , and the regularity conditions fail. If one formally plugs into the formula as the “score”, the resulting “bound” is ; yet the unbiased estimator has variance , only 0.083 of the “bound” at and 0.010 at , its error falling like rather than . There is no paradox: when the premises of an inequality fail, its conclusion does not apply.
James–Stein: better at every point
A deeper example comes from several dimensions. Let with and estimate by itself; the total mean squared error is , which is also the bound for unbiased estimators. In 1956 Stein proved that for , is “inadmissible”: there is another estimator whose mean squared error is smaller for every (Stein 1956). In 1961 James and Stein gave an explicit shrinkage estimator (James and Stein 1961):
Calculation
# Two ways to "beat" the information bound, and what each costs. (1) James-Stein: for
# X ~ N(theta, I_d), d = 10, the shrinkage estimator (1 - (d-2)/|X|^2) X has total MSE
# below d (the bound for unbiased estimators) for EVERY theta, most at theta = 0; it is
# biased. (2) Hodges: T = 0 if |mean| < n^(-1/4), else the mean. At theta = 0 its
# n * MSE tends to 0 (superefficiency), but at theta = c / sqrt(n) its n * MSE is large:
# the bound holds asymptotically almost everywhere (Le Cam), not pointwise in finite n.
import math, random, statistics
d, R = 10, 4000
rng = random.Random(1961)
print(f"James-Stein, d = {d}, {R:,} draws per row; risk = E|est - theta|^2")
print(" |theta| MLE risk JS risk JS+ risk JS / MLE")
for norm in (0.0, 1.0, 2.0, 4.0, 8.0):
th = [norm / math.sqrt(d)] * d
r_ml = r_js = r_jp = 0.0
for _ in range(R):
x = [t + rng.gauss(0, 1) for t in th]
s = sum(v * v for v in x); f = 1 - (d - 2) / s
r_ml += sum((a - b) ** 2 for a, b in zip(x, th))
r_js += sum((f * a - b) ** 2 for a, b in zip(x, th))
r_jp += sum((max(f, 0) * a - b) ** 2 for a, b in zip(x, th))
print(f" {norm:6.1f} {r_ml / R:8.3f} {r_js / R:8.3f} {r_jp / R:8.3f}"
f" {r_js / r_ml:7.3f}")
print(" exact JS risk at theta = 0: d - (d-2)^2 E[1/chi2_d] = d - (d-2) ="
f" {d - (d - 2):.1f}")
def hodges_nmse(n, theta, reps=20000, seed=0):
r = random.Random(seed); cut = n ** -0.25; acc = 0.0
for _ in range(reps):
m = theta + r.gauss(0, 1) / math.sqrt(n) # the sample mean, exactly
t = 0.0 if abs(m) < cut else m
acc += (t - theta) ** 2
return n * acc / reps
print("Hodges estimator, N(theta, 1); n * MSE (the bound is 1 for unbiased estimators)")
print(" n theta=0 theta=1/sqrt(n) theta=2/sqrt(n) theta=n^(-1/4) theta=1")
for n in (100, 10_000, 1_000_000):
row = [hodges_nmse(n, t, seed=n + i) for i, t in
enumerate((0.0, 1 / math.sqrt(n), 2 / math.sqrt(n), n ** -0.25, 1.0))]
print(f" {n:9,d} {row[0]:8.3f} {row[1]:12.3f} {row[2]:16.3f} {row[3]:14.3f}"
f" {row[4]:8.3f}")
print("max over theta of n * MSE grows without bound as n increases (sqrt(n) scale)")
# James-Stein, d = 10, 4,000 draws per row; risk = E|est - theta|^2
# |theta| MLE risk JS risk JS+ risk JS / MLE
# 0.0 9.914 2.037 1.233 0.205
# 1.0 10.012 2.797 2.095 0.279
# 2.0 10.049 4.350 3.985 0.433
# 4.0 10.001 7.238 7.209 0.724
# 8.0 9.867 8.976 8.976 0.910
# exact JS risk at theta = 0: d - (d-2)^2 E[1/chi2_d] = d - (d-2) = 2.0
# Hodges estimator, N(theta, 1); n * MSE (the bound is 1 for unbiased estimators)
# n theta=0 theta=1/sqrt(n) theta=2/sqrt(n) theta=n^(-1/4) theta=1
# 100 0.013 1.078 3.867 5.495 0.983
# 10,000 0.000 1.000 4.000 50.195 1.000
# 1,000,000 0.000 1.000 4.000 497.859 1.005
# max over theta of n * MSE grows without bound as n increases (sqrt(n) scale)The program takes . At the risk of the James–Stein estimator is 2.037 (exactly ), about a fifth of that of ; taking the positive part (truncating a negative shrinkage factor at zero) gives 1.233. As grows the advantage shrinks: the ratio is 0.433 at and 0.910 at , but always below 1. What is the price? The James–Stein estimator is biased, and what it reduces is the total mean squared error: for an individual component it can be worse than . It shows that the bound describes the class of unbiased estimators, not all estimators.
Hodges’s superefficient estimator. In one dimension there is another way to go “below” the bound. Hodges’s estimator reports 0 when the absolute value of the sample mean is below and the sample mean otherwise. The program shows that at , tends to zero (0.013 at , already 0.000 at ), apparently beating the bound of 1; at it behaves like the sample mean (1.000). But near , is 5.495, 50.195 and 497.859, growing without bound in . In 1953 Le Cam proved that such points of superefficiency can form only a set of Lebesgue measure zero (Le Cam 1953); the stronger statement is Hájek’s local asymptotic minimax theorem (Hájek 1972): taking the worst case over any neighbourhood of the true value, no estimator does better than . So the precise meaning of the bound is an asymptotic limit taken as the worst case over a neighbourhood, not “at every point no estimator can do better”.
What the bound says
Putting the chapter’s results together, the precise statement of the Cramér–Rao bound is: under regularity conditions, the variance of any unbiased estimator is at least the inverse of the information; at finite sample size only the mean parameter of an exponential family attains it exactly; maximum likelihood attains it asymptotically in regular models; biased estimators can have smaller mean squared error, but cannot be uniformly better over a neighbourhood of the true value (in the asymptotic minimax sense); and when the regularity conditions fail it does not apply at all. The geometric language of Chapter 1 gives an intuitive reading: describes the shape of the “smallest distinguishable neighbourhood” around a point of the statistical manifold.
Hypothesis · not established · Reporting the ratio to the bound
The author’s hypothesis (not established): when applied studies report the precision of estimates, also reporting “the ratio of the actual standard error to the Cramér–Rao bound” helps readers spot cases where the model specification or the regularity conditions have gone wrong (ratios far above or far below 1). What would show it wrong: in a pre-registered set of simulated studies, some with known problems and some without, blind assessors shown reports with and without the ratio do not identify the problematic studies more accurately when the ratio is shown (the confidence interval for the difference includes zero). The book does not carry out this test.
Summary
The Cramér–Rao bound (Rao 1945, Cramér 1946, found independently by Fréchet and Darmois) constrains the variance of unbiased estimators under regularity conditions. “Always attained” is false: exact attainment occurs only for the mean parameter of an exponential family (ratio 1.000 for the Bernoulli); the best unbiased estimator of an exponential rate is 1.250 times the bound at ; the biased is 0.909 times; James–Stein with and cuts total risk from about 10 to about 2; the “bound” for a uniform endpoint does not apply and the error falls like ; Hodges’s estimator is superefficient at one point and pays an unbounded price nearby.
References
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).