What does Information Geometry and the Natural Gradient conclude, in numbers, about “The Cramér–Rao Bound: Attained, Not Attained and "Beaten"”?

Answer: The Cramér–Rao bound (Rao 1945, Cramér 1946, found independently by Fréchet and Darmois) constrains the variance of unbiased estimators under regularity conditions. "Always attained" is false: exact attainment occurs only for the mean parameter of an exponential family (ratio 1.000 for the Bernoulli); the best unbiased estimator of an exponential rate is 1.250 times the bound at \(n = 10\); the biased \(SS/(n+1)\) is 0.909 times; James–Stein with \(d = 10\) and \(\theta = 0\) cuts total risk from about 10 to about 2; the "bound" for a uniform endpoint does not apply and the error falls like \(1/n^2\); Hodges's estimator is superefficient at one point and pays an unbounded price nearby.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.909, 1.000, 1.250, 10

estimator n bias MSE (exact) MSE (MC) CRB MSE/CRB
Bernoulli p: sample mean10 +0.00002.100e-022.067e-022.100e-021.000
exp rate: (n-1)/sum x (unbiased)10 +0.00005.000e-015.145e-014.000e-011.250
exp rate: n/sum x (MLE)10 +0.22226.667e-016.683e-014.000e-011.667
normal var: SS/(n-1) (unbiased)10 +0.00002.222e-012.215e-012.000e-011.111
normal var: SS/n (MLE)10 -0.10001.900e-011.901e-012.000e-010.950
normal var: SS/(n+1)10 -0.18181.818e-011.788e-012.000e-010.909
uniform(0,t): (n+1)/n max10 +0.00008.333e-038.608e-031.000e-010.083
Bernoulli p: sample mean100 +0.00002.100e-032.050e-032.100e-031.000
exp rate: (n-1)/sum x (unbiased)100 +0.00004.082e-024.030e-024.000e-021.020
exp rate: n/sum x (MLE)100 +0.02024.205e-024.307e-024.000e-021.051
normal var: SS/(n-1) (unbiased)100 +0.00002.020e-022.045e-022.000e-021.010
normal var: SS/n (MLE)100 -0.01001.990e-021.988e-022.000e-020.995
normal var: SS/(n+1)100 -0.01981.980e-021.932e-022.000e-020.990
uniform(0,t): (n+1)/n max100 +0.00009.804e-059.888e-051.000e-020.010
bias column: exact; MC bias agrees to within simulation error (not shown)
uniform row: the 'CRB' uses the formal score -1/theta; the support depends on
theta, the regularity conditions fail and the bound does not apply
no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial

Key formula

\[\mathrm{Var}_\theta(T) \;\ge\; \frac{\tau'(\theta)^2}{n\,I(\theta)} ,\]
Show the full program
# When is the Cramer-Rao bound Var >= 1/(n I(theta)) attained, and when is it beaten?
# Exact formulas (with a seeded Monte Carlo check, 10,000 samples each) for: Bernoulli
# mean (efficient: exponential family in its mean parameter); exponential rate (no
# unbiased estimator attains it at finite n); normal variance (unbiased s^2 misses it,
# the biased SS/(n+1) has MSE BELOW it); uniform(0, theta) endpoint (regularity fails,
# error variance falls like 1/n^2).
import math, random, statistics

R = 10_000

def mc(rng, draw, est, truth):
    e = [est(draw(rng)) for _ in range(R)]
    return statistics.fmean(e) - truth, statistics.fmean((x - truth) ** 2 for x in e)

print("estimator                         n   bias      MSE (exact)  MSE (MC)   CRB"
      "      MSE/CRB")
for n in (10, 100):
    rng = random.Random(1945 + n)
    rows = []
    p = 0.3                                                     # Bernoulli mean
    crb = p * (1 - p) / n
    b, m = mc(rng, lambda r: [r.random() < p for _ in range(n)], statistics.fmean, p)
    rows.append(("Bernoulli p: sample mean", 0.0, crb, b, m, crb))
    lam = 2.0; crb = lam ** 2 / n                               # exponential rate
    draw = lambda r: [r.expovariate(lam) for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n - 1) / sum(x), lam)
    rows.append(("exp rate: (n-1)/sum x (unbiased)", 0.0, lam ** 2 / (n - 2), b, m, crb))
    b, m = mc(rng, draw, lambda x: n / sum(x), lam)
    ex = lam ** 2 * (n + 2) / ((n - 1) * (n - 2))
    rows.append(("exp rate: n/sum x (MLE)", lam / (n - 1), ex, b, m, crb))
    s2 = 1.0; crb = 2 * s2 ** 2 / n                             # normal variance
    draw = lambda r: [r.gauss(0, 1) for _ in range(n)]
    ss = lambda x: sum((v - statistics.fmean(x)) ** 2 for v in x)
    for lab, k in (("normal var: SS/(n-1) (unbiased)", n - 1),
                   ("normal var: SS/n (MLE)", n), ("normal var: SS/(n+1)", n + 1)):
        b, m = mc(rng, draw, lambda x: ss(x) / k, s2)
        exact = (2 * (n - 1) + (n - 1 - k) ** 2) / k ** 2
        rows.append((lab, (n - 1) / k - 1, exact, b, m, crb))
    th = 1.0; crb = th ** 2 / n                                 # formal "bound"
    draw = lambda r: [r.random() for _ in range(n)]
    b, m = mc(rng, draw, lambda x: (n + 1) / n * max(x), th)
    rows.append(("uniform(0,t): (n+1)/n max", 0.0, th ** 2 / (n * (n + 2)), b, m, crb))
    for lab, bias, ex, b, m, c in rows:
        print(f"{lab:33s}{n:4d} {bias:+.4f}  {ex:.3e}   {m:.3e}  {c:.3e}  {ex / c:6.3f}")
print("bias column: exact; MC bias agrees to within simulation error (not shown)")
print("uniform row: the 'CRB' uses the formal score -1/theta; the support depends on")
print("  theta, the regularity conditions fail and the bound does not apply")
print("no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial")

Single source and citation

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 2: The Cramér–Rao Bound: Attained, Not Attained and "Beaten". 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

Other chapter answers in this book