What does Information Geometry and the Natural Gradient conclude, in numbers, about “The Cramér–Rao Bound: Attained, Not Attained and "Beaten"”?
Answer: The Cramér–Rao bound (Rao 1945, Cramér 1946, found independently by Fréchet and Darmois) constrains the variance of unbiased estimators under regularity conditions. "Always attained" is false: exact attainment occurs only for the mean parameter of an exponential family (ratio 1.000 for the Bernoulli); the best unbiased estimator of an exponential rate is 1.250 times the bound at \(n = 10\); the biased \(SS/(n+1)\) is 0.909 times; James–Stein with \(d = 10\) and \(\theta = 0\) cuts total risk from about 10 to about 2; the "bound" for a uniform endpoint does not apply and the error falls like \(1/n^2\); Hodges's estimator is superefficient at one point and pays an unbounded price nearby.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.909, 1.000, 1.250, 10
| estimator n bias MSE (exact) MSE (MC) CRB MSE/CRB | |||||
| Bernoulli p: sample mean | 10 +0.0000 | 2.100e-02 | 2.067e-02 | 2.100e-02 | 1.000 |
| exp rate: (n-1)/sum x (unbiased) | 10 +0.0000 | 5.000e-01 | 5.145e-01 | 4.000e-01 | 1.250 |
| exp rate: n/sum x (MLE) | 10 +0.2222 | 6.667e-01 | 6.683e-01 | 4.000e-01 | 1.667 |
| normal var: SS/(n-1) (unbiased) | 10 +0.0000 | 2.222e-01 | 2.215e-01 | 2.000e-01 | 1.111 |
| normal var: SS/n (MLE) | 10 -0.1000 | 1.900e-01 | 1.901e-01 | 2.000e-01 | 0.950 |
| normal var: SS/(n+1) | 10 -0.1818 | 1.818e-01 | 1.788e-01 | 2.000e-01 | 0.909 |
| uniform(0,t): (n+1)/n max | 10 +0.0000 | 8.333e-03 | 8.608e-03 | 1.000e-01 | 0.083 |
| Bernoulli p: sample mean | 100 +0.0000 | 2.100e-03 | 2.050e-03 | 2.100e-03 | 1.000 |
| exp rate: (n-1)/sum x (unbiased) | 100 +0.0000 | 4.082e-02 | 4.030e-02 | 4.000e-02 | 1.020 |
| exp rate: n/sum x (MLE) | 100 +0.0202 | 4.205e-02 | 4.307e-02 | 4.000e-02 | 1.051 |
| normal var: SS/(n-1) (unbiased) | 100 +0.0000 | 2.020e-02 | 2.045e-02 | 2.000e-02 | 1.010 |
| normal var: SS/n (MLE) | 100 -0.0100 | 1.990e-02 | 1.988e-02 | 2.000e-02 | 0.995 |
| normal var: SS/(n+1) | 100 -0.0198 | 1.980e-02 | 1.932e-02 | 2.000e-02 | 0.990 |
| uniform(0,t): (n+1)/n max | 100 +0.0000 | 9.804e-05 | 9.888e-05 | 1.000e-02 | 0.010 |
| bias column: exact; MC bias agrees to within simulation error (not shown) | |||||
| uniform row: the 'CRB' uses the formal score -1/theta; the support depends on | |||||
| theta, the regularity conditions fail and the bound does not apply | |||||
| no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial | |||||
Key formula
Show the full program
# When is the Cramer-Rao bound Var >= 1/(n I(theta)) attained, and when is it beaten?
# Exact formulas (with a seeded Monte Carlo check, 10,000 samples each) for: Bernoulli
# mean (efficient: exponential family in its mean parameter); exponential rate (no
# unbiased estimator attains it at finite n); normal variance (unbiased s^2 misses it,
# the biased SS/(n+1) has MSE BELOW it); uniform(0, theta) endpoint (regularity fails,
# error variance falls like 1/n^2).
import math, random, statistics
R = 10_000
def mc(rng, draw, est, truth):
e = [est(draw(rng)) for _ in range(R)]
return statistics.fmean(e) - truth, statistics.fmean((x - truth) ** 2 for x in e)
print("estimator n bias MSE (exact) MSE (MC) CRB"
" MSE/CRB")
for n in (10, 100):
rng = random.Random(1945 + n)
rows = []
p = 0.3 # Bernoulli mean
crb = p * (1 - p) / n
b, m = mc(rng, lambda r: [r.random() < p for _ in range(n)], statistics.fmean, p)
rows.append(("Bernoulli p: sample mean", 0.0, crb, b, m, crb))
lam = 2.0; crb = lam ** 2 / n # exponential rate
draw = lambda r: [r.expovariate(lam) for _ in range(n)]
b, m = mc(rng, draw, lambda x: (n - 1) / sum(x), lam)
rows.append(("exp rate: (n-1)/sum x (unbiased)", 0.0, lam ** 2 / (n - 2), b, m, crb))
b, m = mc(rng, draw, lambda x: n / sum(x), lam)
ex = lam ** 2 * (n + 2) / ((n - 1) * (n - 2))
rows.append(("exp rate: n/sum x (MLE)", lam / (n - 1), ex, b, m, crb))
s2 = 1.0; crb = 2 * s2 ** 2 / n # normal variance
draw = lambda r: [r.gauss(0, 1) for _ in range(n)]
ss = lambda x: sum((v - statistics.fmean(x)) ** 2 for v in x)
for lab, k in (("normal var: SS/(n-1) (unbiased)", n - 1),
("normal var: SS/n (MLE)", n), ("normal var: SS/(n+1)", n + 1)):
b, m = mc(rng, draw, lambda x: ss(x) / k, s2)
exact = (2 * (n - 1) + (n - 1 - k) ** 2) / k ** 2
rows.append((lab, (n - 1) / k - 1, exact, b, m, crb))
th = 1.0; crb = th ** 2 / n # formal "bound"
draw = lambda r: [r.random() for _ in range(n)]
b, m = mc(rng, draw, lambda x: (n + 1) / n * max(x), th)
rows.append(("uniform(0,t): (n+1)/n max", 0.0, th ** 2 / (n * (n + 2)), b, m, crb))
for lab, bias, ex, b, m, c in rows:
print(f"{lab:33s}{n:4d} {bias:+.4f} {ex:.3e} {m:.3e} {c:.3e} {ex / c:6.3f}")
print("bias column: exact; MC bias agrees to within simulation error (not shown)")
print("uniform row: the 'CRB' uses the formal score -1/theta; the support depends on")
print(" theta, the regularity conditions fail and the bound does not apply")
print("no unbiased estimator of logit(p) exists at finite n: E[g(X)] is a polynomial")Single source and citation
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 2: The Cramér–Rao Bound: Attained, Not Attained and "Beaten". 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}Other chapter answers in this book
- Chapter 1 · Fisher Information: a Metric on Statistical Manifolds
- Chapter 3 · Jeffreys's Prior: Invariance and Its Price
- Chapter 4 · The Natural Gradient: Invariance, Speed and Approximation
- Chapter 5 · Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem
- Chapter 6 · Singular Learning Theory: When the Fisher Matrix Degenerates
- Chapter 8 · Practice and a Pre-Registered Test