What does Information Geometry and the Natural Gradient conclude, in numbers, about “Jeffreys's Prior: Invariance and Its Price”?
Answer: Jeffreys's prior \(\sqrt{\det G}\) is the Fisher–Rao volume element, so applying it in any coordinates gives the same distribution (an interval probability of 0.164177 in both charts in the program). But it is not "non-informative and always proper": the rule gives improper priors for a normal mean and a Poisson rate; the joint form for several parameters makes the posterior in the Neyman–Scott problem concentrate, with ever smaller standard deviation, at half the true value (0.4934), while the independent form Jeffreys recommended does not (0.9870); it depends on the sampling design, binomial and negative-binomial sampling giving different posteriors in breach of the likelihood principle. On the other hand, its intervals for a binomial proportion have frequentist coverage close to the nominal level. Reference priors are a more systematic generalisation.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.4934, 0.9870
| (1) Neyman-Scott, true sigma^2 | 1; posterior mean (sd) of sigma^2 |
| k joint rule sigma^-(k+1) independent 1/sigma | |
| 10 | 0.3435 (0.1214) 0.7729 (0.4462) |
| 100 | 0.5415 (0.0547) 1.0941 (0.1579) |
| 10,000 | 0.4934 (0.0049) 0.9870 (0.0140) |
| (2) 3 successes in 12 trials; likelihoods are proportional to p^3 (1-p)^9 | |
| binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean 0.2692, P(p < 0.1) 0.0555 | |
| neg. binomial p^-1 (1-p)^-1/2 -> Beta(3, 9.5): mean 0.2400, P(p < 0.1) 0.1000 | |
| (3) exact coverage of nominal 95 % intervals, p on a grid of 99 points | |
| n Wald: mean min Jeffreys: mean min points with coverage < 0.93 | |
| 10 | 0.776 0.096 0.952 0.869 Wald 93/99, Jeffreys 22/99 |
| 30 | 0.883 0.260 0.950 0.889 Wald 52/99, Jeffreys 8/99 |
| 100 | 0.927 0.633 0.950 0.921 Wald 20/99, Jeffreys 2/99 |
Key formula
Show the full program
# Where Jeffreys's rule helps and where it fails. (1) Neyman-Scott: k groups of 2 normal
# observations, separate means, common sigma^2 = 1. The joint rule sqrt(det G) ~
# sigma^-(k+1) gives a posterior for sigma^2 centred near 1/2 however large k is; the
# "independent" prior 1/sigma (Jeffreys's own advice for location-scale) gives ~1.
# (2) Same likelihood, different design: binomial (n fixed) and negative binomial (stop at
# the r-th success) Jeffreys priors differ, so the posteriors differ (likelihood
# principle). (3) Exact frequentist coverage of 95 % intervals for a binomial proportion:
# Wald versus the Jeffreys interval (Beta(x + 1/2, n - x + 1/2) quantiles).
import math, random, statistics
def betacf(a, b, x): # Lentz continued fraction
tiny = 1e-300; c, d = 1.0, 1 - (a + b) * x / (a + 1)
d = 1 / (d if abs(d) > tiny else tiny); h = d
for m in range(1, 300):
for num in (m * (b - m) * x / ((a + 2 * m - 1) * (a + 2 * m)),
-(a + m) * (a + b + m) * x / ((a + 2 * m) * (a + 2 * m + 1))):
d = 1 + num * d; d = 1 / (d if abs(d) > tiny else tiny)
c = 1 + num / c; c = c if abs(c) > tiny else tiny; h *= d * c
if abs(d * c - 1) < 1e-14:
break
return h
def ibeta(a, b, x): # regularised incomplete beta
if x <= 0 or x >= 1:
return 0.0 if x <= 0 else 1.0
lb = math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
f = math.exp(lb + a * math.log(x) + b * math.log(1 - x))
if x < (a + 1) / (a + b + 2):
return f * betacf(a, b, x) / a
return 1 - f * betacf(b, a, 1 - x) / b
def bquant(a, b, q):
lo, hi = 0.0, 1.0
for _ in range(60):
mid = (lo + hi) / 2
lo, hi = (mid, hi) if ibeta(a, b, mid) < q else (lo, mid)
return (lo + hi) / 2
rng = random.Random(1948)
print("(1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2")
print(" k joint rule sigma^-(k+1) independent 1/sigma")
for k in (10, 100, 10_000):
S = sum(rng.gauss(0, 1) ** 2 for _ in range(k)) # within-pair SS ~ chi2_k
for shape in (k, k / 2): # inverse-gamma(shape, S/2)
m = (S / 2) / (shape - 1); sd = m / math.sqrt(shape - 2)
if shape == k:
out = f"{m:8.4f} ({sd:.4f})"
else:
out += f" {m:8.4f} ({sd:.4f})"
print(f" {k:7,d} {out}")
x, n, r = 3, 12, 3
mb, mn = (x + 0.5) / (n + 1), x / (n + 0.5)
print(f"(2) {x} successes in {n} trials; likelihoods are proportional to p^3 (1-p)^9")
print(f" binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean {mb:.4f},"
f" P(p < 0.1) {ibeta(3.5, 9.5, 0.1):.4f}")
print(f" neg. binomial p^-1 (1-p)^-1/2 -> Beta(3, 9.5): mean {mn:.4f},"
f" P(p < 0.1) {ibeta(3, 9.5, 0.1):.4f}")
print("(3) exact coverage of nominal 95 % intervals, p on a grid of 99 points")
z = statistics.NormalDist().inv_cdf(0.975)
print(" n Wald: mean min Jeffreys: mean min points with coverage < 0.93")
for n in (10, 30, 100):
pmf = lambda p: [math.comb(n, k) * p ** k * (1 - p) ** (n - k) for k in range(n + 1)]
wald, jef = [], []
for k in range(n + 1):
ph = k / n; h = z * math.sqrt(ph * (1 - ph) / n)
wald.append((ph - h, ph + h))
jl = 0.0 if k == 0 else bquant(k + 0.5, n - k + 0.5, 0.025)
ju = 1.0 if k == n else bquant(k + 0.5, n - k + 0.5, 0.975)
jef.append((jl, ju))
cw, cj = [], []
for i in range(1, 100):
p = i / 100; f = pmf(p)
cw.append(sum(f[k] for k in range(n + 1) if wald[k][0] <= p <= wald[k][1]))
cj.append(sum(f[k] for k in range(n + 1) if jef[k][0] <= p <= jef[k][1]))
print(f" {n:3d} {statistics.fmean(cw):.3f} {min(cw):.3f} "
f"{statistics.fmean(cj):.3f} {min(cj):.3f} Wald {sum(c < .93 for c in cw)}"
f"/99, Jeffreys {sum(c < .93 for c in cj)}/99")Single source and citation
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 3: Jeffreys's Prior: Invariance and Its Price. 5D Library. https://doi.org/10.5281/zenodo.23836268
DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026049,
author = {Fan, Yafu},
title = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23836268},
url = {https://doi.org/10.5281/zenodo.23836268}
}Other chapter answers in this book
- Chapter 1 · Fisher Information: a Metric on Statistical Manifolds
- Chapter 2 · The Cramér–Rao Bound: Attained, Not Attained and "Beaten"
- Chapter 4 · The Natural Gradient: Invariance, Speed and Approximation
- Chapter 5 · Dual Flatness: Exponential Families, Legendre Duality and the Generalised Pythagorean Theorem
- Chapter 6 · Singular Learning Theory: When the Fisher Matrix Degenerates
- Chapter 8 · Practice and a Pre-Registered Test