What does Information Geometry and the Natural Gradient conclude, in numbers, about “Jeffreys's Prior: Invariance and Its Price”?

Answer: Jeffreys's prior \(\sqrt{\det G}\) is the Fisher–Rao volume element, so applying it in any coordinates gives the same distribution (an interval probability of 0.164177 in both charts in the program). But it is not "non-informative and always proper": the rule gives improper priors for a normal mean and a Poisson rate; the joint form for several parameters makes the posterior in the Neyman–Scott problem concentrate, with ever smaller standard deviation, at half the true value (0.4934), while the independent form Jeffreys recommended does not (0.9870); it depends on the sampling design, binomial and negative-binomial sampling giving different posteriors in breach of the likelihood principle. On the other hand, its intervals for a binomial proportion have frequentist coverage close to the nominal level. Reference priors are a more systematic generalisation.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.4934, 0.9870

(1) Neyman-Scott, true sigma^21; posterior mean (sd) of sigma^2
k joint rule sigma^-(k+1) independent 1/sigma
100.3435 (0.1214) 0.7729 (0.4462)
1000.5415 (0.0547) 1.0941 (0.1579)
10,0000.4934 (0.0049) 0.9870 (0.0140)
(2) 3 successes in 12 trials; likelihoods are proportional to p^3 (1-p)^9
binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean 0.2692, P(p < 0.1) 0.0555
neg. binomial p^-1 (1-p)^-1/2 -> Beta(3, 9.5): mean 0.2400, P(p < 0.1) 0.1000
(3) exact coverage of nominal 95 % intervals, p on a grid of 99 points
n Wald: mean min Jeffreys: mean min points with coverage < 0.93
100.776 0.096 0.952 0.869 Wald 93/99, Jeffreys 22/99
300.883 0.260 0.950 0.889 Wald 52/99, Jeffreys 8/99
1000.927 0.633 0.950 0.921 Wald 20/99, Jeffreys 2/99

Key formula

\[\pi(\theta) \propto \sqrt{\det G(\theta)} .\]
Show the full program
# Where Jeffreys's rule helps and where it fails. (1) Neyman-Scott: k groups of 2 normal
# observations, separate means, common sigma^2 = 1. The joint rule sqrt(det G) ~
# sigma^-(k+1) gives a posterior for sigma^2 centred near 1/2 however large k is; the
# "independent" prior 1/sigma (Jeffreys's own advice for location-scale) gives ~1.
# (2) Same likelihood, different design: binomial (n fixed) and negative binomial (stop at
# the r-th success) Jeffreys priors differ, so the posteriors differ (likelihood
# principle). (3) Exact frequentist coverage of 95 % intervals for a binomial proportion:
# Wald versus the Jeffreys interval (Beta(x + 1/2, n - x + 1/2) quantiles).
import math, random, statistics

def betacf(a, b, x):                                  # Lentz continued fraction
    tiny = 1e-300; c, d = 1.0, 1 - (a + b) * x / (a + 1)
    d = 1 / (d if abs(d) > tiny else tiny); h = d
    for m in range(1, 300):
        for num in (m * (b - m) * x / ((a + 2 * m - 1) * (a + 2 * m)),
                    -(a + m) * (a + b + m) * x / ((a + 2 * m) * (a + 2 * m + 1))):
            d = 1 + num * d; d = 1 / (d if abs(d) > tiny else tiny)
            c = 1 + num / c; c = c if abs(c) > tiny else tiny; h *= d * c
        if abs(d * c - 1) < 1e-14:
            break
    return h

def ibeta(a, b, x):                                   # regularised incomplete beta
    if x <= 0 or x >= 1:
        return 0.0 if x <= 0 else 1.0
    lb = math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
    f = math.exp(lb + a * math.log(x) + b * math.log(1 - x))
    if x < (a + 1) / (a + b + 2):
        return f * betacf(a, b, x) / a
    return 1 - f * betacf(b, a, 1 - x) / b

def bquant(a, b, q):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if ibeta(a, b, mid) < q else (lo, mid)
    return (lo + hi) / 2

rng = random.Random(1948)
print("(1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2")
print("        k   joint rule sigma^-(k+1)   independent 1/sigma")
for k in (10, 100, 10_000):
    S = sum(rng.gauss(0, 1) ** 2 for _ in range(k))     # within-pair SS ~ chi2_k
    for shape in (k, k / 2):                            # inverse-gamma(shape, S/2)
        m = (S / 2) / (shape - 1); sd = m / math.sqrt(shape - 2)
        if shape == k:
            out = f"{m:8.4f} ({sd:.4f})"
        else:
            out += f"          {m:8.4f} ({sd:.4f})"
    print(f"  {k:7,d}   {out}")
x, n, r = 3, 12, 3
mb, mn = (x + 0.5) / (n + 1), x / (n + 0.5)
print(f"(2) {x} successes in {n} trials; likelihoods are proportional to p^3 (1-p)^9")
print(f"   binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean {mb:.4f},"
      f" P(p < 0.1) {ibeta(3.5, 9.5, 0.1):.4f}")
print(f"   neg. binomial p^-1 (1-p)^-1/2   -> Beta(3, 9.5):   mean {mn:.4f},"
      f" P(p < 0.1) {ibeta(3, 9.5, 0.1):.4f}")
print("(3) exact coverage of nominal 95 % intervals, p on a grid of 99 points")
z = statistics.NormalDist().inv_cdf(0.975)
print("     n   Wald: mean  min      Jeffreys: mean  min   points with coverage < 0.93")
for n in (10, 30, 100):
    pmf = lambda p: [math.comb(n, k) * p ** k * (1 - p) ** (n - k) for k in range(n + 1)]
    wald, jef = [], []
    for k in range(n + 1):
        ph = k / n; h = z * math.sqrt(ph * (1 - ph) / n)
        wald.append((ph - h, ph + h))
        jl = 0.0 if k == 0 else bquant(k + 0.5, n - k + 0.5, 0.025)
        ju = 1.0 if k == n else bquant(k + 0.5, n - k + 0.5, 0.975)
        jef.append((jl, ju))
    cw, cj = [], []
    for i in range(1, 100):
        p = i / 100; f = pmf(p)
        cw.append(sum(f[k] for k in range(n + 1) if wald[k][0] <= p <= wald[k][1]))
        cj.append(sum(f[k] for k in range(n + 1) if jef[k][0] <= p <= jef[k][1]))
    print(f"   {n:3d}   {statistics.fmean(cw):.3f}  {min(cw):.3f}       "
          f"{statistics.fmean(cj):.3f}  {min(cj):.3f}   Wald {sum(c < .93 for c in cw)}"
          f"/99, Jeffreys {sum(c < .93 for c in cj)}/99")

Single source and citation

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds (Version 1.0.0), Chapter 3: Jeffreys's Prior: Invariance and Its Price. 5D Library. https://doi.org/10.5281/zenodo.23836268

DOI: 10.5281/zenodo.23836268 · Read the monograph · 5DL-B-049 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026049,
  author    = {Fan, Yafu},
  title     = {Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23836268},
  url       = {https://doi.org/10.5281/zenodo.23836268}
}

Other chapter answers in this book