5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文
Chapter 3 Jeffreys’s Prior: Invariance and Its Price
“Uniform” depends on the coordinates
Bayesian inference needs a prior. For a parameter about which nothing is known, the most natural idea is a “uniform” prior. The trouble is that uniformity is relative to a set of coordinates. A uniform distribution on the probability of heads is no longer uniform on the log-odds ; conversely, a “uniform” prior on (over the whole real line, hence not integrable) becomes on , Haldane’s prior Beta(0, 0). The two kinds of “ignorance” give different posteriors. Book 42 discussed how priors affect decisions in Bayesian search; here we care only about the relation between priors and coordinates.
Jeffreys’s 1946 solution was to build the prior from the Fisher information (Jeffreys 1946):
By the transformation law of Chapter 1, , which is exactly how a density should change under a change of coordinates. So whichever coordinates the rule is applied in, it gives the same distribution. Geometrically, is the Riemannian volume element of the statistical manifold: Jeffreys’s prior is the prior that is “uniform in Fisher–Rao volume”.
Calculation
# Jeffreys's rule pi(theta) ~ sqrt(det G(theta)) and what "invariant" means. For a
# Bernoulli model, the rule applied in p, in the logit phi = log(p/(1-p)) and in
# u = arcsin(sqrt p) gives the SAME distribution (it is the Fisher-Rao volume, uniform in
# u). A flat prior is not invariant: flat in p and flat in logit disagree. Then the
# normalising constants: Beta(1/2, 1/2) is proper; the normal mean and Poisson rate
# rules are not.
import math
def integrate(f, a, b, n=200_000): # midpoint rule
h = (b - a) / n
return h * sum(f(a + (i + 0.5) * h) for i in range(n))
jp = lambda p: 1 / math.sqrt(p * (1 - p)) # Jeffreys in p (unnormalised)
jphi = lambda f: math.sqrt(math.exp(f)) / (1 + math.exp(f)) # sqrt(p(1-p)) in logit
Zp = math.pi # integral of jp over (0, 1)
Zphi = integrate(jphi, -60, 60)
print(f"normaliser in p: pi (= {Zp:.6f}); in logit: {Zphi:.6f}; ratio {Zphi / Zp:.6f}")
print("P(0.1 < p < 0.3) under each rule / prior:")
lo, hi = 0.1, 0.3
lg = lambda p: math.log(p / (1 - p))
a = integrate(jp, lo, hi) / Zp
b = integrate(jphi, lg(lo), lg(hi)) / Zphi
c = (math.asin(math.sqrt(hi)) - math.asin(math.sqrt(lo))) / (math.pi / 2)
print(f" Jeffreys computed in p {a:.6f} | in logit {b:.6f} | uniform in arcsin(sqrt p)"
f"\n {c:.6f}")
flat_p = hi - lo
for L in (5.0, 10.0, 50.0): # 'flat in logit' on [-L, L], the limit is degenerate
d = (min(lg(hi), L) - max(lg(lo), -L)) / (2 * L)
print(f" flat in logit on [-{L:.0f}, {L:.0f}]: {d:.6f}", end="")
print(f" | flat in p: {flat_p:.6f}")
print(" (flat in logit, as L grows, puts its mass at p near 0 and 1)")
print("normalising constants of Jeffreys's rule over growing ranges:")
for L in (10, 100, 1000, 10_000):
norm_mean = 2 * L # N(mu, 1): G = 1, flat
pois = 2 * math.sqrt(L) # Poisson: G = 1/lambda
print(f" range L = {L:6d}: normal mean on [-L, L] {norm_mean:8d};"
f" Poisson rate on (0, L] {pois:8.1f}")
print(" both diverge: improper. Beta(1/2, 1/2) integrates to 1 (proper).")
x, n = 0, 10
print(f"Bernoulli data: {x} successes in {n}; posterior means p")
print(f" Jeffreys Beta(1/2,1/2): {(x + 0.5) / (n + 1):.4f}; uniform Beta(1,1):"
f" {(x + 1) / (n + 2):.4f}")
print(" Haldane Beta(0,0) (flat in logit): improper posterior when x = 0")
# normaliser in p: pi (= 3.141593); in logit: 3.141593; ratio 1.000000
# P(0.1 < p < 0.3) under each rule / prior:
# Jeffreys computed in p 0.164177 | in logit 0.164177 | uniform in arcsin(sqrt p)
# 0.164177
# flat in logit on [-5, 5]: 0.134993 | flat in p: 0.200000
# flat in logit on [-10, 10]: 0.067496 | flat in p: 0.200000
# flat in logit on [-50, 50]: 0.013499 | flat in p: 0.200000
# (flat in logit, as L grows, puts its mass at p near 0 and 1)
# normalising constants of Jeffreys's rule over growing ranges:
# range L = 10: normal mean on [-L, L] 20; Poisson rate on (0, L] 6.3
# range L = 100: normal mean on [-L, L] 200; Poisson rate on (0, L] 20.0
# range L = 1000: normal mean on [-L, L] 2000; Poisson rate on (0, L] 63.2
# range L = 10000: normal mean on [-L, L] 20000; Poisson rate on (0, L] 200.0
# both diverge: improper. Beta(1/2, 1/2) integrates to 1 (proper).
# Bernoulli data: 0 successes in 10; posterior means p
# Jeffreys Beta(1/2,1/2): 0.0455; uniform Beta(1,1): 0.0833
# Haldane Beta(0,0) (flat in logit): improper posterior when x = 0For the Bernoulli model, applying the rule in gives , i.e. Beta(½, ½), with normalising constant ; applying it in the log-odds and integrating numerically also gives a normalising constant of 3.141593. is 0.164177 in both coordinate systems, exactly the value for “uniform in ” — the very coordinate that maps the simplex onto a sphere in Chapter 1. By contrast, the uniform prior on gives 0.2, while a uniform prior on in the log-odds gives 0.134993, 0.067496 and 0.013499 for , 10 and 50: as grows it pushes more and more mass towards near 0 and 1.
Improperness, several parameters and “non-informative”
The author’s framework calls Jeffreys’s prior “non-informative and always proper”. Both points need correcting.
Improperness. The Fisher information of a normal mean (variance known) is constant, so the rule gives , whose integral over the real line diverges; the information of a Poisson rate is , so the rule gives , whose integral over is , also divergent. The program lists the normalising constants as the range grows. An improper prior often (but not always) gives a proper posterior once data are observed; the Bernoulli example shows that with zero successes Haldane’s prior gives an improper posterior, while Beta(½, ½) gives posterior mean 0.0455 and the uniform prior 0.0833. Improper priors are especially troublesome for Bayes factors and model comparison, because arbitrary constant multiples no longer cancel (Kass and Wasserman 1996).
Several parameters. For the normal distribution , . In his Theory of Probability, Jeffreys himself did not recommend this “joint” form; he advised applying the rule separately to location and scale parameters, taking (Jeffreys 1961). One reason appears in the Neyman–Scott problem below.
Calculation
# Where Jeffreys's rule helps and where it fails. (1) Neyman-Scott: k groups of 2 normal
# observations, separate means, common sigma^2 = 1. The joint rule sqrt(det G) ~
# sigma^-(k+1) gives a posterior for sigma^2 centred near 1/2 however large k is; the
# "independent" prior 1/sigma (Jeffreys's own advice for location-scale) gives ~1.
# (2) Same likelihood, different design: binomial (n fixed) and negative binomial (stop at
# the r-th success) Jeffreys priors differ, so the posteriors differ (likelihood
# principle). (3) Exact frequentist coverage of 95 % intervals for a binomial proportion:
# Wald versus the Jeffreys interval (Beta(x + 1/2, n - x + 1/2) quantiles).
import math, random, statistics
def betacf(a, b, x): # Lentz continued fraction
tiny = 1e-300; c, d = 1.0, 1 - (a + b) * x / (a + 1)
d = 1 / (d if abs(d) > tiny else tiny); h = d
for m in range(1, 300):
for num in (m * (b - m) * x / ((a + 2 * m - 1) * (a + 2 * m)),
-(a + m) * (a + b + m) * x / ((a + 2 * m) * (a + 2 * m + 1))):
d = 1 + num * d; d = 1 / (d if abs(d) > tiny else tiny)
c = 1 + num / c; c = c if abs(c) > tiny else tiny; h *= d * c
if abs(d * c - 1) < 1e-14:
break
return h
def ibeta(a, b, x): # regularised incomplete beta
if x <= 0 or x >= 1:
return 0.0 if x <= 0 else 1.0
lb = math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
f = math.exp(lb + a * math.log(x) + b * math.log(1 - x))
if x < (a + 1) / (a + b + 2):
return f * betacf(a, b, x) / a
return 1 - f * betacf(b, a, 1 - x) / b
def bquant(a, b, q):
lo, hi = 0.0, 1.0
for _ in range(60):
mid = (lo + hi) / 2
lo, hi = (mid, hi) if ibeta(a, b, mid) < q else (lo, mid)
return (lo + hi) / 2
rng = random.Random(1948)
print("(1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2")
print(" k joint rule sigma^-(k+1) independent 1/sigma")
for k in (10, 100, 10_000):
S = sum(rng.gauss(0, 1) ** 2 for _ in range(k)) # within-pair SS ~ chi2_k
for shape in (k, k / 2): # inverse-gamma(shape, S/2)
m = (S / 2) / (shape - 1); sd = m / math.sqrt(shape - 2)
if shape == k:
out = f"{m:8.4f} ({sd:.4f})"
else:
out += f" {m:8.4f} ({sd:.4f})"
print(f" {k:7,d} {out}")
x, n, r = 3, 12, 3
mb, mn = (x + 0.5) / (n + 1), x / (n + 0.5)
print(f"(2) {x} successes in {n} trials; likelihoods are proportional to p^3 (1-p)^9")
print(f" binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean {mb:.4f},"
f" P(p < 0.1) {ibeta(3.5, 9.5, 0.1):.4f}")
print(f" neg. binomial p^-1 (1-p)^-1/2 -> Beta(3, 9.5): mean {mn:.4f},"
f" P(p < 0.1) {ibeta(3, 9.5, 0.1):.4f}")
print("(3) exact coverage of nominal 95 % intervals, p on a grid of 99 points")
z = statistics.NormalDist().inv_cdf(0.975)
print(" n Wald: mean min Jeffreys: mean min points with coverage < 0.93")
for n in (10, 30, 100):
pmf = lambda p: [math.comb(n, k) * p ** k * (1 - p) ** (n - k) for k in range(n + 1)]
wald, jef = [], []
for k in range(n + 1):
ph = k / n; h = z * math.sqrt(ph * (1 - ph) / n)
wald.append((ph - h, ph + h))
jl = 0.0 if k == 0 else bquant(k + 0.5, n - k + 0.5, 0.025)
ju = 1.0 if k == n else bquant(k + 0.5, n - k + 0.5, 0.975)
jef.append((jl, ju))
cw, cj = [], []
for i in range(1, 100):
p = i / 100; f = pmf(p)
cw.append(sum(f[k] for k in range(n + 1) if wald[k][0] <= p <= wald[k][1]))
cj.append(sum(f[k] for k in range(n + 1) if jef[k][0] <= p <= jef[k][1]))
print(f" {n:3d} {statistics.fmean(cw):.3f} {min(cw):.3f} "
f"{statistics.fmean(cj):.3f} {min(cj):.3f} Wald {sum(c < .93 for c in cw)}"
f"/99, Jeffreys {sum(c < .93 for c in cj)}/99")
# (1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2
# k joint rule sigma^-(k+1) independent 1/sigma
# 10 0.3435 (0.1214) 0.7729 (0.4462)
# 100 0.5415 (0.0547) 1.0941 (0.1579)
# 10,000 0.4934 (0.0049) 0.9870 (0.0140)
# (2) 3 successes in 12 trials; likelihoods are proportional to p^3 (1-p)^9
# binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean 0.2692, P(p < 0.1) 0.0555
# neg. binomial p^-1 (1-p)^-1/2 -> Beta(3, 9.5): mean 0.2400, P(p < 0.1) 0.1000
# (3) exact coverage of nominal 95 % intervals, p on a grid of 99 points
# n Wald: mean min Jeffreys: mean min points with coverage < 0.93
# 10 0.776 0.096 0.952 0.869 Wald 93/99, Jeffreys 22/99
# 30 0.883 0.260 0.950 0.889 Wald 52/99, Jeffreys 8/99
# 100 0.927 0.633 0.950 0.921 Wald 20/99, Jeffreys 2/99The Neyman–Scott problem. There are groups of data, each with two normal observations, different means and a common variance (Neyman and Scott 1948). As the number of groups grows there is more and more information about , and a reasonable inference should converge to the true value 1. The joint rule gives the prior — the more parameters, the stronger its preference for small ; the program shows posterior means of 0.3435, 0.5415 and 0.4934 for , 100 and 10,000, concentrating near half the true value, while the posterior standard deviation keeps shrinking (to 0.0049): the posterior becomes ever more confident about a wrong value. The independent form gives posterior means of 0.7729, 1.0941 and 0.9870, converging to the truth. This parallels Neyman and Scott’s original finding that the maximum-likelihood estimator is inconsistent here.
Dependence on the sampling design. Suppose 3 successes are observed in 12 trials. If the experimenter had decided in advance to run 12 trials (binomial sampling), Jeffreys’s prior is Beta(½, ½), the posterior is Beta(3.5, 9.5) with mean 0.2692 and ; if the experimenter had decided to stop at the third success (negative-binomial sampling), the Fisher information is , the rule gives , and the posterior is Beta(3, 9.5) with mean 0.2400 and . The two likelihood functions are proportional (both to ), so by the likelihood principle the inferences should be the same (Berger and Wolpert 1988); Jeffreys’s prior violates it because it depends on the Fisher information, an expectation over the sample space. This is one of the standard criticisms of Jeffreys’s rule (Kass and Wasserman 1996; Berger 1985).
Note
Is violating the likelihood principle a defect? That depends on one’s position. Strict likelihoodists and subjective Bayesians consider it a defect; frequentists never accepted the likelihood principle, and stopping rules affecting p-values is routine for them. The book only states the fact: Jeffreys’s prior depends on the design of the experiment, so it cannot be said to express nothing but “ignorance about the parameter”.
Where it does well
In fairness, the strengths of Jeffreys’s rule should be reported too. The third part of the program computes the exact frequentist coverage of 95% intervals for a binomial proportion: for each of 99 true values of it adds up the probabilities of all possible outcomes, with no simulation. The common Wald interval has average coverage of only 0.776 at , 0.096 at worst, with 93 of the 99 points below 0.93; at its average is 0.927, with 20 points still below 0.93. The Jeffreys interval, formed from the quantiles of Beta(+½, +½), has average coverage of about 0.95 at all three sample sizes (0.952, 0.950, 0.950), with worst values 0.869, 0.889 and 0.921. The systematic comparison by Brown, Cai and DasGupta in 2001 recommended the Wilson interval or the equal-tailed Jeffreys interval for small , and the Agresti–Coull interval for larger (Brown et al. 2001). So Jeffreys’s prior is not “non-informative”, but it can have very good frequentist properties; the two are different questions.
Reference priors: a systematic generalisation
The difficulties of Jeffreys’s rule are clearest with several parameters. In 1979 Bernardo proposed reference priors: choose the prior that maximises the expected KL divergence between prior and posterior (in the limit of a large sample), treating nuisance parameters one by one in order of importance (Bernardo 1979); Berger, Bernardo and Sun gave a formal definition in 2009 (Berger et al. 2009). In one-dimensional regular models the reference prior coincides with Jeffreys’s; with several parameters it usually gives the “independent” form Jeffreys himself recommended, such as for the normal model. Kass and Wasserman’s 1996 review compared the various “formal rules” systematically and concluded that no rule is satisfactory for every problem; they should be regarded as “reference conventions” rather than “expressions of ignorance” (Kass and Wasserman 1996).
Hypothesis · not established · Reporting prior sensitivity
The author’s hypothesis (not established): in applied Bayesian analyses that use “non-informative” priors, if posterior summaries are reported under Jeffreys’s prior, a reference prior and a weakly informative prior, then for a substantial share of published conclusions the direction or significance will change with the prior. What would show it wrong: re-running a pre-registered set of published analyses under the three priors, if the judgement of whether the 95% posterior interval contains zero agrees under all three priors in more than 95% of cases, the hypothesis fails. The book does not carry out this test.
Summary
Jeffreys’s prior is the Fisher–Rao volume element, so applying it in any coordinates gives the same distribution (an interval probability of 0.164177 in both charts in the program). But it is not “non-informative and always proper”: the rule gives improper priors for a normal mean and a Poisson rate; the joint form for several parameters makes the posterior in the Neyman–Scott problem concentrate, with ever smaller standard deviation, at half the true value (0.4934), while the independent form Jeffreys recommended does not (0.9870); it depends on the sampling design, binomial and negative-binomial sampling giving different posteriors in breach of the likelihood principle. On the other hand, its intervals for a binomial proportion have frequentist coverage close to the nominal level. Reference priors are a more systematic generalisation.
References
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).