5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文

Chapter 3 Jeffreys’s Prior: Invariance and Its Price

“Uniform” depends on the coordinates

Bayesian inference needs a prior. For a parameter about which nothing is known, the most natural idea is a “uniform” prior. The trouble is that uniformity is relative to a set of coordinates. A uniform distribution on the probability pp of heads is no longer uniform on the log-odds ϕ=log⁡p1−p\phi = \log\frac{p}{1-p}; conversely, a “uniform” prior on ϕ\phi (over the whole real line, hence not integrable) becomes p−1(1−p)−1p^{-1}(1-p)^{-1} on pp, Haldane’s prior Beta(0, 0). The two kinds of “ignorance” give different posteriors. Book 42 discussed how priors affect decisions in Bayesian search; here we care only about the relation between priors and coordinates.

Jeffreys’s 1946 solution was to build the prior from the Fisher information (Jeffreys 1946):

π(θ)∝det⁡G(θ).\pi(\theta) \propto \sqrt{\det G(\theta)} .

By the transformation law Gξ=J⊤GθJG_\xi = J^\top G_\theta J of Chapter 1, det⁡Gξ=|det⁡J|det⁡Gθ\sqrt{\det G_\xi} = |\det J|\,\sqrt{\det G_\theta}, which is exactly how a density should change under a change of coordinates. So whichever coordinates the rule is applied in, it gives the same distribution. Geometrically, det⁡Gdθ\sqrt{\det G}\,d\theta is the Riemannian volume element of the statistical manifold: Jeffreys’s prior is the prior that is “uniform in Fisher–Rao volume”.

Calculation

# Jeffreys's rule pi(theta) ~ sqrt(det G(theta)) and what "invariant" means. For a
# Bernoulli model, the rule applied in p, in the logit phi = log(p/(1-p)) and in
# u = arcsin(sqrt p) gives the SAME distribution (it is the Fisher-Rao volume, uniform in
# u). A flat prior is not invariant: flat in p and flat in logit disagree. Then the
# normalising constants: Beta(1/2, 1/2) is proper; the normal mean and Poisson rate
# rules are not.
import math

def integrate(f, a, b, n=200_000):                   # midpoint rule
    h = (b - a) / n
    return h * sum(f(a + (i + 0.5) * h) for i in range(n))

jp = lambda p: 1 / math.sqrt(p * (1 - p))            # Jeffreys in p (unnormalised)
jphi = lambda f: math.sqrt(math.exp(f)) / (1 + math.exp(f))   # sqrt(p(1-p)) in logit
Zp = math.pi                                        # integral of jp over (0, 1)
Zphi = integrate(jphi, -60, 60)
print(f"normaliser in p: pi (= {Zp:.6f}); in logit: {Zphi:.6f}; ratio {Zphi / Zp:.6f}")
print("P(0.1 < p < 0.3) under each rule / prior:")
lo, hi = 0.1, 0.3
lg = lambda p: math.log(p / (1 - p))
a = integrate(jp, lo, hi) / Zp
b = integrate(jphi, lg(lo), lg(hi)) / Zphi
c = (math.asin(math.sqrt(hi)) - math.asin(math.sqrt(lo))) / (math.pi / 2)
print(f"   Jeffreys computed in p {a:.6f} | in logit {b:.6f} | uniform in arcsin(sqrt p)"
      f"\n   {c:.6f}")
flat_p = hi - lo
for L in (5.0, 10.0, 50.0):      # 'flat in logit' on [-L, L], the limit is degenerate
    d = (min(lg(hi), L) - max(lg(lo), -L)) / (2 * L)
    print(f"   flat in logit on [-{L:.0f}, {L:.0f}]: {d:.6f}", end="")
    print(f"  | flat in p: {flat_p:.6f}")
print("   (flat in logit, as L grows, puts its mass at p near 0 and 1)")
print("normalising constants of Jeffreys's rule over growing ranges:")
for L in (10, 100, 1000, 10_000):
    norm_mean = 2 * L                                     # N(mu, 1): G = 1, flat
    pois = 2 * math.sqrt(L)                               # Poisson: G = 1/lambda
    print(f"   range L = {L:6d}: normal mean on [-L, L] {norm_mean:8d};"
          f"  Poisson rate on (0, L] {pois:8.1f}")
print("   both diverge: improper. Beta(1/2, 1/2) integrates to 1 (proper).")
x, n = 0, 10
print(f"Bernoulli data: {x} successes in {n}; posterior means p")
print(f"   Jeffreys Beta(1/2,1/2): {(x + 0.5) / (n + 1):.4f};  uniform Beta(1,1):"
      f" {(x + 1) / (n + 2):.4f}")
print("   Haldane Beta(0,0) (flat in logit): improper posterior when x = 0")
# normaliser in p: pi (= 3.141593); in logit: 3.141593; ratio 1.000000
# P(0.1 < p < 0.3) under each rule / prior:
#    Jeffreys computed in p 0.164177 | in logit 0.164177 | uniform in arcsin(sqrt p)
#    0.164177
#    flat in logit on [-5, 5]: 0.134993  | flat in p: 0.200000
#    flat in logit on [-10, 10]: 0.067496  | flat in p: 0.200000
#    flat in logit on [-50, 50]: 0.013499  | flat in p: 0.200000
#    (flat in logit, as L grows, puts its mass at p near 0 and 1)
# normalising constants of Jeffreys's rule over growing ranges:
#    range L =     10: normal mean on [-L, L]       20;  Poisson rate on (0, L]      6.3
#    range L =    100: normal mean on [-L, L]      200;  Poisson rate on (0, L]     20.0
#    range L =   1000: normal mean on [-L, L]     2000;  Poisson rate on (0, L]     63.2
#    range L =  10000: normal mean on [-L, L]    20000;  Poisson rate on (0, L]    200.0
#    both diverge: improper. Beta(1/2, 1/2) integrates to 1 (proper).
# Bernoulli data: 0 successes in 10; posterior means p
#    Jeffreys Beta(1/2,1/2): 0.0455;  uniform Beta(1,1): 0.0833
#    Haldane Beta(0,0) (flat in logit): improper posterior when x = 0

For the Bernoulli model, applying the rule in pp gives p−1/2(1−p)−1/2p^{-1/2}(1-p)^{-1/2}, i.e. Beta(½, ½), with normalising constant π\pi; applying it in the log-odds and integrating numerically also gives a normalising constant of 3.141593. P(0.1<p<0.3)P(0.1 < p < 0.3) is 0.164177 in both coordinate systems, exactly the value for “uniform in arcsin⁡p\arcsin\sqrt p” — the very coordinate that maps the simplex onto a sphere in Chapter 1. By contrast, the uniform prior on pp gives 0.2, while a uniform prior on [−L,L][-L, L] in the log-odds gives 0.134993, 0.067496 and 0.013499 for L=5L = 5, 10 and 50: as LL grows it pushes more and more mass towards pp near 0 and 1.

Improperness, several parameters and “non-informative”

The author’s framework calls Jeffreys’s prior “non-informative and always proper”. Both points need correcting.

Improperness. The Fisher information of a normal mean μ\mu (variance known) is constant, so the rule gives π(μ)∝1\pi(\mu) \propto 1, whose integral over the real line diverges; the information of a Poisson rate λ\lambda is 1/λ1/\lambda, so the rule gives λ−1/2\lambda^{-1/2}, whose integral over (0,L](0, L] is 2L2\sqrt L, also divergent. The program lists the normalising constants as the range grows. An improper prior often (but not always) gives a proper posterior once data are observed; the Bernoulli example shows that with zero successes Haldane’s prior gives an improper posterior, while Beta(½, ½) gives posterior mean 0.0455 and the uniform prior 0.0833. Improper priors are especially troublesome for Bayes factors and model comparison, because arbitrary constant multiples no longer cancel (Kass and Wasserman 1996).

Several parameters. For the normal distribution N(μ,σ2)N(\mu, \sigma^2), det⁡G∝1/σ2\sqrt{\det G} \propto 1/\sigma^2. In his Theory of Probability, Jeffreys himself did not recommend this “joint” form; he advised applying the rule separately to location and scale parameters, taking π(μ,σ)∝1/σ\pi(\mu, \sigma) \propto 1/\sigma (Jeffreys 1961). One reason appears in the Neyman–Scott problem below.

Calculation

# Where Jeffreys's rule helps and where it fails. (1) Neyman-Scott: k groups of 2 normal
# observations, separate means, common sigma^2 = 1. The joint rule sqrt(det G) ~
# sigma^-(k+1) gives a posterior for sigma^2 centred near 1/2 however large k is; the
# "independent" prior 1/sigma (Jeffreys's own advice for location-scale) gives ~1.
# (2) Same likelihood, different design: binomial (n fixed) and negative binomial (stop at
# the r-th success) Jeffreys priors differ, so the posteriors differ (likelihood
# principle). (3) Exact frequentist coverage of 95 % intervals for a binomial proportion:
# Wald versus the Jeffreys interval (Beta(x + 1/2, n - x + 1/2) quantiles).
import math, random, statistics

def betacf(a, b, x):                                  # Lentz continued fraction
    tiny = 1e-300; c, d = 1.0, 1 - (a + b) * x / (a + 1)
    d = 1 / (d if abs(d) > tiny else tiny); h = d
    for m in range(1, 300):
        for num in (m * (b - m) * x / ((a + 2 * m - 1) * (a + 2 * m)),
                    -(a + m) * (a + b + m) * x / ((a + 2 * m) * (a + 2 * m + 1))):
            d = 1 + num * d; d = 1 / (d if abs(d) > tiny else tiny)
            c = 1 + num / c; c = c if abs(c) > tiny else tiny; h *= d * c
        if abs(d * c - 1) < 1e-14:
            break
    return h

def ibeta(a, b, x):                                   # regularised incomplete beta
    if x <= 0 or x >= 1:
        return 0.0 if x <= 0 else 1.0
    lb = math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
    f = math.exp(lb + a * math.log(x) + b * math.log(1 - x))
    if x < (a + 1) / (a + b + 2):
        return f * betacf(a, b, x) / a
    return 1 - f * betacf(b, a, 1 - x) / b

def bquant(a, b, q):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if ibeta(a, b, mid) < q else (lo, mid)
    return (lo + hi) / 2

rng = random.Random(1948)
print("(1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2")
print("        k   joint rule sigma^-(k+1)   independent 1/sigma")
for k in (10, 100, 10_000):
    S = sum(rng.gauss(0, 1) ** 2 for _ in range(k))     # within-pair SS ~ chi2_k
    for shape in (k, k / 2):                            # inverse-gamma(shape, S/2)
        m = (S / 2) / (shape - 1); sd = m / math.sqrt(shape - 2)
        if shape == k:
            out = f"{m:8.4f} ({sd:.4f})"
        else:
            out += f"          {m:8.4f} ({sd:.4f})"
    print(f"  {k:7,d}   {out}")
x, n, r = 3, 12, 3
mb, mn = (x + 0.5) / (n + 1), x / (n + 0.5)
print(f"(2) {x} successes in {n} trials; likelihoods are proportional to p^3 (1-p)^9")
print(f"   binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean {mb:.4f},"
      f" P(p < 0.1) {ibeta(3.5, 9.5, 0.1):.4f}")
print(f"   neg. binomial p^-1 (1-p)^-1/2   -> Beta(3, 9.5):   mean {mn:.4f},"
      f" P(p < 0.1) {ibeta(3, 9.5, 0.1):.4f}")
print("(3) exact coverage of nominal 95 % intervals, p on a grid of 99 points")
z = statistics.NormalDist().inv_cdf(0.975)
print("     n   Wald: mean  min      Jeffreys: mean  min   points with coverage < 0.93")
for n in (10, 30, 100):
    pmf = lambda p: [math.comb(n, k) * p ** k * (1 - p) ** (n - k) for k in range(n + 1)]
    wald, jef = [], []
    for k in range(n + 1):
        ph = k / n; h = z * math.sqrt(ph * (1 - ph) / n)
        wald.append((ph - h, ph + h))
        jl = 0.0 if k == 0 else bquant(k + 0.5, n - k + 0.5, 0.025)
        ju = 1.0 if k == n else bquant(k + 0.5, n - k + 0.5, 0.975)
        jef.append((jl, ju))
    cw, cj = [], []
    for i in range(1, 100):
        p = i / 100; f = pmf(p)
        cw.append(sum(f[k] for k in range(n + 1) if wald[k][0] <= p <= wald[k][1]))
        cj.append(sum(f[k] for k in range(n + 1) if jef[k][0] <= p <= jef[k][1]))
    print(f"   {n:3d}   {statistics.fmean(cw):.3f}  {min(cw):.3f}       "
          f"{statistics.fmean(cj):.3f}  {min(cj):.3f}   Wald {sum(c < .93 for c in cw)}"
          f"/99, Jeffreys {sum(c < .93 for c in cj)}/99")
# (1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2
#         k   joint rule sigma^-(k+1)   independent 1/sigma
#        10     0.3435 (0.1214)            0.7729 (0.4462)
#       100     0.5415 (0.0547)            1.0941 (0.1579)
#    10,000     0.4934 (0.0049)            0.9870 (0.0140)
# (2) 3 successes in 12 trials; likelihoods are proportional to p^3 (1-p)^9
#    binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean 0.2692, P(p < 0.1) 0.0555
#    neg. binomial p^-1 (1-p)^-1/2   -> Beta(3, 9.5):   mean 0.2400, P(p < 0.1) 0.1000
# (3) exact coverage of nominal 95 % intervals, p on a grid of 99 points
#      n   Wald: mean  min      Jeffreys: mean  min   points with coverage < 0.93
#     10   0.776  0.096       0.952  0.869   Wald 93/99, Jeffreys 22/99
#     30   0.883  0.260       0.950  0.889   Wald 52/99, Jeffreys 8/99
#    100   0.927  0.633       0.950  0.921   Wald 20/99, Jeffreys 2/99

The Neyman–Scott problem. There are kk groups of data, each with two normal observations, different means and a common variance σ2\sigma^2 (Neyman and Scott 1948). As the number of groups grows there is more and more information about σ2\sigma^2, and a reasonable inference should converge to the true value 1. The joint rule gives the prior σ−(k+1)\sigma^{-(k+1)} — the more parameters, the stronger its preference for small σ\sigma; the program shows posterior means of 0.3435, 0.5415 and 0.4934 for k=10k = 10, 100 and 10,000, concentrating near half the true value, while the posterior standard deviation keeps shrinking (to 0.0049): the posterior becomes ever more confident about a wrong value. The independent form 1/σ1/\sigma gives posterior means of 0.7729, 1.0941 and 0.9870, converging to the truth. This parallels Neyman and Scott’s original finding that the maximum-likelihood estimator is inconsistent here.

Dependence on the sampling design. Suppose 3 successes are observed in 12 trials. If the experimenter had decided in advance to run 12 trials (binomial sampling), Jeffreys’s prior is Beta(½, ½), the posterior is Beta(3.5, 9.5) with mean 0.2692 and P(p<0.1)=0.0555P(p < 0.1) = 0.0555; if the experimenter had decided to stop at the third success (negative-binomial sampling), the Fisher information is r/(p2(1−p))r/(p^2(1-p)), the rule gives p−1(1−p)−1/2p^{-1}(1-p)^{-1/2}, and the posterior is Beta(3, 9.5) with mean 0.2400 and P(p<0.1)=0.1000P(p < 0.1) = 0.1000. The two likelihood functions are proportional (both to p3(1−p)9p^3(1-p)^9), so by the likelihood principle the inferences should be the same (Berger and Wolpert 1988); Jeffreys’s prior violates it because it depends on the Fisher information, an expectation over the sample space. This is one of the standard criticisms of Jeffreys’s rule (Kass and Wasserman 1996; Berger 1985).

Note

Is violating the likelihood principle a defect? That depends on one’s position. Strict likelihoodists and subjective Bayesians consider it a defect; frequentists never accepted the likelihood principle, and stopping rules affecting p-values is routine for them. The book only states the fact: Jeffreys’s prior depends on the design of the experiment, so it cannot be said to express nothing but “ignorance about the parameter”.

Where it does well

In fairness, the strengths of Jeffreys’s rule should be reported too. The third part of the program computes the exact frequentist coverage of 95% intervals for a binomial proportion: for each of 99 true values of pp it adds up the probabilities of all possible outcomes, with no simulation. The common Wald interval p̂±1.96p̂(1−p̂)/n\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n} has average coverage of only 0.776 at n=10n = 10, 0.096 at worst, with 93 of the 99 points below 0.93; at n=100n = 100 its average is 0.927, with 20 points still below 0.93. The Jeffreys interval, formed from the quantiles of Beta(xx+½, n−xn-x+½), has average coverage of about 0.95 at all three sample sizes (0.952, 0.950, 0.950), with worst values 0.869, 0.889 and 0.921. The systematic comparison by Brown, Cai and DasGupta in 2001 recommended the Wilson interval or the equal-tailed Jeffreys interval for small nn, and the Agresti–Coull interval for larger nn (Brown et al. 2001). So Jeffreys’s prior is not “non-informative”, but it can have very good frequentist properties; the two are different questions.

Exact coverage of 95% intervals for a binomial proportion against the true value p (n = 30): the Wald interval (gold) falls badly short near p = 0 and 1, while the Jeffreys interval (blue) fluctuates around 0.95. The saw-tooth comes from the discreteness of the binomial distribution.

Reference priors: a systematic generalisation

The difficulties of Jeffreys’s rule are clearest with several parameters. In 1979 Bernardo proposed reference priors: choose the prior that maximises the expected KL divergence between prior and posterior (in the limit of a large sample), treating nuisance parameters one by one in order of importance (Bernardo 1979); Berger, Bernardo and Sun gave a formal definition in 2009 (Berger et al. 2009). In one-dimensional regular models the reference prior coincides with Jeffreys’s; with several parameters it usually gives the “independent” form Jeffreys himself recommended, such as 1/σ1/\sigma for the normal model. Kass and Wasserman’s 1996 review compared the various “formal rules” systematically and concluded that no rule is satisfactory for every problem; they should be regarded as “reference conventions” rather than “expressions of ignorance” (Kass and Wasserman 1996).

Hypothesis · not established · Reporting prior sensitivity

The author’s hypothesis (not established): in applied Bayesian analyses that use “non-informative” priors, if posterior summaries are reported under Jeffreys’s prior, a reference prior and a weakly informative prior, then for a substantial share of published conclusions the direction or significance will change with the prior. What would show it wrong: re-running a pre-registered set of published analyses under the three priors, if the judgement of whether the 95% posterior interval contains zero agrees under all three priors in more than 95% of cases, the hypothesis fails. The book does not carry out this test.

Summary

Jeffreys’s prior det⁡G\sqrt{\det G} is the Fisher–Rao volume element, so applying it in any coordinates gives the same distribution (an interval probability of 0.164177 in both charts in the program). But it is not “non-informative and always proper”: the rule gives improper priors for a normal mean and a Poisson rate; the joint form for several parameters makes the posterior in the Neyman–Scott problem concentrate, with ever smaller standard deviation, at half the true value (0.4934), while the independent form Jeffreys recommended does not (0.9870); it depends on the sampling design, binomial and negative-binomial sampling giving different posteriors in breach of the likelihood principle. On the other hand, its intervals for a binomial proportion have frequentist coverage close to the nominal level. Reference priors are a more systematic generalisation.

References

Berger, James O. 1985. Statistical Decision Theory and Bayesian Analysis. 2nd ed. Springer.
Berger, James O., José M. Bernardo, and Dongchu Sun. 2009. “The Formal Definition of Reference Priors.” Annals of Statistics 37 (2): 905–38.
Berger, James O., and Robert L. Wolpert. 1988. The Likelihood Principle. 2nd ed. Vol. 6. IMS Lecture Notes – Monograph Series. Institute of Mathematical Statistics.
Bernardo, José M. 1979. “Reference Posterior Distributions for Bayesian Inference.” Journal of the Royal Statistical Society. Series B 41 (2): 113–47.
Brown, Lawrence D., T. Tony Cai, and Anirban DasGupta. 2001. “Interval Estimation for a Binomial Proportion.” Statistical Science 16 (2): 101–33.
Jeffreys, Harold. 1946. “An Invariant Form for the Prior Probability in Estimation Problems.” Proceedings of the Royal Society of London. Series A 186 (1007): 453–61. https://doi.org/10.1098/rspa.1946.0056.
Jeffreys, Harold. 1961. Theory of Probability. 3rd ed. Oxford University Press.
Kass, Robert E., and Larry Wasserman. 1996. “The Selection of Prior Distributions by Formal Rules.” Journal of the American Statistical Association 91 (435): 1343–70.
Neyman, J., and Elizabeth L. Scott. 1948. “Consistent Estimates Based on Partially Consistent Observations.” Econometrica 16 (1): 1–32.

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).