5D Library › 信息几何、费希尔信息度量与自然梯度优化的统计流形 · English

第三章 杰弗里斯先验:不变性及其代价

「均匀」依赖坐标

贝叶斯推断需要先验。对一个一无所知的参数,最自然的想法是取「均匀」先验。问题在于:均匀是相对于某一套坐标而言的。对硬币正面的概率 pp 取均匀分布,换算到对数几率 ϕ=log⁡p1−p\phi = \log\frac{p}{1-p} 上就不再均匀;反过来,对 ϕ\phi 取「均匀」(在整条实轴上,因而不可积),换算到 pp 上就是 p−1(1−p)−1p^{-1}(1-p)^{-1},即霍尔丹先验 Beta(0, 0)。两种「无知」给出不同的后验。第 42 本在贝叶斯搜索中讨论过先验对决策的影响,这里只关心先验与坐标的关系。

杰弗里斯 1946 年提出的办法是用费希尔信息构造先验 (Jeffreys 1946):

π(θ)∝det⁡G(θ).\pi(\theta) \propto \sqrt{\det G(\theta)} .

由第一章的变换规律 Gξ=J⊤GθJG_\xi = J^\top G_\theta J,det⁡Gξ=|det⁡J|det⁡Gθ\sqrt{\det G_\xi} = |\det J|\,\sqrt{\det G_\theta},这恰好是密度在换坐标时应有的变化方式。所以无论在哪一套坐标中应用这条规则,得到的都是同一个分布。几何上,det⁡Gdθ\sqrt{\det G}\,d\theta 就是统计流形上的黎曼体积元:杰弗里斯先验是「按费希尔—拉奥体积均匀」的先验。

推导

# Jeffreys's rule pi(theta) ~ sqrt(det G(theta)) and what "invariant" means. For a
# Bernoulli model, the rule applied in p, in the logit phi = log(p/(1-p)) and in
# u = arcsin(sqrt p) gives the SAME distribution (it is the Fisher-Rao volume, uniform in
# u). A flat prior is not invariant: flat in p and flat in logit disagree. Then the
# normalising constants: Beta(1/2, 1/2) is proper; the normal mean and Poisson rate
# rules are not.
import math

def integrate(f, a, b, n=200_000):                   # midpoint rule
    h = (b - a) / n
    return h * sum(f(a + (i + 0.5) * h) for i in range(n))

jp = lambda p: 1 / math.sqrt(p * (1 - p))            # Jeffreys in p (unnormalised)
jphi = lambda f: math.sqrt(math.exp(f)) / (1 + math.exp(f))   # sqrt(p(1-p)) in logit
Zp = math.pi                                        # integral of jp over (0, 1)
Zphi = integrate(jphi, -60, 60)
print(f"normaliser in p: pi (= {Zp:.6f}); in logit: {Zphi:.6f}; ratio {Zphi / Zp:.6f}")
print("P(0.1 < p < 0.3) under each rule / prior:")
lo, hi = 0.1, 0.3
lg = lambda p: math.log(p / (1 - p))
a = integrate(jp, lo, hi) / Zp
b = integrate(jphi, lg(lo), lg(hi)) / Zphi
c = (math.asin(math.sqrt(hi)) - math.asin(math.sqrt(lo))) / (math.pi / 2)
print(f"   Jeffreys computed in p {a:.6f} | in logit {b:.6f} | uniform in arcsin(sqrt p)"
      f"\n   {c:.6f}")
flat_p = hi - lo
for L in (5.0, 10.0, 50.0):      # 'flat in logit' on [-L, L], the limit is degenerate
    d = (min(lg(hi), L) - max(lg(lo), -L)) / (2 * L)
    print(f"   flat in logit on [-{L:.0f}, {L:.0f}]: {d:.6f}", end="")
    print(f"  | flat in p: {flat_p:.6f}")
print("   (flat in logit, as L grows, puts its mass at p near 0 and 1)")
print("normalising constants of Jeffreys's rule over growing ranges:")
for L in (10, 100, 1000, 10_000):
    norm_mean = 2 * L                                     # N(mu, 1): G = 1, flat
    pois = 2 * math.sqrt(L)                               # Poisson: G = 1/lambda
    print(f"   range L = {L:6d}: normal mean on [-L, L] {norm_mean:8d};"
          f"  Poisson rate on (0, L] {pois:8.1f}")
print("   both diverge: improper. Beta(1/2, 1/2) integrates to 1 (proper).")
x, n = 0, 10
print(f"Bernoulli data: {x} successes in {n}; posterior means p")
print(f"   Jeffreys Beta(1/2,1/2): {(x + 0.5) / (n + 1):.4f};  uniform Beta(1,1):"
      f" {(x + 1) / (n + 2):.4f}")
print("   Haldane Beta(0,0) (flat in logit): improper posterior when x = 0")
# normaliser in p: pi (= 3.141593); in logit: 3.141593; ratio 1.000000
# P(0.1 < p < 0.3) under each rule / prior:
#    Jeffreys computed in p 0.164177 | in logit 0.164177 | uniform in arcsin(sqrt p)
#    0.164177
#    flat in logit on [-5, 5]: 0.134993  | flat in p: 0.200000
#    flat in logit on [-10, 10]: 0.067496  | flat in p: 0.200000
#    flat in logit on [-50, 50]: 0.013499  | flat in p: 0.200000
#    (flat in logit, as L grows, puts its mass at p near 0 and 1)
# normalising constants of Jeffreys's rule over growing ranges:
#    range L =     10: normal mean on [-L, L]       20;  Poisson rate on (0, L]      6.3
#    range L =    100: normal mean on [-L, L]      200;  Poisson rate on (0, L]     20.0
#    range L =   1000: normal mean on [-L, L]     2000;  Poisson rate on (0, L]     63.2
#    range L =  10000: normal mean on [-L, L]    20000;  Poisson rate on (0, L]    200.0
#    both diverge: improper. Beta(1/2, 1/2) integrates to 1 (proper).
# Bernoulli data: 0 successes in 10; posterior means p
#    Jeffreys Beta(1/2,1/2): 0.0455;  uniform Beta(1,1): 0.0833
#    Haldane Beta(0,0) (flat in logit): improper posterior when x = 0

对伯努利模型,在 pp 中应用规则得到 p−1/2(1−p)−1/2p^{-1/2}(1-p)^{-1/2},即 Beta(½, ½),归一化常数为 π\pi;在对数几率中应用规则,数值积分得到的归一化常数也是 3.141593。P(0.1<p<0.3)P(0.1 < p < 0.3) 在两套坐标中都是 0.164177,与「在 arcsin⁡p\arcsin\sqrt p 上均匀」完全一致——这正是第一章中把单纯形映到球面的那个坐标。相反,均匀先验在 pp 上给出 0.2,而在对数几率上取 [−L,L][-L, L] 上的均匀分布,L=5L = 5、10、50 时分别给出 0.134993、0.067496 与 0.013499:随着 LL 增大,它把质量越来越多地放到 pp 接近 0 与 1 的地方。

不可积、多参数与「无信息」

作者框架把杰弗里斯先验说成「无信息的、总是合适的」。这两点都需要改正。

不可积。 正态均值 μ\mu(方差已知)的费希尔信息是常数,规则给出 π(μ)∝1\pi(\mu) \propto 1,在实轴上积分发散;泊松率 λ\lambda 的信息为 1/λ1/\lambda,规则给出 λ−1/2\lambda^{-1/2},在 (0,L](0, L] 上的积分为 2L2\sqrt L,同样发散。程序列出了随范围增长的归一化常数。不可积的先验在观测到数据后常常(但不总是)给出可积的后验;伯努利的例子显示,零次成功时霍尔丹先验的后验不可积,而 Beta(½, ½) 的后验均值为 0.0455,均匀先验为 0.0833。不可积先验在贝叶斯因子与模型比较中尤其麻烦,因为任意常数倍不再相互抵消 (Kass and Wasserman 1996)。

多参数。 对正态分布 N(μ,σ2)N(\mu, \sigma^2),det⁡G∝1/σ2\sqrt{\det G} \propto 1/\sigma^2。杰弗里斯本人在《概率论》中并不推荐这个「联合」形式,而是建议对位置参数与尺度参数分别应用规则,取 π(μ,σ)∝1/σ\pi(\mu, \sigma) \propto 1/\sigma (Jeffreys 1961)。理由之一见下面的内曼—斯科特问题。

推导

# Where Jeffreys's rule helps and where it fails. (1) Neyman-Scott: k groups of 2 normal
# observations, separate means, common sigma^2 = 1. The joint rule sqrt(det G) ~
# sigma^-(k+1) gives a posterior for sigma^2 centred near 1/2 however large k is; the
# "independent" prior 1/sigma (Jeffreys's own advice for location-scale) gives ~1.
# (2) Same likelihood, different design: binomial (n fixed) and negative binomial (stop at
# the r-th success) Jeffreys priors differ, so the posteriors differ (likelihood
# principle). (3) Exact frequentist coverage of 95 % intervals for a binomial proportion:
# Wald versus the Jeffreys interval (Beta(x + 1/2, n - x + 1/2) quantiles).
import math, random, statistics

def betacf(a, b, x):                                  # Lentz continued fraction
    tiny = 1e-300; c, d = 1.0, 1 - (a + b) * x / (a + 1)
    d = 1 / (d if abs(d) > tiny else tiny); h = d
    for m in range(1, 300):
        for num in (m * (b - m) * x / ((a + 2 * m - 1) * (a + 2 * m)),
                    -(a + m) * (a + b + m) * x / ((a + 2 * m) * (a + 2 * m + 1))):
            d = 1 + num * d; d = 1 / (d if abs(d) > tiny else tiny)
            c = 1 + num / c; c = c if abs(c) > tiny else tiny; h *= d * c
        if abs(d * c - 1) < 1e-14:
            break
    return h

def ibeta(a, b, x):                                   # regularised incomplete beta
    if x <= 0 or x >= 1:
        return 0.0 if x <= 0 else 1.0
    lb = math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b)
    f = math.exp(lb + a * math.log(x) + b * math.log(1 - x))
    if x < (a + 1) / (a + b + 2):
        return f * betacf(a, b, x) / a
    return 1 - f * betacf(b, a, 1 - x) / b

def bquant(a, b, q):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if ibeta(a, b, mid) < q else (lo, mid)
    return (lo + hi) / 2

rng = random.Random(1948)
print("(1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2")
print("        k   joint rule sigma^-(k+1)   independent 1/sigma")
for k in (10, 100, 10_000):
    S = sum(rng.gauss(0, 1) ** 2 for _ in range(k))     # within-pair SS ~ chi2_k
    for shape in (k, k / 2):                            # inverse-gamma(shape, S/2)
        m = (S / 2) / (shape - 1); sd = m / math.sqrt(shape - 2)
        if shape == k:
            out = f"{m:8.4f} ({sd:.4f})"
        else:
            out += f"          {m:8.4f} ({sd:.4f})"
    print(f"  {k:7,d}   {out}")
x, n, r = 3, 12, 3
mb, mn = (x + 0.5) / (n + 1), x / (n + 0.5)
print(f"(2) {x} successes in {n} trials; likelihoods are proportional to p^3 (1-p)^9")
print(f"   binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean {mb:.4f},"
      f" P(p < 0.1) {ibeta(3.5, 9.5, 0.1):.4f}")
print(f"   neg. binomial p^-1 (1-p)^-1/2   -> Beta(3, 9.5):   mean {mn:.4f},"
      f" P(p < 0.1) {ibeta(3, 9.5, 0.1):.4f}")
print("(3) exact coverage of nominal 95 % intervals, p on a grid of 99 points")
z = statistics.NormalDist().inv_cdf(0.975)
print("     n   Wald: mean  min      Jeffreys: mean  min   points with coverage < 0.93")
for n in (10, 30, 100):
    pmf = lambda p: [math.comb(n, k) * p ** k * (1 - p) ** (n - k) for k in range(n + 1)]
    wald, jef = [], []
    for k in range(n + 1):
        ph = k / n; h = z * math.sqrt(ph * (1 - ph) / n)
        wald.append((ph - h, ph + h))
        jl = 0.0 if k == 0 else bquant(k + 0.5, n - k + 0.5, 0.025)
        ju = 1.0 if k == n else bquant(k + 0.5, n - k + 0.5, 0.975)
        jef.append((jl, ju))
    cw, cj = [], []
    for i in range(1, 100):
        p = i / 100; f = pmf(p)
        cw.append(sum(f[k] for k in range(n + 1) if wald[k][0] <= p <= wald[k][1]))
        cj.append(sum(f[k] for k in range(n + 1) if jef[k][0] <= p <= jef[k][1]))
    print(f"   {n:3d}   {statistics.fmean(cw):.3f}  {min(cw):.3f}       "
          f"{statistics.fmean(cj):.3f}  {min(cj):.3f}   Wald {sum(c < .93 for c in cw)}"
          f"/99, Jeffreys {sum(c < .93 for c in cj)}/99")
# (1) Neyman-Scott, true sigma^2 = 1; posterior mean (sd) of sigma^2
#         k   joint rule sigma^-(k+1)   independent 1/sigma
#        10     0.3435 (0.1214)            0.7729 (0.4462)
#       100     0.5415 (0.0547)            1.0941 (0.1579)
#    10,000     0.4934 (0.0049)            0.9870 (0.0140)
# (2) 3 successes in 12 trials; likelihoods are proportional to p^3 (1-p)^9
#    binomial Jeffreys Beta(1/2,1/2) -> Beta(3.5, 9.5): mean 0.2692, P(p < 0.1) 0.0555
#    neg. binomial p^-1 (1-p)^-1/2   -> Beta(3, 9.5):   mean 0.2400, P(p < 0.1) 0.1000
# (3) exact coverage of nominal 95 % intervals, p on a grid of 99 points
#      n   Wald: mean  min      Jeffreys: mean  min   points with coverage < 0.93
#     10   0.776  0.096       0.952  0.869   Wald 93/99, Jeffreys 22/99
#     30   0.883  0.260       0.950  0.889   Wald 52/99, Jeffreys 8/99
#    100   0.927  0.633       0.950  0.921   Wald 20/99, Jeffreys 2/99

内曼—斯科特问题。 有 kk 组数据,每组两个正态观测,各组均值不同,方差 σ2\sigma^2 相同 (Neyman and Scott 1948)。随着组数增加,关于 σ2\sigma^2 的信息越来越多,一个合理的推断应当收敛到真值 1。联合规则给出的先验是 σ−(k+1)\sigma^{-(k+1)}——参数越多,先验对小 σ\sigma 的偏好越强;程序显示后验均值在 k=10k = 10、100 与 10,000 时为 0.3435、0.5415 与 0.4934,集中在真值的一半附近,后验标准差却越来越小(0.0049),也就是说它越来越确信一个错误的值。独立形式 1/σ1/\sigma 的后验均值为 0.7729、1.0941 与 0.9870,收敛到真值。这与内曼与斯科特关于极大似然估计不一致的原始结论平行。

依赖抽样设计。 观测到 12 次试验中 3 次成功。如果实验者事先定好做 12 次(二项抽样),杰弗里斯先验是 Beta(½, ½),后验 Beta(3.5, 9.5),均值 0.2692,P(p<0.1)=0.0555P(p < 0.1) = 0.0555;如果实验者事先定好做到第 3 次成功为止(负二项抽样),费希尔信息为 r/(p2(1−p))r/(p^2(1-p)),规则给出 p−1(1−p)−1/2p^{-1}(1-p)^{-1/2},后验 Beta(3, 9.5),均值 0.2400,P(p<0.1)=0.1000P(p < 0.1) = 0.1000。两种情形的似然函数成比例(都正比于 p3(1−p)9p^3(1-p)^9),按似然原理,推断应当相同 (Berger and Wolpert 1988);杰弗里斯先验因为依赖费希尔信息(一个对样本空间取期望的量)而违反了它。这是对杰弗里斯规则的标准批评之一 (Kass and Wasserman 1996; Berger 1985)。

注

「违反似然原理」是不是缺点? 这取决于立场。严格的似然主义者与主观贝叶斯者认为是缺点;而频率学派本来就不接受似然原理,停止规则影响 p 值是他们的常规做法。本书只陈述事实:杰弗里斯先验依赖实验设计,所以不能说它只表达了「对参数的无知」。

它在哪里表现得好

公平起见,也要报告杰弗里斯规则的长处。程序的第三部分计算二项比例 95% 区间的精确频率覆盖率:对 99 个真值 pp 逐一把所有可能结果的概率加起来,不用模拟。常用的沃尔德区间 p̂±1.96p̂(1−p̂)/n\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n} 在 n=10n = 10 时平均覆盖率只有 0.776,最差处 0.096,99 个点中有 93 个低于 0.93;n=100n = 100 时平均 0.927,仍有 20 个点低于 0.93。以 Beta(xx+½, n−xn-x+½) 的分位数构成的杰弗里斯区间平均覆盖率在三种样本量下都是 0.95 左右(0.952、0.950、0.950),最差处 0.869、0.889 与 0.921。布朗、蔡与达斯古普塔 2001 年的系统比较在小样本时推荐威尔逊区间或等尾的杰弗里斯区间,在较大样本时推荐阿格雷斯蒂—库尔区间 (Brown et al. 2001)。所以杰弗里斯先验不是「无信息」,却可以有很好的频率性质;两者是不同的问题。

二项比例 95% 区间的精确覆盖率随真值 p 的变化(n = 30):沃尔德区间(金色)在 p 接近 0 与 1 时严重不足,杰弗里斯区间(蓝色)围绕 0.95 波动。覆盖率的锯齿来自二项分布的离散性。

参考先验:系统的推广

杰弗里斯规则的困难在多参数时最明显。伯纳多 1979 年提出参考先验:选择使先验与后验之间的期望 KL 散度(在样本量趋于无穷的极限下)最大的先验,并按参数的重要性排序逐个处理冗余参数 (Bernardo 1979);伯杰、伯纳多与孙 2009 年给出了正式定义 (Berger et al. 2009)。在一维正则模型中,参考先验与杰弗里斯先验一致;在多参数时,它通常给出杰弗里斯本人推荐的「独立」形式,例如正态模型中的 1/σ1/\sigma。卡斯与沃瑟曼 1996 年的综述系统比较了各种「形式规则」,结论是没有一种规则在所有问题中都令人满意,它们应被看作「约定的参照」,而不是「无知的表达」(Kass and Wasserman 1996)。

假说 · 尚未验证 · 先验敏感度报告

作者的假说(尚未确立): 在使用「无信息」先验的应用贝叶斯分析中,若同时报告杰弗里斯先验、参考先验与一个弱信息先验下的后验摘要,至少有相当一部分已发表结论的方向或显著性会随先验改变。什么样的结果能证明它是错的: 在事先登记的一组已发表分析中重做三种先验,若后验 95% 区间是否包含零的判断在三种先验下一致的比例超过 95%,则假说不成立。本书不做这一检验。

本章小结

杰弗里斯先验 det⁡G\sqrt{\det G} 是费希尔—拉奥体积元,因而在任何坐标下应用都得到同一个分布(程序中两种坐标的区间概率都是 0.164177)。但它不是「无信息、总是合适」的:正态均值与泊松率的规则先验不可积;多参数的联合形式在内曼—斯科特问题中使后验以越来越小的标准差集中在真值的一半(0.4934),杰弗里斯本人建议的独立形式没有这个问题(0.9870);它依赖抽样设计,二项与负二项给出不同的后验,违反似然原理。另一方面,它的区间在二项比例上有接近名义水平的频率覆盖。参考先验是更系统的推广。

参考文献

Berger, James O. 1985. Statistical Decision Theory and Bayesian Analysis. 2nd ed. Springer.
Berger, James O., José M. Bernardo, and Dongchu Sun. 2009. “The Formal Definition of Reference Priors.” Annals of Statistics 37 (2): 905–38.
Berger, James O., and Robert L. Wolpert. 1988. The Likelihood Principle. 2nd ed. Vol. 6. IMS Lecture Notes – Monograph Series. Institute of Mathematical Statistics.
Bernardo, José M. 1979. “Reference Posterior Distributions for Bayesian Inference.” Journal of the Royal Statistical Society. Series B 41 (2): 113–47.
Brown, Lawrence D., T. Tony Cai, and Anirban DasGupta. 2001. “Interval Estimation for a Binomial Proportion.” Statistical Science 16 (2): 101–33.
Jeffreys, Harold. 1946. “An Invariant Form for the Prior Probability in Estimation Problems.” Proceedings of the Royal Society of London. Series A 186 (1007): 453–61. https://doi.org/10.1098/rspa.1946.0056.
Jeffreys, Harold. 1961. Theory of Probability. 3rd ed. Oxford University Press.
Kass, Robert E., and Larry Wasserman. 1996. “The Selection of Prior Distributions by Formal Rules.” Journal of the American Statistical Association 91 (435): 1343–70.
Neyman, J., and Elizabeth L. Scott. 1948. “Consistent Estimates Based on Partially Consistent Observations.” Econometrica 16 (1): 1–32.

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).