5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文

Chapter 8 Practice and a Pre-Registered Test

Note

The checklist and trial protocol in this chapter are for research and self-examination, not engineering advice. The checklist cannot tell you which optimiser or which prior to use; it can only help you write down your assumptions before drawing a conclusion. The tasks, losses and timings in the book are synthetic or simulated; performance in any real system may differ.

Part 1: a four-step checklist

Chapters 1 to 6 can be condensed into a four-step checklist for examining a claim involving Fisher information, priors or optimisers — for example “the new method converges three times faster”, “this prior is non-informative” or “this estimate has reached the theoretical limit”. The questions are ones the user should answer in writing in advance; they are not recommended practices.

  1. Coordinates. Does the conclusion depend on the parametrisation? With an equivalent set of parameters (say log⁡σ\log\sigma instead of σ\sigma, or normalised weights), do step sizes, priors, confidence intervals or rankings change? If so, relative to which coordinates is the claimed “better” (Chapters 1, 3 and 4)?
  2. Curvature. Which matrix is used: Fisher, empirical Fisher, Gauss–Newton, Hessian, or some approximation (diagonal, K-FAC)? Is it invertible in the region under discussion? Does the scale of the empirical Fisher depend on the residuals and the assumed noise (Chapters 4 and 6)?
  3. Cost. Is the comparison by steps, gradient evaluations or wall-clock time? Do the methods have the same tuning budget? Was the target fixed in advance or chosen afterwards (Chapter 4)?
  4. Regularity and asymptotics. Are the conditions behind any quoted bound, BIC, Laplace approximation or asymptotic normality satisfied: does the support depend on the parameter? Is the model identifiable, with a positive-definite Fisher matrix? Is the sample size in the asymptotic regime (Chapters 2 and 6)?

An example. Suppose a report says: “On our logistic regression task the natural gradient converges in only 3 steps while gradient descent needs 600, so the natural gradient is 200 times faster.” Applying the checklist: step 1, the natural gradient’s step count does not depend on the coordinates (in the small-step sense), but gradient descent’s depends on the scaling of the features, and the report should say whether the features were standardised; step 2, in logistic regression the Fisher equals the Hessian, so this “natural gradient” is Newton’s method and the conclusion does not carry over to models where Fisher and Hessian differ; step 3, by the accounting of Chapter 4, a natural-gradient step costs about 4 units at d=3d = 3, so “200 times” becomes about 50 times in cost, and at κ=1\kappa = 1 gradient descent is in fact cheaper; step 4, there is no asymptotic issue here, but if the report went on to say “and so the Cramér–Rao bound is attained”, one must ask which parameter is being estimated and with what sample size. After the four steps, a more accurate statement is: “On this low-dimensional, ill-conditioned problem with canonical link, Newton’s method is faster than gradient descent by more than an order of magnitude in cost; this does not show that the natural gradient is faster on general problems.”

Part 2: a pre-registered computational benchmark

Question and design choices

On a stratified suite of synthetic optimisation tasks generated and frozen by an independent third party, is the wall-clock time for natural-gradient methods (the exact damped Fisher matrix for low-dimensional generalised linear models, K-FAC for small neural networks) to reach a pre-specified target loss shorter than that for Adam? The trial is designed for 90% power at the smallest effect of interest (a time ratio of 1.25, i.e. log⁡21.25\log_2 1.25); the decision rule separately distinguishes “faster” from “faster by at least the smallest effect of interest”.

The author’s framework suggests such a benchmark comparison. The book keeps the comparison groups and makes three design choices. First, wall-clock time rather than steps: Chapter 4 showed that the natural gradient takes the fewest steps without always costing the least; comparing steps would systematically favour methods whose steps are expensive. Gradient-evaluation counts are a secondary outcome. Second, a single primary outcome: the base-2 logarithm of the ratio of the two methods’ times on each task, averaged over the population of tasks. Third, execution by an independent third party: the task generator, random seeds, budgets, reference hardware and software image are prepared and sealed by researchers unconnected with the author, who submits only the implementations of the two methods and their tuning search spaces.

The trial involves no human participants, so “not recruiting from the author’s community, students or customers” takes the form that the tasks, the hardware and the analysts are all unconnected with the author’s business.

Protocol

Element Specification
Task suite (stratified) Drawn by the third party from a registered generator in 5 strata of equal size: (1) logistic regression, d=10d = 10; (2) logistic regression, d=100d = 100; (3) logistic regression, d=1000d = 1000; (4) 10-class softmax regression, d=100d = 100; (5) small multilayer perceptrons with two hidden layers (at most 50,000 parameters, synthetic data from a teacher network). In the generalised-linear-model strata the condition number of the feature covariance is log-uniform on [1,104][1, 10^4], half randomly rotated. Each task has its own validation set. The generator code is public; seeds are sealed until the results are locked
Target loss and budget Fixed for each task by the third party before the trial, using a reference optimiser unconnected with either arm (a long L-BFGS run): the reference final training loss plus 1% of the initial gap. The budget BB is 20 times the wall-clock time the reference optimiser takes to reach the target on the same reference machine. Targets and budgets are sealed with the tasks
The two methods X: natural gradient (exact Fisher with damping for generalised linear models, K-FAC for networks (Martens and Grosse 2015)); Y: Adam (Kingma and Ba 2015). Batches: full batch for generalised linear models and a fixed batch size of 256 for multilayer perceptrons, in both arms. Both are tuned by the same protocol: on 50 separate tuning tasks (drawn from the same 5 strata, not used for evaluation), each draws 20 hyperparameter settings from its registered search space and keeps the setting with the shortest median time on the tuning tasks, which is then frozen
Hardware and timing An exclusive reference machine with fixed processor frequency (turbo off, fixed frequency governor), BLAS/OpenMP threads set to 1 in both arms, and the registered container image; timing starts at the first optimisation step, after one untimed warm-up step (including compilation and memory allocation) that is recorded separately and not counted; the order of the two arms is randomised and alternated, and the 3 repeats are also alternated, taking the median. In a pre-trial run the third party reports the coefficient of variation of timings across repeats
Pairing and blinding On each task both methods use the same data and the same initial parameters. Their run-time behaviour (memory, computation of curvature factors) makes it impossible to blind those running the trial, so outcomes are recorded automatically by the program, with no human judgement; the two implementations are registered by hash and randomly labelled “A / B” by the third party, and the analysts are blinded until the results are locked
Invalid tasks and failures Only infrastructure faults (hardware, container, timer) make a task invalid. Divergence, numerical overflow or NaN caused by a method itself, and failure to reach the target within the budget BB, are all recorded as 2B2B and counted in the primary outcome
Primary outcome D=log⁡2(TX/TY)D = \log_2(T_X / T_Y), where TT is the wall-clock time to reach the target loss on the reference machine (median of 3 repeats, failures recorded as 2B2B). The estimate is the mean of DD over valid tasks (with equal strata, the equally weighted mean of the stratum means), with a 95% confidence interval from the tt distribution
Pre-specified qualifier If the share of tasks on which each method misses the target differs by 10 percentage points or more, the conclusion is followed by a note on which method failed more often
Secondary outcomes Pre-specified stratum estimates: the mean of DD with its interval in each of the 5 strata; ratio of gradient-evaluation counts; failure rates; final loss on the validation set; peak memory; DD stratified by condition number (the hypothesis of Chapter 4); the median of DD; all descriptive or exploratory

There is only one primary outcome. The logarithm of the time ratio is chosen over the time difference because times on different tasks differ by orders of magnitude, ratios are more comparable, and the logarithm makes “twice as fast” and “twice as slow” symmetric; the mean rather than the median is chosen so that the standard sample-size formula applies, with the median as a robustness check. Recording a failure as 2B2B is a pre-specified penalty that counts failures in the primary outcome instead of deleting them; because BB is sealed with the tasks, the size of the penalty is fixed in advance. Only infrastructure faults count as invalid, so that the 15% limit on invalid tasks cannot absorb one method’s own failures (in Chapter 4 the exact natural gradient diverged at step size 1, and so did K-FAC).

Guarding against data snooping

There is only one pre-frozen set of hyperparameters and one primary outcome, and the author does not tune on the evaluation tasks; the task generator, the strata and their sizes, the procedures for fixing targets and budgets, the two implementations, the tuning protocol, the timing procedure and the analysis code are publicly registered before any run, with hashes of the code registered alongside the protocol (Nosek et al. 2018); all runs and analyses are carried out by the third party; secondary and exploratory results are clearly labelled and cannot replace the main conclusion. The AlgoPerf benchmark of Dahl and co-authors takes a similar approach to tuning protocols and “time to target” (Dahl et al. 2023). Book 33 discussed the statistical rationale for these practices.

Sample size

The sample size requires an assumption about the standard deviation of DD across tasks. The book sets it with a pilot environment: a scaled-down version of the logistic regression of Chapter 4 (n=40n = 40, d=4d = 4, 40 tasks, condition number log-uniform on [1,104][1, 10^4] with random rotation), with “time” converted to gradient-evaluation units by the rule of Chapter 4. This is the author’s toy model, not the task suite the trial will use.

Calculation

# Sample size for the Chapter 8 pre-registered benchmark. Unit = a synthetic task drawn
# by the third party; both optimisers run on it (paired). Primary outcome per task:
# D = log2(time to target, natural gradient / time to target, Adam). Estimand: mean D
# over the task population (log2 of the geometric-mean time ratio). SESOI log2(1.25).
# PILOT (planning assumption only, not the trial suite): 40 logistic-regression tasks,
# n = 40, d = 4, kappa log-uniform on [1, 10^4], random rotation; "time" = gradient-
# evaluation units (Newton/NG step = 1 + d + d^2/(3n)); Adam: best of 3 steps, cap 500.
# Last part: the registered blinded re-estimation rule (sd of D from the first 25 % of
# tasks; sd is unchanged by swapping the arm labels; n may only go up, cap 2,000).
import math, random, statistics

NZ = statistics.NormalDist(); Z = NZ.inv_cdf

def task(rng, n=40, d=4):
    kappa = 10 ** rng.uniform(0, 4); ev = [kappa ** (-i / (d - 1)) for i in range(d)]
    Q = []
    while len(Q) < d:
        v = [rng.gauss(0, 1) for _ in range(d)]
        for q in Q:
            c = sum(a * b for a, b in zip(v, q)); v = [a - c * b for a, b in zip(v, q)]
        s = math.sqrt(sum(a * a for a in v)); Q.append([a / s for a in v])
    X = []
    for _ in range(n):
        z = [rng.gauss(0, 1) * math.sqrt(e) for e in ev]
        X.append([sum(Q[i][j] * z[j] for j in range(d)) for i in range(d)])
    w = [sum(Q[i][j] / math.sqrt(ev[j]) for j in range(d)) / d for i in range(d)]
    y = [int(rng.random() < 1 / (1 + math.exp(-sum(a * b for a, b in zip(w, x)))))
         for x in X]
    return X, y

def lgf(X, y, w, hess=False):
    d, n = len(w), len(X); L, g = 0.0, [0.0] * d; H = [[0.0] * d for _ in range(d)]
    for x, t in zip(X, y):
        z = sum(a * b for a, b in zip(w, x))
        p = 1 / (1 + math.exp(-z)) if z >= 0 else math.exp(z) / (1 + math.exp(z))
        L += math.log1p(math.exp(-abs(z))) + max(z, 0) - t * z
        for i in range(d):
            g[i] += (p - t) * x[i] / n
            if hess:
                for j in range(d):
                    H[i][j] += p * (1 - p) * x[i] * x[j] / n
    return L / n, g, H

def newton(X, y, w):
    L, g, H = lgf(X, y, w, True); d = len(w)
    M = [H[i][:] + [g[i]] for i in range(d)]
    for i in range(d):
        M[i][i] += 1e-8
    for c in range(d):
        for r in range(c + 1, d):
            f = M[r][c] / M[c][c]; M[r] = [a - f * b for a, b in zip(M[r], M[c])]
    s = [0.0] * d
    for r in range(d - 1, -1, -1):
        s[r] = (M[r][d] - sum(M[r][k] * s[k] for k in range(r + 1, d))) / M[r][r]
    return [a - b for a, b in zip(w, s)], L

def pilot_D(seed):
    rng = random.Random(seed); X, y = task(rng); d, n = len(X[0]), len(X)
    w = [0.0] * d
    for _ in range(30):
        w, _ = newton(X, y, w)
    target = lgf(X, y, w)[0] + 1e-3; w = [0.0] * d; k_ng = 0
    while lgf(X, y, w)[0] > target:
        w, _ = newton(X, y, w); k_ng += 1
    best = None
    for lr in (0.03, 0.1, 0.3):
        w = [0.0] * d; m = [0.0] * d; v = [0.0] * d
        for k in range(1, 501):
            L, g, _ = lgf(X, y, w)
            if L <= target:
                best = k - 1 if best is None else min(best, k - 1); break
            m = [0.9 * a + 0.1 * b for a, b in zip(m, g)]
            v = [0.999 * a + 0.001 * b * b for a, b in zip(v, g)]
            w = [a - lr * (b / (1 - 0.9 ** k)) / (math.sqrt(c / (1 - 0.999 ** k)) + 1e-8)
                 for a, b, c in zip(w, m, v)]
    t_ng = k_ng * (1 + d + d * d / (3 * n)); t_ad = best if best is not None else 1000
    return math.log2(max(t_ng, 1) / max(t_ad, 1)), best is None

NP = 40
Ds = [pilot_D(s) for s in range(NP)]
D = [a for a, _ in Ds]; sd = statistics.stdev(D)
print(f"pilot ({NP} tasks): mean D {statistics.fmean(D):+.2f} (NG faster if < 0),"
      f" sd {sd:.2f},"
      f"\n   range {min(D):+.2f} to {max(D):+.2f}; Adam missed the target on"
      f" {sum(f for _, f in Ds)} task(s) (time set to 2 x cap)")
delta = math.log2(1.25); za, zb = Z(0.975), Z(0.90)
n0 = ((za + zb) * sd / delta) ** 2 + za * za / 2
n_t = math.ceil(n0)
print(f"SESOI log2(1.25) = {delta:.3f}; 90 % power, two-sided alpha 0.05:"
      f" n = {n0:.1f} -> {n_t} tasks")
rng = random.Random(15); hits = 0; R = 2000
cen = [a - statistics.fmean(D) - delta for a in D]       # true mean D = -SESOI
for _ in range(R):
    s = [rng.choice(cen) for _ in range(n_t)]
    m, se = statistics.fmean(s), statistics.stdev(s) / math.sqrt(n_t)
    hits += m + 1.96 * se < 0                              # "NG faster" shown
print(f"   simulation ({R:,} resampled suites of {n_t}, true mean D = -SESOI):"
      f" power {hits / R:.3f}")
plan = math.ceil(n_t / 0.85 / 10) * 10
print(f"   allow 15 % lost to infrastructure faults: generate {plan}"
      f" ({plan // 5} in each of 5 strata)")
print("sensitivity: tasks needed (before the 15 % allowance)")
SDS = (0.5 * sd, sd, 1.5 * sd, 2 * sd)
print("   sd of D:    " + "  ".join(f"{s:5.2f}" for s in SDS))
need = lambda s: math.ceil(((za + zb) * s / delta) ** 2 + za * za / 2)
print("   n:          " + "  ".join(f"{need(s):5d}" for s in SDS))

def reestimate(sd_blind, n_plan=plan, cap=2000):     # registered rule, never decreases
    n_new = math.ceil(need(sd_blind) / 0.85 / 10) * 10
    return min(cap, max(n_plan, n_new))
print("blinded re-estimation after 25 % of tasks: sd of D -> tasks to generate")
SD2 = (1.5, 2.0, 2.5, 3.0, 4.0)
print("   " + "  ".join(f"{s:.1f} -> {reestimate(s):,}" for s in SD2))
# pilot (40 tasks): mean D -2.78 (NG faster if < 0), sd 1.96,
#    range -6.61 to +0.36; Adam missed the target on 4 task(s) (time set to 2 x cap)
# SESOI log2(1.25) = 0.322; 90 % power, two-sided alpha 0.05: n = 392.5 -> 393 tasks
#    simulation (2,000 resampled suites of 393, true mean D = -SESOI): power 0.906
#    allow 15 % lost to infrastructure faults: generate 470 (94 in each of 5 strata)
# sensitivity: tasks needed (before the 15 % allowance)
#    sd of D:     0.98   1.96   2.94   3.93
#    n:            100    393    881   1565
# blinded re-estimation after 25 % of tasks: sd of D -> tasks to generate
#    1.5 -> 470  2.0 -> 480  2.5 -> 750  3.0 -> 1,080  4.0 -> 1,920

In the pilot the mean of DD is −2.78 (the natural gradient is much cheaper on these low-dimensional ill-conditioned tasks), with standard deviation 1.96 and range −6.61 to +0.36; Adam missed the target within 500 steps on 4 tasks. These numbers are planning assumptions, not evidence: the pilot has a single task family of very low dimension, the natural gradient’s cost per step is tiny, and the “natural gradient” here is Newton’s method. The real suite contains d=1000d = 1000 and neural networks, the distribution of DD is likely to be multimodal (the natural gradient far ahead on generalised linear models, possibly behind on multilayer perceptrons), and its standard deviation may be much larger than 1.96; the mean may also be close to zero or even positive. The sample size depends only on the standard deviation, not on the pilot’s mean.

The smallest effect of interest is a time ratio of 1.25, i.e. log⁡21.25=0.322\log_2 1.25 = 0.322: an improvement of less than a quarter is not worth the extra complexity of implementing and maintaining a curvature method. With two-sided α=0.05\alpha = 0.05 and 90% power, the normal approximation (with a small-sample correction for the tt distribution) requires 393 paired tasks; in 2,000 simulations resampling the pilot’s DD values shifted to a true mean of exactly −0.322, the power is 0.906. Allowing for 15% of tasks to be lost to infrastructure faults, 470 tasks are to be generated, 94 in each of the 5 strata. The sensitivity table shows that 100 tasks are needed if the standard deviation is half the pilot’s, 881 if it is 1.5 times, and 1,565 if it is twice.

Because the pilot’s standard deviation is likely to be too small, the protocol registers a blinded re-estimation rule: once the first 25% of tasks (equally across strata) are complete, a statistician who does not know the “A / B” assignment computes only the standard deviation of DD (swapping the arm labels changes the sign of DD but not its standard deviation) and does not report its mean; the required number of tasks is recomputed by the same formula with the 15% allowance, the plan may only increase, never decrease, with a cap of 2,000, and the additional tasks are added equally across strata. The last line of the program lists the results of the rule: 480 tasks for a standard deviation of 2.0, 750 for 2.5, 1,080 for 3.0 and 1,920 for 4.0. Computational tasks are cheap, so the sample size itself is not the bottleneck; the bottleneck is freezing the tasks, targets, budgets and tuning protocol in advance.

Following Chapter 6 of Book 33, “90% power” means that if the true effect is exactly the smallest effect of interest, the probability of concluding “the natural gradient is faster” (upper confidence limit below 0) is about 90%; in that case the strongest conclusion, “faster by at least the smallest effect of interest”, has only about a 2.5% chance of appearing.

Decision rule

Calculation

# Decision rule, fixed before the third party runs the frozen benchmark. Primary outcome
# per task: D = log2(T_NG / T_Adam), T = wall-clock time to the task's target loss on the
# frozen reference machine (median of 3 repeats; a run that misses the target within the
# registered budget B, diverges or returns NaN is given 2B). Estimate: mean D over valid
# tasks, 95 % CI (t interval). SESOI = log2(1.25). Only infrastructure faults (hardware,
# container, timer) make a task invalid. Gates, checked first: more than 15 % of tasks
# lost to infrastructure faults; machine differing from the registered image; arm labels
# revealed to the analysts before the lock -> NOT INTERPRETABLE. Pre-specified
# qualifier: failure rates differing by >= 10 points.
import math

SESOI = math.log2(1.25)

def verdict(d, lo, hi, fail_ng, fail_adam, invalid=0.02, image_ok=True,
            unblinded=False):
    for bad, why in ((invalid > 0.15, f"{invalid:.0%} of tasks lost to infrastructure"
                                      " faults (limit 15 %)"),
                     (not image_ok, "machine differs from the registered image"),
                     (unblinded, "labels revealed to analysts before lock")):
        if bad:
            return f"D = {d:+.2f}: NOT INTERPRETABLE - {why}"
    if hi <= -SESOI:  v = "NG faster by at least the SESOI (x1.25)"
    elif hi < 0:      v = "NG faster; SESOI not shown"
    elif lo >= SESOI: v = "Adam faster by at least the SESOI (x1.25)"
    elif lo > 0:      v = "Adam faster; SESOI not shown"
    elif lo >= -SESOI and hi <= SESOI: v = "no difference worth acting on"
    else:             v = "unclear: interval too wide to decide"
    if abs(fail_ng - fail_adam) >= 0.10:
        worse = "NG" if fail_ng > fail_adam else "Adam"
        hi_f, lo_f = max(fail_ng, fail_adam), min(fail_ng, fail_adam)
        v += (f"\n    qualifier: {worse} missed the target on {hi_f:.0%} of tasks,"
              f" the other arm on {lo_f:.0%}")
    ratio = 2 ** d
    return f"D = {d:+.2f} ({lo:+.2f} to {hi:+.2f}), ratio {ratio:.2f}: {v}"

for case in ((-0.80, -1.00, -0.60, 0.05, 0.08), (-0.20, -0.38, -0.02, 0.04, 0.05),
             (0.05, -0.15, 0.25, 0.03, 0.03), (0.60, 0.40, 0.80, 0.02, 0.04),
             (-0.10, -0.55, 0.35, 0.05, 0.06), (-0.90, -1.10, -0.70, 0.22, 0.04),
             (-0.80, -1.00, -0.60, 0.05, 0.08, 0.19),
             (0.10, -0.10, 0.30, 0, 0, 0, False)):
    print(verdict(*case))
print("secondary outcomes (gradient-evaluation counts, failure rates, final test loss,")
print("  peak memory, results by task family and by condition number) are reported with")
print("  intervals; they never decide the verdict")
# D = -0.80 (-1.00 to -0.60), ratio 0.57: NG faster by at least the SESOI (x1.25)
# D = -0.20 (-0.38 to -0.02), ratio 0.87: NG faster; SESOI not shown
# D = +0.05 (-0.15 to +0.25), ratio 1.04: no difference worth acting on
# D = +0.60 (+0.40 to +0.80), ratio 1.52: Adam faster by at least the SESOI (x1.25)
# D = -0.10 (-0.55 to +0.35), ratio 0.93: unclear: interval too wide to decide
# D = -0.90 (-1.10 to -0.70), ratio 0.54: NG faster by at least the SESOI (x1.25)
#     qualifier: NG missed the target on 22% of tasks, the other arm on 4%
# D = -0.80: NOT INTERPRETABLE - 19% of tasks lost to infrastructure faults (limit 15 %)
# D = +0.10: NOT INTERPRETABLE - machine differs from the registered image
# secondary outcomes (gradient-evaluation counts, failure rates, final test loss,
#   peak memory, results by task family and by condition number) are reported with
#   intervals; they never decide the verdict

The example results are invented. If any precondition fails (more than 15% of tasks lost to infrastructure faults, a run environment that differs from the registered image, labels revealed to the analysts before the results are locked), the report says “not interpretable” whatever the primary outcome. Only when the upper confidence limit is at most −0.322 does the report say “the natural gradient is faster by at least the smallest effect of interest”; an upper limit below 0 but above −0.322 gives “the natural gradient is faster, but the size is not established”; a lower limit of at least 0.322 gives “Adam is faster by at least the smallest effect of interest”; a lower limit above 0 but below 0.322 gives “Adam is faster, but the size is not established”; an interval within ±0.322 gives “no difference worth acting on”; anything else gives “unclear”. The qualifier does not change the category of the conclusion but is appended to it: in the sixth example, for instance, the natural gradient missed the target on 22% of tasks and Adam on only 4%, and the report must say so.

What this trial cannot show

First, it does not extrapolate to large-scale training: the suite contains only low-dimensional generalised linear models and small networks of at most 50,000 parameters, and its conclusions do not apply to models with billions of parameters or to distributed training. Second, it times runs on a single CPU thread: in that setting the cost of building and inverting a curvature matrix relative to one gradient differs from that in common GPU training (where matrix operations are highly parallel and memory bandwidth is the bottleneck), so the time ratios cannot be carried over to GPUs. Third, it compares two particular implementations, one tuning protocol and the stated batch sizes; better implementations, different damping, learning-rate schedules or batch sizes (Zhang and co-authors showed that the advantage of preconditioned methods changes with batch size (Zhang et al. 2019)) could change the result. Fourth, it looks only at time to a training target, not at generalisation; validation loss is reported as a secondary outcome, but the trial is not designed for it (Wilson and co-authors reported adaptive methods generalising worse on some tasks (Wilson et al. 2017)). Fifth, synthetic tasks are not real data; condition numbers, noise and model specification are all set by the generator, and the value of the primary outcome also depends on the equal mixture of the 5 strata. Sixth, it does not test the invariance analysis of Chapter 4, the empirical Fisher or the singular learning theory of Chapter 6, which need tests of their own. Seventh, the natural gradient’s large advantage in the d=3d = 3 and d=4d = 4 pilots comes from the author’s own model, which is exactly why an independent task suite and pre-registration are needed; if the real trial finds “no difference worth acting on” or “Adam is faster”, that is the trial’s conclusion.

Ethics and conflicts of interest

The trial involves no human participants and needs no ethics-committee approval; but the protocol, the task generator, the two implementations and the analysis code must be publicly registered before any run. Whether the results are positive or negative, the results, the code and summary data for all tasks will be made public. To limit the influence of interests: the protocol and code are publicly registered before the third party runs them, the author has no access to the seeds, tasks or target values, and the results will be published whatever they are.

Disclosure

The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check. The businesses the author runs use machine-learning and AI tools in content production and in the knowledge base, but there is no direct sales relationship between the methods discussed in this book and their commercial products.

Summary

The four-step checklist examines a claim about Fisher information, priors or optimisers from four sides: coordinates, curvature, cost and regularity; the example shows that “200 times faster”, once converted to cost and once one sees that the “natural gradient” in logistic regression is Newton’s method, leaves only a conditional conclusion. The pre-registered computational benchmark compares the wall-clock time for the natural gradient and Adam to reach a target loss on 5 strata of synthetic tasks frozen by an independent third party, with 470 paired tasks (393 valid tasks for 90% power, adjusted upwards only by a blinded re-estimate of the standard deviation); the single primary outcome is the mean log time ratio, the smallest effect of interest log⁡21.25\log_2 1.25, the decision rule is written as code, and a qualifier on failure rates is pre-specified; the results will be published whatever they are. It tests only two implementations on one class of small synthetic tasks and cannot be extended to large-scale training.

References

Dahl, George E., Frank Schneider, Zachary Nado, Naman Agarwal, et al. 2023. Benchmarking Neural Network Training Algorithms. arXiv:2306.07179.
Kingma, Diederik P., and Jimmy Ba. 2015. “Adam: A Method for Stochastic Optimization.” 3rd International Conference on Learning Representations (ICLR).
Martens, James, and Roger Grosse. 2015. “Optimizing Neural Networks with Kronecker-Factored Approximate Curvature.” Proceedings of the 32nd International Conference on Machine Learning, Proceedings of machine learning research, vol. 37: 2408–17.
Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115 (11): 2600–2606. https://doi.org/10.1073/pnas.1708274114.
Wilson, Ashia C., Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. 2017. “The Marginal Value of Adaptive Gradient Methods in Machine Learning.” Advances in Neural Information Processing Systems 30.
Zhang, Guodong, Lala Li, Zachary Nado, et al. 2019. “Which Algorithmic Choices Matter at Which Batch Sizes? Insights from a Noisy Quadratic Model.” Advances in Neural Information Processing Systems 32.

Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).