5D Library › Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds · 中文
Chapter 8 Practice and a Pre-Registered Test
Note
The checklist and trial protocol in this chapter are for research and self-examination, not engineering advice. The checklist cannot tell you which optimiser or which prior to use; it can only help you write down your assumptions before drawing a conclusion. The tasks, losses and timings in the book are synthetic or simulated; performance in any real system may differ.
Part 1: a four-step checklist
Chapters 1 to 6 can be condensed into a four-step checklist for examining a claim involving Fisher information, priors or optimisers — for example “the new method converges three times faster”, “this prior is non-informative” or “this estimate has reached the theoretical limit”. The questions are ones the user should answer in writing in advance; they are not recommended practices.
- Coordinates. Does the conclusion depend on the parametrisation? With an equivalent set of parameters (say instead of , or normalised weights), do step sizes, priors, confidence intervals or rankings change? If so, relative to which coordinates is the claimed “better” (Chapters 1, 3 and 4)?
- Curvature. Which matrix is used: Fisher, empirical Fisher, Gauss–Newton, Hessian, or some approximation (diagonal, K-FAC)? Is it invertible in the region under discussion? Does the scale of the empirical Fisher depend on the residuals and the assumed noise (Chapters 4 and 6)?
- Cost. Is the comparison by steps, gradient evaluations or wall-clock time? Do the methods have the same tuning budget? Was the target fixed in advance or chosen afterwards (Chapter 4)?
- Regularity and asymptotics. Are the conditions behind any quoted bound, BIC, Laplace approximation or asymptotic normality satisfied: does the support depend on the parameter? Is the model identifiable, with a positive-definite Fisher matrix? Is the sample size in the asymptotic regime (Chapters 2 and 6)?
An example. Suppose a report says: “On our logistic regression task the natural gradient converges in only 3 steps while gradient descent needs 600, so the natural gradient is 200 times faster.” Applying the checklist: step 1, the natural gradient’s step count does not depend on the coordinates (in the small-step sense), but gradient descent’s depends on the scaling of the features, and the report should say whether the features were standardised; step 2, in logistic regression the Fisher equals the Hessian, so this “natural gradient” is Newton’s method and the conclusion does not carry over to models where Fisher and Hessian differ; step 3, by the accounting of Chapter 4, a natural-gradient step costs about 4 units at , so “200 times” becomes about 50 times in cost, and at gradient descent is in fact cheaper; step 4, there is no asymptotic issue here, but if the report went on to say “and so the Cramér–Rao bound is attained”, one must ask which parameter is being estimated and with what sample size. After the four steps, a more accurate statement is: “On this low-dimensional, ill-conditioned problem with canonical link, Newton’s method is faster than gradient descent by more than an order of magnitude in cost; this does not show that the natural gradient is faster on general problems.”
Part 2: a pre-registered computational benchmark
Question and design choices
On a stratified suite of synthetic optimisation tasks generated and frozen by an independent third party, is the wall-clock time for natural-gradient methods (the exact damped Fisher matrix for low-dimensional generalised linear models, K-FAC for small neural networks) to reach a pre-specified target loss shorter than that for Adam? The trial is designed for 90% power at the smallest effect of interest (a time ratio of 1.25, i.e. ); the decision rule separately distinguishes “faster” from “faster by at least the smallest effect of interest”.
The author’s framework suggests such a benchmark comparison. The book keeps the comparison groups and makes three design choices. First, wall-clock time rather than steps: Chapter 4 showed that the natural gradient takes the fewest steps without always costing the least; comparing steps would systematically favour methods whose steps are expensive. Gradient-evaluation counts are a secondary outcome. Second, a single primary outcome: the base-2 logarithm of the ratio of the two methods’ times on each task, averaged over the population of tasks. Third, execution by an independent third party: the task generator, random seeds, budgets, reference hardware and software image are prepared and sealed by researchers unconnected with the author, who submits only the implementations of the two methods and their tuning search spaces.
The trial involves no human participants, so “not recruiting from the author’s community, students or customers” takes the form that the tasks, the hardware and the analysts are all unconnected with the author’s business.
Protocol
| Element | Specification |
|---|---|
| Task suite (stratified) | Drawn by the third party from a registered generator in 5 strata of equal size: (1) logistic regression, ; (2) logistic regression, ; (3) logistic regression, ; (4) 10-class softmax regression, ; (5) small multilayer perceptrons with two hidden layers (at most 50,000 parameters, synthetic data from a teacher network). In the generalised-linear-model strata the condition number of the feature covariance is log-uniform on , half randomly rotated. Each task has its own validation set. The generator code is public; seeds are sealed until the results are locked |
| Target loss and budget | Fixed for each task by the third party before the trial, using a reference optimiser unconnected with either arm (a long L-BFGS run): the reference final training loss plus 1% of the initial gap. The budget is 20 times the wall-clock time the reference optimiser takes to reach the target on the same reference machine. Targets and budgets are sealed with the tasks |
| The two methods | X: natural gradient (exact Fisher with damping for generalised linear models, K-FAC for networks (Martens and Grosse 2015)); Y: Adam (Kingma and Ba 2015). Batches: full batch for generalised linear models and a fixed batch size of 256 for multilayer perceptrons, in both arms. Both are tuned by the same protocol: on 50 separate tuning tasks (drawn from the same 5 strata, not used for evaluation), each draws 20 hyperparameter settings from its registered search space and keeps the setting with the shortest median time on the tuning tasks, which is then frozen |
| Hardware and timing | An exclusive reference machine with fixed processor frequency (turbo off, fixed frequency governor), BLAS/OpenMP threads set to 1 in both arms, and the registered container image; timing starts at the first optimisation step, after one untimed warm-up step (including compilation and memory allocation) that is recorded separately and not counted; the order of the two arms is randomised and alternated, and the 3 repeats are also alternated, taking the median. In a pre-trial run the third party reports the coefficient of variation of timings across repeats |
| Pairing and blinding | On each task both methods use the same data and the same initial parameters. Their run-time behaviour (memory, computation of curvature factors) makes it impossible to blind those running the trial, so outcomes are recorded automatically by the program, with no human judgement; the two implementations are registered by hash and randomly labelled “A / B” by the third party, and the analysts are blinded until the results are locked |
| Invalid tasks and failures | Only infrastructure faults (hardware, container, timer) make a task invalid. Divergence, numerical overflow or NaN caused by a method itself, and failure to reach the target within the budget , are all recorded as and counted in the primary outcome |
| Primary outcome | , where is the wall-clock time to reach the target loss on the reference machine (median of 3 repeats, failures recorded as ). The estimate is the mean of over valid tasks (with equal strata, the equally weighted mean of the stratum means), with a 95% confidence interval from the distribution |
| Pre-specified qualifier | If the share of tasks on which each method misses the target differs by 10 percentage points or more, the conclusion is followed by a note on which method failed more often |
| Secondary outcomes | Pre-specified stratum estimates: the mean of with its interval in each of the 5 strata; ratio of gradient-evaluation counts; failure rates; final loss on the validation set; peak memory; stratified by condition number (the hypothesis of Chapter 4); the median of ; all descriptive or exploratory |
There is only one primary outcome. The logarithm of the time ratio is chosen over the time difference because times on different tasks differ by orders of magnitude, ratios are more comparable, and the logarithm makes “twice as fast” and “twice as slow” symmetric; the mean rather than the median is chosen so that the standard sample-size formula applies, with the median as a robustness check. Recording a failure as is a pre-specified penalty that counts failures in the primary outcome instead of deleting them; because is sealed with the tasks, the size of the penalty is fixed in advance. Only infrastructure faults count as invalid, so that the 15% limit on invalid tasks cannot absorb one method’s own failures (in Chapter 4 the exact natural gradient diverged at step size 1, and so did K-FAC).
Guarding against data snooping
There is only one pre-frozen set of hyperparameters and one primary outcome, and the author does not tune on the evaluation tasks; the task generator, the strata and their sizes, the procedures for fixing targets and budgets, the two implementations, the tuning protocol, the timing procedure and the analysis code are publicly registered before any run, with hashes of the code registered alongside the protocol (Nosek et al. 2018); all runs and analyses are carried out by the third party; secondary and exploratory results are clearly labelled and cannot replace the main conclusion. The AlgoPerf benchmark of Dahl and co-authors takes a similar approach to tuning protocols and “time to target” (Dahl et al. 2023). Book 33 discussed the statistical rationale for these practices.
Sample size
The sample size requires an assumption about the standard deviation of across tasks. The book sets it with a pilot environment: a scaled-down version of the logistic regression of Chapter 4 (, , 40 tasks, condition number log-uniform on with random rotation), with “time” converted to gradient-evaluation units by the rule of Chapter 4. This is the author’s toy model, not the task suite the trial will use.
Calculation
# Sample size for the Chapter 8 pre-registered benchmark. Unit = a synthetic task drawn
# by the third party; both optimisers run on it (paired). Primary outcome per task:
# D = log2(time to target, natural gradient / time to target, Adam). Estimand: mean D
# over the task population (log2 of the geometric-mean time ratio). SESOI log2(1.25).
# PILOT (planning assumption only, not the trial suite): 40 logistic-regression tasks,
# n = 40, d = 4, kappa log-uniform on [1, 10^4], random rotation; "time" = gradient-
# evaluation units (Newton/NG step = 1 + d + d^2/(3n)); Adam: best of 3 steps, cap 500.
# Last part: the registered blinded re-estimation rule (sd of D from the first 25 % of
# tasks; sd is unchanged by swapping the arm labels; n may only go up, cap 2,000).
import math, random, statistics
NZ = statistics.NormalDist(); Z = NZ.inv_cdf
def task(rng, n=40, d=4):
kappa = 10 ** rng.uniform(0, 4); ev = [kappa ** (-i / (d - 1)) for i in range(d)]
Q = []
while len(Q) < d:
v = [rng.gauss(0, 1) for _ in range(d)]
for q in Q:
c = sum(a * b for a, b in zip(v, q)); v = [a - c * b for a, b in zip(v, q)]
s = math.sqrt(sum(a * a for a in v)); Q.append([a / s for a in v])
X = []
for _ in range(n):
z = [rng.gauss(0, 1) * math.sqrt(e) for e in ev]
X.append([sum(Q[i][j] * z[j] for j in range(d)) for i in range(d)])
w = [sum(Q[i][j] / math.sqrt(ev[j]) for j in range(d)) / d for i in range(d)]
y = [int(rng.random() < 1 / (1 + math.exp(-sum(a * b for a, b in zip(w, x)))))
for x in X]
return X, y
def lgf(X, y, w, hess=False):
d, n = len(w), len(X); L, g = 0.0, [0.0] * d; H = [[0.0] * d for _ in range(d)]
for x, t in zip(X, y):
z = sum(a * b for a, b in zip(w, x))
p = 1 / (1 + math.exp(-z)) if z >= 0 else math.exp(z) / (1 + math.exp(z))
L += math.log1p(math.exp(-abs(z))) + max(z, 0) - t * z
for i in range(d):
g[i] += (p - t) * x[i] / n
if hess:
for j in range(d):
H[i][j] += p * (1 - p) * x[i] * x[j] / n
return L / n, g, H
def newton(X, y, w):
L, g, H = lgf(X, y, w, True); d = len(w)
M = [H[i][:] + [g[i]] for i in range(d)]
for i in range(d):
M[i][i] += 1e-8
for c in range(d):
for r in range(c + 1, d):
f = M[r][c] / M[c][c]; M[r] = [a - f * b for a, b in zip(M[r], M[c])]
s = [0.0] * d
for r in range(d - 1, -1, -1):
s[r] = (M[r][d] - sum(M[r][k] * s[k] for k in range(r + 1, d))) / M[r][r]
return [a - b for a, b in zip(w, s)], L
def pilot_D(seed):
rng = random.Random(seed); X, y = task(rng); d, n = len(X[0]), len(X)
w = [0.0] * d
for _ in range(30):
w, _ = newton(X, y, w)
target = lgf(X, y, w)[0] + 1e-3; w = [0.0] * d; k_ng = 0
while lgf(X, y, w)[0] > target:
w, _ = newton(X, y, w); k_ng += 1
best = None
for lr in (0.03, 0.1, 0.3):
w = [0.0] * d; m = [0.0] * d; v = [0.0] * d
for k in range(1, 501):
L, g, _ = lgf(X, y, w)
if L <= target:
best = k - 1 if best is None else min(best, k - 1); break
m = [0.9 * a + 0.1 * b for a, b in zip(m, g)]
v = [0.999 * a + 0.001 * b * b for a, b in zip(v, g)]
w = [a - lr * (b / (1 - 0.9 ** k)) / (math.sqrt(c / (1 - 0.999 ** k)) + 1e-8)
for a, b, c in zip(w, m, v)]
t_ng = k_ng * (1 + d + d * d / (3 * n)); t_ad = best if best is not None else 1000
return math.log2(max(t_ng, 1) / max(t_ad, 1)), best is None
NP = 40
Ds = [pilot_D(s) for s in range(NP)]
D = [a for a, _ in Ds]; sd = statistics.stdev(D)
print(f"pilot ({NP} tasks): mean D {statistics.fmean(D):+.2f} (NG faster if < 0),"
f" sd {sd:.2f},"
f"\n range {min(D):+.2f} to {max(D):+.2f}; Adam missed the target on"
f" {sum(f for _, f in Ds)} task(s) (time set to 2 x cap)")
delta = math.log2(1.25); za, zb = Z(0.975), Z(0.90)
n0 = ((za + zb) * sd / delta) ** 2 + za * za / 2
n_t = math.ceil(n0)
print(f"SESOI log2(1.25) = {delta:.3f}; 90 % power, two-sided alpha 0.05:"
f" n = {n0:.1f} -> {n_t} tasks")
rng = random.Random(15); hits = 0; R = 2000
cen = [a - statistics.fmean(D) - delta for a in D] # true mean D = -SESOI
for _ in range(R):
s = [rng.choice(cen) for _ in range(n_t)]
m, se = statistics.fmean(s), statistics.stdev(s) / math.sqrt(n_t)
hits += m + 1.96 * se < 0 # "NG faster" shown
print(f" simulation ({R:,} resampled suites of {n_t}, true mean D = -SESOI):"
f" power {hits / R:.3f}")
plan = math.ceil(n_t / 0.85 / 10) * 10
print(f" allow 15 % lost to infrastructure faults: generate {plan}"
f" ({plan // 5} in each of 5 strata)")
print("sensitivity: tasks needed (before the 15 % allowance)")
SDS = (0.5 * sd, sd, 1.5 * sd, 2 * sd)
print(" sd of D: " + " ".join(f"{s:5.2f}" for s in SDS))
need = lambda s: math.ceil(((za + zb) * s / delta) ** 2 + za * za / 2)
print(" n: " + " ".join(f"{need(s):5d}" for s in SDS))
def reestimate(sd_blind, n_plan=plan, cap=2000): # registered rule, never decreases
n_new = math.ceil(need(sd_blind) / 0.85 / 10) * 10
return min(cap, max(n_plan, n_new))
print("blinded re-estimation after 25 % of tasks: sd of D -> tasks to generate")
SD2 = (1.5, 2.0, 2.5, 3.0, 4.0)
print(" " + " ".join(f"{s:.1f} -> {reestimate(s):,}" for s in SD2))
# pilot (40 tasks): mean D -2.78 (NG faster if < 0), sd 1.96,
# range -6.61 to +0.36; Adam missed the target on 4 task(s) (time set to 2 x cap)
# SESOI log2(1.25) = 0.322; 90 % power, two-sided alpha 0.05: n = 392.5 -> 393 tasks
# simulation (2,000 resampled suites of 393, true mean D = -SESOI): power 0.906
# allow 15 % lost to infrastructure faults: generate 470 (94 in each of 5 strata)
# sensitivity: tasks needed (before the 15 % allowance)
# sd of D: 0.98 1.96 2.94 3.93
# n: 100 393 881 1565
# blinded re-estimation after 25 % of tasks: sd of D -> tasks to generate
# 1.5 -> 470 2.0 -> 480 2.5 -> 750 3.0 -> 1,080 4.0 -> 1,920In the pilot the mean of is −2.78 (the natural gradient is much cheaper on these low-dimensional ill-conditioned tasks), with standard deviation 1.96 and range −6.61 to +0.36; Adam missed the target within 500 steps on 4 tasks. These numbers are planning assumptions, not evidence: the pilot has a single task family of very low dimension, the natural gradient’s cost per step is tiny, and the “natural gradient” here is Newton’s method. The real suite contains and neural networks, the distribution of is likely to be multimodal (the natural gradient far ahead on generalised linear models, possibly behind on multilayer perceptrons), and its standard deviation may be much larger than 1.96; the mean may also be close to zero or even positive. The sample size depends only on the standard deviation, not on the pilot’s mean.
The smallest effect of interest is a time ratio of 1.25, i.e. : an improvement of less than a quarter is not worth the extra complexity of implementing and maintaining a curvature method. With two-sided and 90% power, the normal approximation (with a small-sample correction for the distribution) requires 393 paired tasks; in 2,000 simulations resampling the pilot’s values shifted to a true mean of exactly −0.322, the power is 0.906. Allowing for 15% of tasks to be lost to infrastructure faults, 470 tasks are to be generated, 94 in each of the 5 strata. The sensitivity table shows that 100 tasks are needed if the standard deviation is half the pilot’s, 881 if it is 1.5 times, and 1,565 if it is twice.
Because the pilot’s standard deviation is likely to be too small, the protocol registers a blinded re-estimation rule: once the first 25% of tasks (equally across strata) are complete, a statistician who does not know the “A / B” assignment computes only the standard deviation of (swapping the arm labels changes the sign of but not its standard deviation) and does not report its mean; the required number of tasks is recomputed by the same formula with the 15% allowance, the plan may only increase, never decrease, with a cap of 2,000, and the additional tasks are added equally across strata. The last line of the program lists the results of the rule: 480 tasks for a standard deviation of 2.0, 750 for 2.5, 1,080 for 3.0 and 1,920 for 4.0. Computational tasks are cheap, so the sample size itself is not the bottleneck; the bottleneck is freezing the tasks, targets, budgets and tuning protocol in advance.
Following Chapter 6 of Book 33, “90% power” means that if the true effect is exactly the smallest effect of interest, the probability of concluding “the natural gradient is faster” (upper confidence limit below 0) is about 90%; in that case the strongest conclusion, “faster by at least the smallest effect of interest”, has only about a 2.5% chance of appearing.
Decision rule
Calculation
# Decision rule, fixed before the third party runs the frozen benchmark. Primary outcome
# per task: D = log2(T_NG / T_Adam), T = wall-clock time to the task's target loss on the
# frozen reference machine (median of 3 repeats; a run that misses the target within the
# registered budget B, diverges or returns NaN is given 2B). Estimate: mean D over valid
# tasks, 95 % CI (t interval). SESOI = log2(1.25). Only infrastructure faults (hardware,
# container, timer) make a task invalid. Gates, checked first: more than 15 % of tasks
# lost to infrastructure faults; machine differing from the registered image; arm labels
# revealed to the analysts before the lock -> NOT INTERPRETABLE. Pre-specified
# qualifier: failure rates differing by >= 10 points.
import math
SESOI = math.log2(1.25)
def verdict(d, lo, hi, fail_ng, fail_adam, invalid=0.02, image_ok=True,
unblinded=False):
for bad, why in ((invalid > 0.15, f"{invalid:.0%} of tasks lost to infrastructure"
" faults (limit 15 %)"),
(not image_ok, "machine differs from the registered image"),
(unblinded, "labels revealed to analysts before lock")):
if bad:
return f"D = {d:+.2f}: NOT INTERPRETABLE - {why}"
if hi <= -SESOI: v = "NG faster by at least the SESOI (x1.25)"
elif hi < 0: v = "NG faster; SESOI not shown"
elif lo >= SESOI: v = "Adam faster by at least the SESOI (x1.25)"
elif lo > 0: v = "Adam faster; SESOI not shown"
elif lo >= -SESOI and hi <= SESOI: v = "no difference worth acting on"
else: v = "unclear: interval too wide to decide"
if abs(fail_ng - fail_adam) >= 0.10:
worse = "NG" if fail_ng > fail_adam else "Adam"
hi_f, lo_f = max(fail_ng, fail_adam), min(fail_ng, fail_adam)
v += (f"\n qualifier: {worse} missed the target on {hi_f:.0%} of tasks,"
f" the other arm on {lo_f:.0%}")
ratio = 2 ** d
return f"D = {d:+.2f} ({lo:+.2f} to {hi:+.2f}), ratio {ratio:.2f}: {v}"
for case in ((-0.80, -1.00, -0.60, 0.05, 0.08), (-0.20, -0.38, -0.02, 0.04, 0.05),
(0.05, -0.15, 0.25, 0.03, 0.03), (0.60, 0.40, 0.80, 0.02, 0.04),
(-0.10, -0.55, 0.35, 0.05, 0.06), (-0.90, -1.10, -0.70, 0.22, 0.04),
(-0.80, -1.00, -0.60, 0.05, 0.08, 0.19),
(0.10, -0.10, 0.30, 0, 0, 0, False)):
print(verdict(*case))
print("secondary outcomes (gradient-evaluation counts, failure rates, final test loss,")
print(" peak memory, results by task family and by condition number) are reported with")
print(" intervals; they never decide the verdict")
# D = -0.80 (-1.00 to -0.60), ratio 0.57: NG faster by at least the SESOI (x1.25)
# D = -0.20 (-0.38 to -0.02), ratio 0.87: NG faster; SESOI not shown
# D = +0.05 (-0.15 to +0.25), ratio 1.04: no difference worth acting on
# D = +0.60 (+0.40 to +0.80), ratio 1.52: Adam faster by at least the SESOI (x1.25)
# D = -0.10 (-0.55 to +0.35), ratio 0.93: unclear: interval too wide to decide
# D = -0.90 (-1.10 to -0.70), ratio 0.54: NG faster by at least the SESOI (x1.25)
# qualifier: NG missed the target on 22% of tasks, the other arm on 4%
# D = -0.80: NOT INTERPRETABLE - 19% of tasks lost to infrastructure faults (limit 15 %)
# D = +0.10: NOT INTERPRETABLE - machine differs from the registered image
# secondary outcomes (gradient-evaluation counts, failure rates, final test loss,
# peak memory, results by task family and by condition number) are reported with
# intervals; they never decide the verdictThe example results are invented. If any precondition fails (more than 15% of tasks lost to infrastructure faults, a run environment that differs from the registered image, labels revealed to the analysts before the results are locked), the report says “not interpretable” whatever the primary outcome. Only when the upper confidence limit is at most −0.322 does the report say “the natural gradient is faster by at least the smallest effect of interest”; an upper limit below 0 but above −0.322 gives “the natural gradient is faster, but the size is not established”; a lower limit of at least 0.322 gives “Adam is faster by at least the smallest effect of interest”; a lower limit above 0 but below 0.322 gives “Adam is faster, but the size is not established”; an interval within ±0.322 gives “no difference worth acting on”; anything else gives “unclear”. The qualifier does not change the category of the conclusion but is appended to it: in the sixth example, for instance, the natural gradient missed the target on 22% of tasks and Adam on only 4%, and the report must say so.
What this trial cannot show
First, it does not extrapolate to large-scale training: the suite contains only low-dimensional generalised linear models and small networks of at most 50,000 parameters, and its conclusions do not apply to models with billions of parameters or to distributed training. Second, it times runs on a single CPU thread: in that setting the cost of building and inverting a curvature matrix relative to one gradient differs from that in common GPU training (where matrix operations are highly parallel and memory bandwidth is the bottleneck), so the time ratios cannot be carried over to GPUs. Third, it compares two particular implementations, one tuning protocol and the stated batch sizes; better implementations, different damping, learning-rate schedules or batch sizes (Zhang and co-authors showed that the advantage of preconditioned methods changes with batch size (Zhang et al. 2019)) could change the result. Fourth, it looks only at time to a training target, not at generalisation; validation loss is reported as a secondary outcome, but the trial is not designed for it (Wilson and co-authors reported adaptive methods generalising worse on some tasks (Wilson et al. 2017)). Fifth, synthetic tasks are not real data; condition numbers, noise and model specification are all set by the generator, and the value of the primary outcome also depends on the equal mixture of the 5 strata. Sixth, it does not test the invariance analysis of Chapter 4, the empirical Fisher or the singular learning theory of Chapter 6, which need tests of their own. Seventh, the natural gradient’s large advantage in the and pilots comes from the author’s own model, which is exactly why an independent task suite and pre-registration are needed; if the real trial finds “no difference worth acting on” or “Adam is faster”, that is the trial’s conclusion.
Ethics and conflicts of interest
The trial involves no human participants and needs no ethics-committee approval; but the protocol, the task generator, the two implementations and the analysis code must be publicly registered before any run. Whether the results are positive or negative, the results, the code and summary data for all tasks will be made public. To limit the influence of interests: the protocol and code are publicly registered before the third party runs them, the author has no access to the seeds, tasks or target values, and the results will be published whatever they are.
Disclosure
The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check. The businesses the author runs use machine-learning and AI tools in content production and in the knowledge base, but there is no direct sales relationship between the methods discussed in this book and their commercial products.
Summary
The four-step checklist examines a claim about Fisher information, priors or optimisers from four sides: coordinates, curvature, cost and regularity; the example shows that “200 times faster”, once converted to cost and once one sees that the “natural gradient” in logistic regression is Newton’s method, leaves only a conditional conclusion. The pre-registered computational benchmark compares the wall-clock time for the natural gradient and Adam to reach a target loss on 5 strata of synthetic tasks frozen by an independent third party, with 470 paired tasks (393 valid tasks for 90% power, adjusted upwards only by a blinded re-estimate of the standard deviation); the single primary outcome is the mean log time ratio, the smallest effect of interest , the decision rule is written as code, and a qualifier on failure rates is pre-specified; the results will be published whatever they are. It tests only two implementations on one class of small synthetic tasks and cannot be extended to large-scale training.
References
Fan, Y. (2026). Information Geometry, the Fisher Information Metric and Natural-Gradient Optimisation on Statistical Manifolds: From the Cramér–Rao Bound and Jeffreys's Prior to Singular Learning Theory, Read Beside Classical Chinese Thought on Change and Coordinates [信息几何、费希尔信息度量与自然梯度优化的统计流形] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836268 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).