What does Kalman Filtering for Readings and Ad Reports conclude, in numbers, about “How Large Is the Noise? Maximum Likelihood, Innovation Tests and Moments of Differences”?

Answer: Chapter 5 covers maximum likelihood, innovation tests and moments of differences: a gain estimated from four weeks of data lies between 0.031 and 0.481 and is only a starting point.

Scope of the numbers: Its noise variances, drifts, returns and budgets are illustrative parameters; all calculations are generated by its programs. On wearable readings it is not medical advice, claims no effect, judges no one's state of health, and asks readers to consult a licensed physician before any self-experiment; on advertising attribution it is not individual business advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.031, 0.481

true rho = r/q16, true gain K = 0.2207
days rho 5% median95% K 5% median 95%
282.00 28.18 1000.0 0.031 0.171 0.481
fits at the grid edge rho1000 (level looks flat): 73 of 300
1206.31 15.85 70.8 0.112 0.222 0.327
fits at the grid edge rho1000 (level looks flat): 1 of 300

Key formula

\[\log L=-\tfrac12\sum_t\left(\log F_t+\frac{v_t^2}{F_t}\right).\]
Show the full program
# Chapter 5. p51: Estimating the noise ratio from the readings alone.
# Prediction error decomposition: with q = s2 and r = s2 * rho, the
# log-likelihood is -0.5 sum(log F + v^2 / F); s2 is concentrated out,
# s2_hat = mean(v^2 / F*), and rho is found on a log grid from 0.01 to
# 1000. The filter starts from the first reading (posterior var r).
# True q = 0.25, r = 4, so rho = 16 and the steady gain is 0.2207.
# 300 simulated logs each of 28 days (four weeks) and 120 days; fixed
# seed. Reported: median and 5 % / 95 % points of rho_hat and K_hat.
import math
import random
import statistics

Q, R = 0.25, 4.0
GRID = [10 ** (-2 + 5 * i / 100) for i in range(101)]
rng = random.Random(20261120)


def gain(rho):
    p = (1 + math.sqrt(1 + 4 * rho)) / 2
    return p / (p + rho)


def profile(ys, rho):
    m, p, terms, logf = ys[0], rho, [], []
    for y in ys[1:]:
        pp = p + 1.0
        f = pp + rho
        v = y - m
        terms.append(v * v / f)
        logf.append(math.log(f))
        k = pp / f
        m, p = m + k * v, pp * (1 - k)
    n = len(terms)
    s2 = math.fsum(terms) / n
    return -0.5 * (math.fsum(logf) + n * math.log(s2) + n), s2


def fit(ys):
    best = max(GRID, key=lambda g: profile(ys, g)[0])
    return best, profile(ys, best)[1]


def pct(xs, a):
    xs = sorted(xs)
    return xs[int(a * (len(xs) - 1))]


print("true rho = r/q = %.0f, true gain K = %.4f" % (R / Q, gain(R / Q)))
print("days  rho 5%     median   95%      K 5%    median  95%")
for n in (28, 120):
    rhos, ks = [], []
    for _ in range(300):
        mu, ys = 0.0, []
        for _ in range(n):
            mu += rng.gauss(0, math.sqrt(Q))
            ys.append(mu + rng.gauss(0, math.sqrt(R)))
        rho, _ = fit(ys)
        rhos.append(rho)
        ks.append(gain(rho))
    print("%4d  %7.2f  %7.2f  %7.1f   %.3f   %.3f  %.3f"
          % (n, pct(rhos, 0.05), statistics.median(rhos),
             pct(rhos, 0.95), pct(ks, 0.05), statistics.median(ks),
             pct(ks, 0.95)))
    edge = [g >= GRID[-1] for g in rhos].count(True)
    print("      fits at the grid edge rho = 1000 (level looks flat):"
          " %d of 300" % edge)

Single source and citation

Fan, Y. (2026). Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports (Version 1.0.0), Chapter 5: How Large Is the Noise? Maximum Likelihood, Innovation Tests and Moments of Differences. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-245 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026245,
  author    = {Fan, Yafu},
  title     = {Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book