What does Kalman Filtering for Readings and Ad Reports conclude, in numbers, about “Looking Back and Filling Gaps: the RTS Smoother, Missing Days and Artefact Readings”?

Answer: In review, backward smoothing lowers the error of the middle days from 0.940 to 0.705, but it does nothing for the last day. Missing days need no imputation: the filter lets its uncertainty grow honestly and speeds up its catch-up when readings resume. Artefact readings should be stopped by an innovation gate before they enter the estimate and passed to a person for review. Up to this point the book has assumed that we know the two noise variances \(q\) and \(r\). In reality nobody tells us these numbers. Can they be estimated from the readings alone, and are four weeks of data enough? The next chapter answers.

Scope of the numbers: Its noise variances, drifts, returns and budgets are illustrative parameters; all calculations are generated by its programs. On wearable readings it is not medical advice, claims no effect, judges no one's state of health, and asks readers to consult a licensed physician before any self-experiment; on advertising attribution it is not individual business advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.705, 0.940

log of 28 days
  middle day: sd filter 0.940  smoother 0.705   (theory)
  middle day: RMSE filter 0.937  smoother 0.695
  last day:   RMSE filter 0.951  smoother 0.951
log of 120 days
  middle day: sd filter 0.940  smoother 0.704   (theory)
  middle day: RMSE filter 0.953  smoother 0.710
  last day:   RMSE filter 0.941  smoother 0.941
single reading sd 2.000 for comparison

Key formula

\[m^s_t=m_t+C_t\,(m^s_{t+1}-m_t),\qquad C_t=\frac{p_t}{P_{t+1}}.\]
Show the full program
# Chapter 4. p41: Looking back with the RTS smoother.
# Rauch, Tung and Striebel (1965): after the forward Kalman pass, a
# backward pass  m_s[t] = m_f[t] + C (m_s[t+1] - m_f[t]),
# C = P_f[t] / P_p[t+1],  P_s[t] = P_f[t] + C^2 (P_s[t+1] - P_p[t+1]).
# Local level, q = 0.25 and r = 4 as in p11; 28-day logs (four weeks)
# and 120-day logs. Reported: theoretical sd of the filtered and the
# smoothed estimate at the middle and at the last day, and the RMSE of
# both over 600 simulated logs. Fixed seed; illustrative numbers.
import math
import random
import statistics

Q, R = 0.25, 4.0
rng = random.Random(20261118)


def forward(ys, m=0.0, p=100.0):
    mf, pf, pp_ = [], [], []
    for y in ys:
        pp = p + Q
        k = pp / (pp + R)
        m = m + k * (y - m)
        p = pp * (1 - k)
        mf.append(m)
        pf.append(p)
        pp_.append(pp)
    return mf, pf, pp_


def rts(mf, pf, pp_):
    ms, ps = mf[:], pf[:]
    for t in range(len(mf) - 2, -1, -1):
        c = pf[t] / pp_[t + 1]
        ms[t] = mf[t] + c * (ms[t + 1] - mf[t])
        ps[t] = pf[t] + c * c * (ps[t + 1] - pp_[t + 1])
    return ms, ps


for n in (28, 120):
    ef, es, el_f, el_s = [], [], [], []
    for _ in range(600):
        mu, mus, ys = 0.0, [], []
        for _ in range(n):
            mu += rng.gauss(0, math.sqrt(Q))
            mus.append(mu)
            ys.append(mu + rng.gauss(0, math.sqrt(R)))
        mf, pf, pp_ = forward(ys)
        ms, ps = rts(mf, pf, pp_)
        mid = n // 2
        ef.append((mf[mid] - mus[mid]) ** 2)
        es.append((ms[mid] - mus[mid]) ** 2)
        el_f.append((mf[-1] - mus[-1]) ** 2)
        el_s.append((ms[-1] - mus[-1]) ** 2)
    print("log of %d days" % n)
    print("  middle day: sd filter %.3f  smoother %.3f   (theory)"
          % (math.sqrt(pf[mid]), math.sqrt(ps[mid])))
    print("  middle day: RMSE filter %.3f  smoother %.3f"
          % (math.sqrt(statistics.fmean(ef)),
             math.sqrt(statistics.fmean(es))))
    print("  last day:   RMSE filter %.3f  smoother %.3f"
          % (math.sqrt(statistics.fmean(el_f)),
             math.sqrt(statistics.fmean(el_s))))
print("single reading sd %.3f for comparison" % math.sqrt(R))

Single source and citation

Fan, Y. (2026). Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports (Version 1.0.0), Chapter 4: Looking Back and Filling Gaps: the RTS Smoother, Missing Days and Artefact Readings. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-245 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026245,
  author    = {Fan, Yafu},
  title     = {Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book