What does Kalman Filtering for Readings and Ad Reports conclude, in numbers, about “Looking Back and Filling Gaps: the RTS Smoother, Missing Days and Artefact Readings”?
Answer: In review, backward smoothing lowers the error of the middle days from 0.940 to 0.705, but it does nothing for the last day. Missing days need no imputation: the filter lets its uncertainty grow honestly and speeds up its catch-up when readings resume. Artefact readings should be stopped by an innovation gate before they enter the estimate and passed to a person for review. Up to this point the book has assumed that we know the two noise variances \(q\) and \(r\). In reality nobody tells us these numbers. Can they be estimated from the readings alone, and are four weeks of data enough? The next chapter answers.
Scope of the numbers: Its noise variances, drifts, returns and budgets are illustrative parameters; all calculations are generated by its programs. On wearable readings it is not medical advice, claims no effect, judges no one's state of health, and asks readers to consult a licensed physician before any self-experiment; on advertising attribution it is not individual business advice.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.705, 0.940
log of 28 days middle day: sd filter 0.940 smoother 0.705 (theory) middle day: RMSE filter 0.937 smoother 0.695 last day: RMSE filter 0.951 smoother 0.951 log of 120 days middle day: sd filter 0.940 smoother 0.704 (theory) middle day: RMSE filter 0.953 smoother 0.710 last day: RMSE filter 0.941 smoother 0.941 single reading sd 2.000 for comparison
Key formula
Show the full program
# Chapter 4. p41: Looking back with the RTS smoother.
# Rauch, Tung and Striebel (1965): after the forward Kalman pass, a
# backward pass m_s[t] = m_f[t] + C (m_s[t+1] - m_f[t]),
# C = P_f[t] / P_p[t+1], P_s[t] = P_f[t] + C^2 (P_s[t+1] - P_p[t+1]).
# Local level, q = 0.25 and r = 4 as in p11; 28-day logs (four weeks)
# and 120-day logs. Reported: theoretical sd of the filtered and the
# smoothed estimate at the middle and at the last day, and the RMSE of
# both over 600 simulated logs. Fixed seed; illustrative numbers.
import math
import random
import statistics
Q, R = 0.25, 4.0
rng = random.Random(20261118)
def forward(ys, m=0.0, p=100.0):
mf, pf, pp_ = [], [], []
for y in ys:
pp = p + Q
k = pp / (pp + R)
m = m + k * (y - m)
p = pp * (1 - k)
mf.append(m)
pf.append(p)
pp_.append(pp)
return mf, pf, pp_
def rts(mf, pf, pp_):
ms, ps = mf[:], pf[:]
for t in range(len(mf) - 2, -1, -1):
c = pf[t] / pp_[t + 1]
ms[t] = mf[t] + c * (ms[t + 1] - mf[t])
ps[t] = pf[t] + c * c * (ps[t + 1] - pp_[t + 1])
return ms, ps
for n in (28, 120):
ef, es, el_f, el_s = [], [], [], []
for _ in range(600):
mu, mus, ys = 0.0, [], []
for _ in range(n):
mu += rng.gauss(0, math.sqrt(Q))
mus.append(mu)
ys.append(mu + rng.gauss(0, math.sqrt(R)))
mf, pf, pp_ = forward(ys)
ms, ps = rts(mf, pf, pp_)
mid = n // 2
ef.append((mf[mid] - mus[mid]) ** 2)
es.append((ms[mid] - mus[mid]) ** 2)
el_f.append((mf[-1] - mus[-1]) ** 2)
el_s.append((ms[-1] - mus[-1]) ** 2)
print("log of %d days" % n)
print(" middle day: sd filter %.3f smoother %.3f (theory)"
% (math.sqrt(pf[mid]), math.sqrt(ps[mid])))
print(" middle day: RMSE filter %.3f smoother %.3f"
% (math.sqrt(statistics.fmean(ef)),
math.sqrt(statistics.fmean(es))))
print(" last day: RMSE filter %.3f smoother %.3f"
% (math.sqrt(statistics.fmean(el_f)),
math.sqrt(statistics.fmean(el_s))))
print("single reading sd %.3f for comparison" % math.sqrt(R))Single source and citation
Fan, Y. (2026). Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports (Version 1.0.0), Chapter 4: Looking Back and Filling Gaps: the RTS Smoother, Missing Days and Artefact Readings. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-245 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026245,
author = {Fan, Yafu},
title = {Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Why Raw Readings Overreact: the Noise-to-Signal Ratio and Decision Flips
- Chapter 2 · Where the Kalman Gain Comes From: One Prediction, One Correction and Exponential Smoothing
- Chapter 3 · Trends and Steps: Slow Drift Takes Weeks to Show, a Real Jump Costs Days of Lag
- Chapter 5 · How Large Is the Noise? Maximum Likelihood, Innovation Tests and Moments of Differences
- Chapter 6 · The Advertising Report: Attribution Noise, On/Off Flighting and Budget Switching
- Chapter 7 · Classical Sources East and West: the Mencius and Gauss's Least Squares
- Chapter 8 · Closing the Loop: a Four-Week Filtered Log, Filtered Advertising Reports and Pre-Registered Tests