What does Kalman Filtering for Readings and Ad Reports conclude, in numbers, about “Trends and Steps: Slow Drift Takes Weeks to Show, a Real Jump Costs Days of Lag”?
Answer: Adding a slope state turns the local level model into the local trend model. With the illustrative parameters, a single reading's error is twelve times the daily drift, and comparing two readings tells almost nothing about trend; the filter needs about eight weeks of continuous records to have a 0.882 chance of seeing it. For a real step the filter does need a few days, but at equal false-alarm rates it is no slower than alarming on raw readings. The first three chapters have looked at today in real time. If we are willing to look back and use today's readings to revise our judgement of last week, how much further can the error be cut? And what should be done with missing days and readings that are plainly absurd? The next chapter answers.
Scope of the numbers: Its noise variances, drifts, returns and budgets are illustrative parameters; all calculations are generated by its programs. On wearable readings it is not medical advice, claims no effect, judges no one's state of health, and asks readers to consult a licensed physician before any self-experiment; on advertising attribution it is not individual business advice.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.882
| true drift 0.05 per day, reading sd 0.6 | |||
| naive week-over-week slope sd: 0.1212 | |||
| days | filter sd | OLS sd | share |slope| > 2 sd |
| 7 | 0.1201 | 0.1134 | 0.060 |
| 14 | 0.0494 | 0.0398 | 0.182 |
| 21 | 0.0323 | 0.0216 | 0.340 |
| 28 | 0.0251 | 0.0140 | 0.510 |
| 42 | 0.0187 | 0.0076 | 0.725 |
| 56 | 0.0157 | 0.0050 | 0.882 |
Key formula
Show the full program
# Chapter 3. p31: A slow drift needs weeks, not two readings, to show.
# Local linear trend: level l[t+1] = l[t] + b[t] + u, slope
# b[t+1] = b[t] + w, reading y = l + e. Illustrative scale-like units:
# true drift 0.05 per day, level noise sd 0.1, slope noise sd 0.001,
# reading error sd 0.6. Columns: posterior sd of the slope from the
# filter (exact Riccati recursion), the least-squares slope sd for a
# straight line with no level noise, the sd of the naive "this week's
# reading minus last week's / 7", and the share of 400 runs where the
# filter's slope is more than two sd away from zero. Fixed seed.
import math
import random
B0, QL, QB, R = 0.05, 0.1 ** 2, 0.001 ** 2, 0.6 ** 2
rng = random.Random(20261116)
CHECK = (7, 14, 21, 28, 42, 56)
def filt(ys):
a0, a1 = ys[0], 0.0
p00, p01, p11 = R, 0.0, 1.0
rows = []
for t, y in enumerate(ys[1:], 2):
a0 = a0 + a1
p00, p01, p11 = (p00 + 2 * p01 + p11 + QL, p01 + p11, p11 + QB)
s = p00 + R
k0, k1 = p00 / s, p01 / s
v = y - a0
a0, a1 = a0 + k0 * v, a1 + k1 * v
p00, p01, p11 = (p00 - k0 * p00, p01 - k0 * p01, p11 - k1 * p01)
rows.append((t, a1, math.sqrt(p11)))
return rows
hits = {n: 0 for n in CHECK}
sds = {}
for _ in range(400):
lvl, b, ys = 0.0, B0, []
for _ in range(max(CHECK)):
ys.append(lvl + rng.gauss(0, math.sqrt(R)))
lvl += b + rng.gauss(0, math.sqrt(QL))
b += rng.gauss(0, math.sqrt(QB))
for t, slope, sd in filt(ys):
if t in hits:
sds[t] = sd
if abs(slope) > 2 * sd:
hits[t] += 1
naive = math.sqrt(2 * R) / 7
print("true drift %.2f per day, reading sd %.1f" % (B0, math.sqrt(R)))
print("naive week-over-week slope sd: %.4f" % naive)
print("days filter sd OLS sd share |slope| > 2 sd")
for n in CHECK:
ols = math.sqrt(R * 12 / (n * (n * n - 1)))
print("%4d %.4f %.4f %.3f" % (n, sds[n], ols, hits[n] / 400))Single source and citation
Fan, Y. (2026). Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports (Version 1.0.0), Chapter 3: Trends and Steps: Slow Drift Takes Weeks to Show, a Real Jump Costs Days of Lag. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-245 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026245,
author = {Fan, Yafu},
title = {Kalman Filtering and State-Space Assimilation: Separating Observation Noise from True Signals in Wearable Readings and Advertising Reports},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Why Raw Readings Overreact: the Noise-to-Signal Ratio and Decision Flips
- Chapter 2 · Where the Kalman Gain Comes From: One Prediction, One Correction and Exponential Smoothing
- Chapter 4 · Looking Back and Filling Gaps: the RTS Smoother, Missing Days and Artefact Readings
- Chapter 5 · How Large Is the Noise? Maximum Likelihood, Innovation Tests and Moments of Differences
- Chapter 6 · The Advertising Report: Attribution Noise, On/Off Flighting and Budget Switching
- Chapter 7 · Classical Sources East and West: the Mencius and Gauss's Least Squares
- Chapter 8 · Closing the Loop: a Four-Week Filtered Log, Filtered Advertising Reports and Pre-Registered Tests