5D Library › Liquidation Cascades, Order-Book Depth Depletion and Cointegrated Statistical Arbitrage · 中文

Chapter 6 The Kalman Filter and Dynamic Hedge Ratios

A hedge ratio that moves

The cointegrating coefficient bb of the last chapter is a constant. But two companies’ business mixes, leverage and the market’s pricing of them all change slowly, and the hedge ratio may drift with them. There are two common responses. Rolling least squares: re-run the regression each day on the last ww days. The Kalman filter: treat the hedge ratio as an unobserved state that changes randomly over time, and update it with each new observation using a certain weight.

The filter Kalman proposed in 1960 (Kalman 1960) applies to linear Gaussian state-space models. In pairs trading the state is (at,bt)(a_t, b_t) and the model is

yt=at+btxt+vt,vt∼N(0,R);(at,bt)=(at−1,bt−1)+wt,wt∼N(0,Q).y_t = a_t + b_t x_t + v_t,\quad v_t \sim N(0, R); \qquad (a_t, b_t) = (a_{t-1}, b_{t-1}) + w_t,\quad w_t \sim N(0, Q).

The observation equation says the relation between the prices carries a little noise RR each day; the state equation says the hedge ratio takes a small random step each day, with step variance set by QQ. Each day the filter does two things. Predict: carry yesterday’s estimate to today; the uncertainty PP grows by QQ. Update: compute the difference between the new observation and the prediction (the “innovation”) et=yt−at|t−1−bt|t−1xte_t = y_t – a_{t|t-1} – b_{t|t-1}x_t and its variance St=HPH⊤+RS_t = HPH^\top + R (with H=(1,xt)H = (1, x_t)), correct the state by the Kalman gain K=PH⊤/StK = PH^\top/S_t, and shrink PP. For a systematic treatment see Durbin and Koopman (Durbin and Koopman 2012). Elliott, van der Hoek and Malcolm wrote the pairs spread itself as a mean-reverting hidden state observed with noise and estimated it by methods of the same family (Elliott et al. 2005).

The ratio of QQ to RR sets the filter’s temperament: the larger QQ is relative to RR, the more readily the filter believes “the hedge ratio has changed”, the faster it reacts and the more easily it mistakes noise for signal; the smaller QQ, the closer it comes to a fixed regression on all past data.

Tracking the hedge ratio

A two-state Kalman filter written by hand with the standard library (2×2 matrices expanded), compared with rolling least squares over 30, 60, 120 and 250 days. Synthetic data, 1,000 days: in one case the hedge ratio starts at 1.0 and follows a random walk (daily standard deviation 0.005); in the other it jumps from 1.0 to 1.3 on day 500. Each estimate uses data only up to that day; scoring starts at day 250; “reach” is the number of days after the jump until the estimate first enters 1.3 ± 0.05.

Calculation

# Two-state Kalman filter (a, b random walks) vs rolling OLS on synthetic data:
# b drifts (daily sd 0.005) or jumps 1.0 -> 1.3 on day 500. Scored from day 250.
import math, random
rng = random.Random(1960)

def kalman(xs, ys, qa, qb, R=1.0):
    a, b = 0.0, 0.0
    P = [[100.0, 0.0], [0.0, 100.0]]     # vague start
    out = []
    for x, y in zip(xs, ys):
        P = [[P[0][0] + qa, P[0][1]], [P[1][0], P[1][1] + qb]]          # predict
        e = y - a - b * x
        PH = [P[0][0] + P[0][1] * x, P[1][0] + P[1][1] * x]             # P H'
        S = PH[0] + PH[1] * x + R
        K = [PH[0] / S, PH[1] / S]
        a += K[0] * e; b += K[1] * e
        P = [[P[i][j] - K[i] * PH[j] for j in range(2)] for i in range(2)]
        out.append(b)
    return out

def rolling(xs, ys, w):
    out = []
    for t in range(len(xs)):
        lo = max(0, t - w + 1); X, Y = xs[lo:t + 1], ys[lo:t + 1]; n = len(X)
        if n < 3: out.append(0.0); continue
        mx, my = sum(X) / n, sum(Y) / n
        sxx = sum((u - mx) ** 2 for u in X)
        out.append(sum((u - mx) * (v - my) for u, v in zip(X, Y)) / sxx if sxx else 0.0)
    return out

def data(kind):
    xs, bs, x, b = [], [], 50.0, 1.0
    for t in range(1000):
        x += rng.gauss(0, 1)
        if kind == "drift": b += rng.gauss(0, 0.005)
        else: b = 1.3 if t >= 500 else 1.0
        xs.append(x); bs.append(b)
    ys = [b * x + rng.gauss(0, 1) for x, b in zip(xs, bs)]
    return xs, ys, bs

def rmse(est, bs):
    return math.sqrt(sum((e - b) ** 2 for e, b in zip(est[250:], bs[250:])) / 750)

def first(est):                          # days after the step until first within 0.05
    for t in range(500, 1000):
        if abs(est[t] - 1.3) < 0.05: return t - 500


print("estimator            | drift: RMSE of b | step: RMSE of b | reach (days)")
xs1, ys1, b1 = data("drift"); xs2, ys2, b2 = data("step")
rows = [(f"Kalman, qb = {q}^2", (lambda q: lambda x, y: kalman(x, y, 1e-6, q * q))(q))
        for q in (0.005, 0.02)]
rows += [(f"rolling OLS, w = {w}", (lambda w: lambda x, y: rolling(x, y, w))(w))
         for w in (30, 60, 120, 250)]
low = []
for name, f in rows:
    e1, e2 = f(xs1, ys1), f(xs2, ys2)
    tag = ("KF " if name[0] == "K" else "OLS ") + name.split("= ")[1]
    low.append(f"{tag} {min(e2[500:]):+.2f}")
    print(f"{name:20s} |      {rmse(e1, b1):6.4f}      |     {rmse(e2, b2):6.4f}      |"
          f"      {first(e2):3d}")
print("step: lowest estimate after day 500 (true value 1.3):")
print("   " + "; ".join(low[:3])); print("   " + "; ".join(low[3:]))
# estimator            | drift: RMSE of b | step: RMSE of b | reach (days)
# Kalman, qb = 0.005^2 |      0.0229      |     0.0471      |       81
# Kalman, qb = 0.02^2  |      0.0497      |     0.0479      |        3
# rolling OLS, w = 30  |      0.1996      |     0.3188      |       30
# rolling OLS, w = 60  |      0.1663      |     0.3738      |       59
# rolling OLS, w = 120 |      0.1836      |     0.3288      |      116
# rolling OLS, w = 250 |      0.1914      |     0.5079      |      242
# step: lowest estimate after day 500 (true value 1.3):
#    KF 0.005^2 +1.05; KF 0.02^2 +1.14; OLS 30 -0.74
#    OLS 60 -0.55; OLS 120 +0.17; OLS 250 +0.24
Estimates of a hedge ratio that jumps from 1.0 to 1.3 on day 500. While a window straddles the jump, the rolling-regression slope strays far from both true values (the 60-day window briefly goes negative, the 250-day window falls as low as 0.24) until the window has fully passed the jump; the Kalman filter set for slow drift quickly rises to about 1.2 and then creeps towards 1.3.

When the hedge ratio drifts slowly and the filter is set exactly right (QQ equal to the true drift variance, intercept nearly fixed), the Kalman filter’s root-mean-square error is 0.023, while every rolling regression lies between 0.17 and 0.20: short windows are noisy, long windows cannot keep up with the drift, and every window must also estimate an intercept, which makes the slope very unstable in windows where prices have not moved much. Faced with a sudden jump, things become subtle: the filter set for slow drift needs 81 days to first enter 1.3 ± 0.05; with QQ sixteen times larger (standard deviation 0.02) it needs only 3, but its error in the drift case doubles (0.050). Rolling regression fares worse at the jump: while a window holds data from both sides of it, the slope strays far from both true values, with lowest estimates of −0.74 and −0.55 for the 30- and 60-day windows and 0.24 for the 250-day window, until the window has fully passed the jump. No single setting is best in both cases, and with real data we do not know whether change comes as slow drift or sudden jumps.

Note

This comparison favours the Kalman filter. The synthetic data are generated exactly as the filter assumes (fixed intercept, random-walk hedge ratio, white observation noise), while each rolling regression re-estimates the intercept. Real spreads are not white noise but mean-reverting, which is precisely what pairs trading tries to profit from. The next program shows that this changes the conclusion.

Tracking best is not trading best

200 synthetic pairs of 750 days each: the hedge ratio drifts slowly (daily standard deviation 0.002) and the spread is an OU process with a 10-day half-life and standard deviation 1. The intercept and hedge ratio behind the signal on day tt are estimated from data up to day t−1t-1 only and then applied to day-tt prices: for the Kalman filter, the innovation divided by its standard deviation, et/Ste_t/\sqrt{S_t}; for rolling regression, the residual from the regression on the previous ww days divided by the residual standard deviation. The rule is that of Chapter 5 (open at |z|>2|z| > 2, close at 0), costs 0.1 price units per leg per transaction (each leg has a notional of about 50, so roughly 20 basis points; returns are expressed as a percentage of the gross exposure of about 100); days 250–749 are scored. The filter’s QQ is 0.01, 1, 100 and 10,000 times the true drift variance, with R=1R = 1.

Calculation

# 200 synthetic pairs (b drifts, spread OU with half-life 10 d). Fitted to t-1, applied
# to day-t prices; |z| > 2 open, 0 close; cost 0.10 per leg per trade (each leg ~50,
# so ~20 bp of a leg); days 250-749 scored. Illustrative.
import math, random
rng = random.Random(2010)
QB, B_OU = 0.002 ** 2, 0.5 ** (1 / 10)

def pair():
    xs, ys, bs, x, b, s = [], [], [], 50.0, 1.0, rng.gauss(0, 1)
    for _ in range(750):
        x += rng.gauss(0, 1); b += rng.gauss(0, 0.002)
        s = B_OU * s + rng.gauss(0, math.sqrt(1 - B_OU ** 2))
        xs.append(x); ys.append(b * x + s); bs.append(b)
    return xs, ys, bs

def kalman_signals(xs, ys, qa, qb, R=1.0):
    a, b, P, out = 0.0, 0.0, [[100.0, 0.0], [0.0, 100.0]], []
    for x, y in zip(xs, ys):
        P = [[P[0][0] + qa, P[0][1]], [P[1][0], P[1][1] + qb]]
        e = y - a - b * x
        PH = [P[0][0] + P[0][1] * x, P[1][0] + P[1][1] * x]; S = PH[0] + PH[1] * x + R
        out.append((b, e / math.sqrt(S)))  # prediction made before seeing y
        K = [PH[0] / S, PH[1] / S]; a += K[0] * e; b += K[1] * e
        P = [[P[i][j] - K[i] * PH[j] for j in range(2)] for i in range(2)]
    return out

def ols_signals(xs, ys, w):
    out = [(0.0, 0.0)]
    for t in range(1, len(xs)):
        X, Y = xs[max(0, t - w):t], ys[max(0, t - w):t]; n = len(X)
        mx, my = sum(X) / n, sum(Y) / n; sxx = sum((u - mx) ** 2 for u in X)
        b = sum((u - mx) * (v - my) for u, v in zip(X, Y)) / sxx if sxx > 0 else 0.0
        a = my - b * mx
        sd = math.sqrt(sum((v - a - b * u) ** 2 for u, v in zip(X, Y)) / max(n - 2, 1))
        out.append((b, (ys[t] - a - b * xs[t]) / sd if sd > 0 else 0.0))
    return out

def trade(xs, ys, sig, cost=0.10):
    pos, pnl = 0, 0.0
    for t in range(250, 749):
        b, z = sig[t]
        if pos == 0 and abs(z) > 2: pos = -1 if z > 0 else 1; pnl -= 2 * cost
        elif pos and z * pos >= 0: pos = 0; pnl -= 2 * cost
        pnl += pos * ((ys[t + 1] - ys[t]) - b * (xs[t + 1] - xs[t]))
    return (pnl - 2 * cost * (pos != 0)) / 2     # per year, % of ~100 gross

pairs = [pair() for _ in range(200)]
def kf(k): return lambda x, y: kalman_signals(x, y, 1e-6 * k, QB * k)
def ols(w): return lambda x, y: ols_signals(x, y, w)
cfg = [(f"Kalman, q x {k:g}", kf(k)) for k in (0.01, 1, 100, 10000)]
cfg += [(f"rolling OLS, w = {w}", ols(w)) for w in (60, 120)]
res = {}
print("method               | RMSE of b | net P&L % per year: mean (SE) | pairs > 0")
for name, f in cfg:
    pn, err = [], []
    for xs, ys, bs in pairs:
        sig = f(xs, ys)
        err += [(sig[t][0] - bs[t]) ** 2 for t in range(250, 750)]
        pn.append(trade(xs, ys, sig))
    res[name] = pn
    m = sum(pn) / 200; se = math.sqrt(sum((p - m) ** 2 for p in pn) / 199 / 200)
    pos = sum(p > 0 for p in pn) / 200
    print(f"{name:20s} |  {math.sqrt(sum(err) / len(err)):6.3f}   |"
          f"         {m:5.2f} ({se:4.2f})          |  {pos:5.1%}")
d = [k - o for k, o in zip(res["Kalman, q x 1"], res["rolling OLS, w = 60"])]
md = sum(d) / 200; sd = math.sqrt(sum((v - md) ** 2 for v in d) / 199)
print(f"paired difference, Kalman (q x 1) minus OLS w = 60: {md:+.2f} % per year")
print(f"   SD across pairs {sd:.2f}, SE {sd / math.sqrt(200):.2f}")
# method               | RMSE of b | net P&L % per year: mean (SE) | pairs > 0
# Kalman, q x 0.01     |   0.059   |          3.21 (0.11)          |  99.0%
# Kalman, q x 1        |   0.042   |          0.58 (0.06)          |  58.0%
# Kalman, q x 100      |   0.039   |          0.02 (0.01)          |   2.5%
# Kalman, q x 10000    |   0.045   |          0.00 (0.00)          |   0.5%
# rolling OLS, w = 60  |   0.148   |          2.49 (0.13)          |  91.5%
# rolling OLS, w = 120 |   0.112   |          3.31 (0.12)          |  96.0%
# paired difference, Kalman (q x 1) minus OLS w = 60: -1.91 % per year
#    SD across pairs 1.93, SE 0.14

The result runs against intuition. With QQ at its true value, the filter tracks the hedge ratio with far less error (0.042) than the 60-day rolling regression (0.148), yet earns only 0.58 % a year against 2.49 % for the 60-day rolling regression; the mean pair-by-pair difference is −1.91 percentage points (standard error 0.14). Raising QQ further leaves tracking error almost unchanged but drives trading returns to zero. The reason: part of a mean-reverting spread looks to the filter like “the hedge ratio has changed”; the larger QQ, the more the filter absorbs the spread’s deviations into ata_t and btb_t, the smaller the innovations and the fewer the trading signals. With QQ at one-hundredth of its true value the filter barely moves, and its trading return (3.21 %) is similar to the 120-day rolling regression (3.31 %).

Three lessons follow. First, “the Kalman filter beats rolling regression” is not a claim that can be made in general; it depends on the settings of QQ and RR, on how the true hedge ratio changes, and on whether the yardstick is tracking error or trading return after costs. Second, QQ and RR cannot be tuned and evaluated on the same data: picking the best backtest out of many settings is exactly what Bailey and colleagues call backtest overfitting (Bailey et al. 2014). Third, these are results on synthetic data with illustrative parameters; they show why a test is needed but cannot replace a test on real data.

Hypothesis · not established

Kalman-filter hedging beats rolling regression on real data (not established). The framework’s claim is that pairs trading with a hedge ratio estimated dynamically by a Kalman filter earns a higher out-of-sample return after realistic costs than with 60-day rolling least squares. The author treats this as a hypothesis, not a conclusion. Falsifiable prediction: in the pre-registered out-of-sample test of Chapter 8, if the confidence interval for the pair-by-pair difference in annualised net return lies entirely within ±1 percentage point, or entirely below zero, the hypothesis is rejected or not worth adopting.

Chapter summary

The Kalman filter treats the hedge ratio as a hidden random-walk state and corrects it each day by the innovation times the gain; Q/RQ/R sets how quickly it reacts. On synthetic data where its settings are exactly right and observation noise is white, it tracks the hedge ratio far better than rolling regression; but faced with sudden jumps no setting is best everywhere. More importantly, when the spread itself is mean-reverting, the setting that tracks best absorbs the trading signal, and on synthetic pairs its net return falls below that of 60-day rolling regression. Whether the Kalman filter does better on real data must be answered by a pre-registered, out-of-sample test with realistic costs, which is the protocol of Chapter 8. First, the next chapter returns to Sichuan about a thousand years ago, to see how a paper money backed by a “reserve” lost its credit through over-issue.

References

Bailey, David H., Jonathan M. Borwein, Marcos López de Prado, and Qiji Jim Zhu. 2014. “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance.” Notices of the American Mathematical Society 61 (5): 458–71.
Durbin, James, and Siem Jan Koopman. 2012. Time Series Analysis by State Space Methods. 2nd ed. Oxford University Press.
Elliott, Robert J., John van der Hoek, and William P. Malcolm. 2005. “Pairs Trading.” Quantitative Finance 5 (3): 271–76. https://doi.org/10.1080/14697680500149370.
Kalman, Rudolph E. 1960. “A New Approach to Linear Filtering and Prediction Problems.” Journal of Basic Engineering 82 (1): 35–45. https://doi.org/10.1115/1.3662552.

Fan, Y. (2026). Liquidation Cascades, Order-Book Depth Depletion and Cointegrated Statistical Arbitrage: Ornstein-Uhlenbeck Processes and Kalman-Filter Dynamic Hedging [高杠杆衍生品清算瀑布、订单簿深度衰竭与协整统计套利] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23796776 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).