5D Library › Macro-Liquidity Regime Switching, Extreme Tail Dependence and Barbell Asset Allocation · 中文
Chapter 1 Hidden Markov Models and Volatility Regimes
Calm and stress
Plot a long series of daily returns and the most striking feature is usually not the average level but the clustering of volatility: large moves follow large moves and quiet days follow quiet days. Book 20 described this clustering with the GARCH model, in which today’s variance is a weighted sum of yesterday’s variance and yesterday’s squared return, so that volatility changes continuously (Bollerslev 1986). This chapter takes another angle. Suppose the market switches between a few discrete “regimes”, each with its own mean and volatility, and we cannot see which regime we are in but can only infer it from returns. That is a hidden Markov model (HMM).
Hamilton’s Markov-switching model
Evidence
Hamilton proposed in 1989 to let an unobservable two-state Markov chain drive the mean of a time series, in order to describe expansions and recessions in the business cycle (Hamilton 1989). He used quarterly growth of US post-war real GNP (1951 to 1984), and what switched was the mean growth rate, not volatility. The estimated persistence probabilities of expansion and recession were about 0.90 and 0.75, giving expected durations of about 10 and 4 quarters, and the recessions inferred by the model broadly matched the recession dates of the National Bureau of Economic Research (NBER). The framework was later extended widely to models in which the means, volatilities and correlations of returns switch together; Ang and Bekaert, for example, studied international equity allocation and found that correlations between markets are higher in the high-volatility bear regime, but that international diversification remains valuable and that ignoring regimes is costly mainly when a risk-free asset can be held (Ang and Bekaert 2002).
The author’s framework cites “Hamilton 1989” as the source for identifying volatility regimes. After checking, it should be said that Hamilton’s original model switched the mean of GNP growth; using HMMs to identify volatility regimes is an extension of the same mathematics, not part of the original paper.
The model and three algorithms
Let the hidden state be a Markov chain with transition probabilities ; given the state, the observed return is . Three problems need solving: given the parameters, what is the probability of each state on each day (filtering and smoothing); how are the parameters estimated (the Baum–Welch algorithm); and what is the most likely whole path of states (the Viterbi algorithm)?
Forward–backward. The forward variable follows the recursion
normalised at each step by a constant ; the sum of the logs of these constants is the log-likelihood, . The backward recursion gives , and their product is the smoothed probability , inferred from all the data. uses only information up to day and is the filtered probability available in real time; uses the future and can be computed only after the fact.
Baum–Welch (EM). The algorithm Baum and co-authors gave in 1970, later placed by Dempster, Laird and Rubin within the general EM framework (Baum et al. 1970; Dempster et al. 1977): the E-step uses forward–backward to compute and the joint probabilities of adjacent days, and the M-step uses them as weights to update the parameters:
No iteration lowers the likelihood, but convergence is guaranteed only to some local maximum.
Viterbi. The dynamic programme Viterbi introduced in 1967 for decoding convolutional codes (Viterbi 1967): for each state, remember the log-probability of the most likely path that reaches it and where that path came from at the previous step, then trace back from the end to obtain the whole path with the highest joint probability. It answers a different question from the smoothed probabilities: they are day-by-day marginal judgements, whereas the Viterbi path is the single most likely sequence as a whole.
The program below simulates 2,500 trading days (about ten years) from a two-regime model: in the calm regime the mean daily return is 0.04 % and daily volatility 0.8 %; in the stressed regime −0.08 % and 2.0 %; the calm regime persists with probability 0.99 and the stressed regime with 0.97. It then fits the model from three different starting points.
Calculation
# Two-state Gaussian HMM for daily returns (%), fitted by Baum-Welch (EM) with scaled
# forward-backward recursions and decoded by Viterbi. SIMULATED, ILLUSTRATIVE: calm
# (mean 0.04, sd 0.8), stress (mean -0.08, sd 2.0); P(stay) 0.99 and 0.97; 2,500 days.
import math, random
TRUE = dict(mu=[0.04, -0.08], sd=[0.8, 2.0], P=[[0.99, 0.01], [0.03, 0.97]], pi=[1, 0])
def simulate(T=2500, seed=1989, par=TRUE):
rng = random.Random(seed); s = 0 if rng.random() < par["pi"][0] else 1; x, z = [], []
for _ in range(T):
z.append(s); x.append(rng.gauss(par["mu"][s], par["sd"][s]))
s = s if rng.random() < par["P"][s][s] else 1 - s
return x, z
def dens(x, mu, sd):
return math.exp(-0.5 * ((x - mu) / sd) ** 2) / (sd * math.sqrt(2 * math.pi))
def forward(x, mu, sd, P, pi): # a[t][k] = P(state k | x_1..x_t)
K, a, c = len(mu), [], []
for t, xt in enumerate(x):
b = [dens(xt, mu[k], sd[k]) for k in range(K)]
f = ([pi[k] * b[k] for k in range(K)] if t == 0 else
[b[k] * sum(a[-1][j] * P[j][k] for j in range(K)) for k in range(K)])
s = sum(f); c.append(s); a.append([v / s for v in f])
return a, c # log L = sum of log c
def backward(x, mu, sd, P, c):
K, T = len(mu), len(x); be = [[1.0] * K for _ in range(T)]
for t in range(T - 2, -1, -1):
b = [dens(x[t + 1], mu[k], sd[k]) for k in range(K)]
be[t] = [sum(P[j][k] * b[k] * be[t + 1][k] for k in range(K)) / c[t + 1]
for j in range(K)]
return be
def fit(x, mu, sd, P, pi, tol=1e-7, max_iter=500):
K, T, old = len(mu), len(x), -1e300
for it in range(1, max_iter + 1):
a, c = forward(x, mu, sd, P, pi); be = backward(x, mu, sd, P, c)
ll = sum(math.log(v) for v in c)
g = [[a[t][k] * be[t][k] for k in range(K)] for t in range(T)]
xi = [[0.0] * K for _ in range(K)]
for t in range(T - 1):
b = [dens(x[t + 1], mu[k], sd[k]) for k in range(K)]
for j in range(K):
for k in range(K):
xi[j][k] += a[t][j] * P[j][k] * b[k] * be[t + 1][k] / c[t + 1]
P = [[xi[j][k] / sum(xi[j]) for k in range(K)] for j in range(K)]; pi = g[0][:]
for k in range(K):
w = sum(r[k] for r in g); mu[k] = sum(r[k] * v for r, v in zip(g, x)) / w
sd[k] = max(1e-3, math.sqrt(sum(r[k] * (v - mu[k]) ** 2
for r, v in zip(g, x)) / w))
if ll - old < tol:
break
old = ll
return dict(mu=mu, sd=sd, P=P, pi=pi, ll=ll, iters=it, gamma=g)
def viterbi(x, mu, sd, P, pi):
K = len(mu); lg = lambda v: math.log(max(v, 1e-300)); back = []
d = [lg(pi[k]) + lg(dens(x[0], mu[k], sd[k])) for k in range(K)]
for xt in x[1:]:
best = [max(range(K), key=lambda j: d[j] + lg(P[j][k])) for k in range(K)]
d = [d[best[k]] + lg(P[best[k]][k]) + lg(dens(xt, mu[k], sd[k]))
for k in range(K)]
back.append(best)
s = [max(range(K), key=lambda k: d[k])]
for best in reversed(back):
s.append(best[s[-1]])
return s[::-1]
if __name__ == "__main__":
x, z = simulate(); P0 = [[0.95, 0.05], [0.05, 0.95]]
print(f"simulated {len(x)} days; true share of stress days {sum(z) / len(z):.3f}")
for name, m0, s0 in (("A (calm first)", [0.0, 0.0], [0.5, 1.5]),
("B (stress first)", [0.0, 0.0], [1.5, 0.5]),
("C (nearly equal)", [0.01, -0.01], [1.0, 1.01])):
r = fit(x, m0, s0, P0, [0.5, 0.5])
print(f"start {name}: log L {r['ll']:.1f} after {r['iters']} iterations")
for k in range(2):
p = r["P"][k][k]
print(f" state {k}: mean {r['mu'][k]:+.3f} sd {r['sd'][k]:.3f}"
f" P(stay) {p:.4f} duration {1 / (1 - p):6.1f} days")
r = fit(x, [0.0, 0.0], [0.5, 1.5], P0, [0.5, 0.5])
path = viterbi(x, r["mu"], r["sd"], r["P"], r["pi"])
smooth = [1 if g[1] > 0.5 else 0 for g in r["gamma"]]
agree = lambda s: sum(p == q for p, q in zip(s, z)) / len(z)
print(f"Viterbi path agrees with the true states on {agree(path):.1%} of days;"
f" smoothed P > 0.5 on {agree(smooth):.1%}")
print("true durations: calm 100 days, stress 33.3 days")
# simulated 2500 days; true share of stress days 0.142
# start A (calm first): log L -3360.1 after 22 iterations
# state 0: mean +0.030 sd 0.801 P(stay) 0.9933 duration 148.8 days
# state 1: mean +0.022 sd 1.890 P(stay) 0.9579 duration 23.8 days
# start B (stress first): log L -3360.1 after 22 iterations
# state 0: mean +0.022 sd 1.890 P(stay) 0.9579 duration 23.8 days
# state 1: mean +0.030 sd 0.801 P(stay) 0.9933 duration 148.8 days
# start C (nearly equal): log L -3360.1 after 20 iterations
# state 0: mean +0.030 sd 0.801 P(stay) 0.9933 duration 148.8 days
# state 1: mean +0.022 sd 1.890 P(stay) 0.9579 duration 23.8 days
# Viterbi path agrees with the true states on 95.4% of days; smoothed P > 0.5 on 96.0%
# true durations: calm 100 days, stress 33.3 daysAll three starts converge to the same log-likelihood, −3360.1, so no bad local maximum was met here; but start B labels the stressed regime “state 0” and the calm regime “state 1”, exactly the reverse of the other two. This is label switching: the likelihood is symmetric in the numbering of states, and the model itself does not know which state is “calm”. In practice a definite rule (for instance, ordering states by volatility) must name them; in Bayesian mixture models the problem is harder and needs special treatment (Stephens 2000).
More striking is the estimation error. The true durations are 100 days (calm) and 33.3 days (stress), but the estimates are 148.8 and 23.8 days; the mean daily return of the stressed regime is estimated at +0.022 %, the opposite sign to the true −0.08 %. Volatilities are estimated very accurately (0.801 and 1.890), because every day’s return carries information about variance, whereas means and durations can be inferred only from the few regime switches in the sample. The Viterbi path matches the true states on 95.4 % of days, and classifying by smoothed probability above 0.5 matches on 96.0 %. In hindsight the division into regimes is quite reliable, but the parameters describing the regimes are not.
Duration and transition probability
In a Markov chain a regime ends each day with probability , so its duration is geometric:
The framework’s “expected duration = 1/(1 − p_ii)” is this result, and the derivation is correct. But it has two implications that are often overlooked. First, the geometric distribution is memoryless: a regime that has already lasted 100 days has exactly the same distribution of remaining duration as one that has just begun. That is an assumption of the model, not a discovery about reality; if real business cycles or crises become more (or less) likely to end the longer they last, a duration-dependent model is needed. Second, when is close to 1 the duration is extremely sensitive to it: gives an expected duration of 33 days, gives 50. The program below first checks the properties of the geometric distribution and then asks how hard it is to estimate durations from a finite sample in the best case, with the true states known.
Calculation
# Regime durations in a Markov chain: geometric with mean 1/(1 - p_ii), memoryless, and
# hard to estimate from short samples even when the states are KNOWN (the best case).
# ILLUSTRATIVE: daily chain with P(stay calm) 0.99, P(stay stressed) 0.97; quarterly chain
# with the persistence Hamilton (1989) reported for US real GNP, about 0.90 (expansion)
# and 0.75 (recession); 131 quarters as in his post-war sample.
import random, statistics
def chain(T, p_stay, rng, s=0):
z = []
for _ in range(T):
z.append(s)
s = s if rng.random() < p_stay[s] else 1 - s
return z
def spells(z):
out, start = [], 0
for t in range(1, len(z) + 1):
if t == len(z) or z[t] != z[start]:
out.append((z[start], t - start, start > 0 and t < len(z)))
start = t
return out # (state, length, complete = not cut by the sample ends)
def p_hat(z, k):
n = sum(1 for t in range(len(z) - 1) if z[t] == k)
stay = sum(1 for t in range(len(z) - 1) if z[t] == k and z[t + 1] == k)
return stay / n if n else float("nan")
if __name__ == "__main__":
rng = random.Random(2)
z = chain(2_000_000, (0.99, 0.97), rng)
sp = [(s, d) for s, d, ok in spells(z) if ok]
for k, p in ((0, 0.99), (1, 0.97)):
d = [n for s, n in sp if s == k]
long = [n - 100 for n in d if n > 100]
print(f"state {k}: p = {p}: 1/(1-p) = {1 / (1 - p):5.1f}; simulated mean"
f" {statistics.mean(d):5.1f};"
f" P(D > 50) {sum(n > 50 for n in d) / len(d):.3f}"
f" (p^50 {p ** 50:.3f})")
print(f" spells already past day 100: mean further length"
f" {statistics.mean(long):6.1f}")
print("10 years of daily data, states KNOWN, 2,000 samples:")
est, eps = [], []
for _ in range(2000):
z = chain(2500, (0.99, 0.97), rng)
p = p_hat(z, 1)
est.append(1 / (1 - p) if p < 1 else float("inf"))
eps.append(sum(1 for t in range(1, len(z)) if z[t] == 1 and z[t - 1] == 0))
q = statistics.quantiles(est, n=20)
print(f" stress episodes per sample: median {statistics.median(eps):.0f}"
f" (5-95 %: {statistics.quantiles(eps, n=20)[0]:.0f}"
f"-{statistics.quantiles(eps, n=20)[-1]:.0f})")
print(f" estimated stress duration: median {q[9]:.1f} days,"
f" 5-95 % {q[0]:.1f}-{q[-1]:.1f}"
f" (true 33.3)")
print("131 quarters, p(expansion) 0.904, p(recession) 0.755, states KNOWN,"
" 2,000 samples:")
est, rec = [], []
for _ in range(2000):
z = chain(131, (0.904, 0.755), rng)
p = p_hat(z, 1)
if p == p:
est.append(1 / (1 - p) if p < 1 else 131.0)
rec.append(sum(1 for t in range(1, len(z)) if z[t] == 1 and z[t - 1] == 0))
q = statistics.quantiles(est, n=20)
print(f" recessions per sample: median {statistics.median(rec):.0f};"
f" implied recession duration median {q[9]:.1f} quarters,")
print(f" 5-95 % {q[0]:.1f}-{q[-1]:.1f} (true {1 / 0.245:.1f})")
# state 0: p = 0.99: 1/(1-p) = 100.0; simulated mean 99.8; P(D > 50) 0.609 (p^50 0.605)
# spells already past day 100: mean further length 99.0
# state 1: p = 0.97: 1/(1-p) = 33.3; simulated mean 33.1; P(D > 50) 0.217 (p^50 0.218)
# spells already past day 100: mean further length 33.1
# 10 years of daily data, states KNOWN, 2,000 samples:
# stress episodes per sample: median 19 (5-95 %: 13-25)
# estimated stress duration: median 32.7 days, 5-95 % 22.6-48.1 (true 33.3)
# 131 quarters, p(expansion) 0.904, p(recession) 0.755, states KNOWN, 2,000 samples:
# recessions per sample: median 9; implied recession duration median 4.0 quarters,
# 5-95 % 2.5-6.6 (true 4.1)The simulated mean durations (99.8 and 33.1 days) agree with , and agrees with ; regimes that have already passed day 100 go on for a further 99.0 and 33.1 days on average, just as at the start: that is memorylessness. Ten years of daily data contain a median of only 19 stressed episodes (90 % interval 13 to 25), so even with every day’s state known, the 90 % interval of the estimated stress duration runs from 22.6 to 48.1 days. Simulating 131 quarters with Hamilton’s estimated quarterly parameters gives only about 9 recessions per sample, and a 90 % interval for recession duration of 2.5 to 6.6 quarters (true value 4.1). In real applications the states are unknown as well, so the errors can only be larger.
Clear in hindsight, unclear in real time
Calculation
# (1) Regimes are clearer after the fact: filtered P(stress | data to day t) against
# smoothed P(stress | all data), with the TRUE parameters of p01 (the best case).
# (2) How many states? Data from TWO regimes whose daily shocks are fat-tailed (Student t,
# 4 d.f., unit variance); Gaussian HMMs with 1, 2, 3 states are fitted on 2,500 days and
# scored by BIC and by log-likelihood on the next 2,500 days. All data SIMULATED.
import math, random
MU, SD, P = [0.04, -0.08], [0.8, 2.0], [[0.99, 0.01], [0.03, 0.97]]
def simulate(T, seed, df=None):
rng = random.Random(seed); s, x, z = 0, [], []
for _ in range(T):
e = rng.gauss(0, 1)
if df:
e /= math.sqrt(rng.gammavariate(df / 2, 2) / df) * math.sqrt(df / (df - 2))
z.append(s); x.append(MU[s] + SD[s] * e)
s = s if rng.random() < P[s][s] else 1 - s
return x, z
def dens(x, m, s):
return math.exp(-0.5 * ((x - m) / s) ** 2) / (s * 2.5066282746310002)
def fwd_bwd(x, mu, sd, P, pi):
K, T = len(mu), len(x); a, c, B = [], [], []
for t in range(T):
b = [dens(x[t], mu[k], sd[k]) for k in range(K)]; B.append(b)
f = ([pi[k] * b[k] for k in range(K)] if t == 0 else
[b[k] * sum(a[-1][j] * P[j][k] for j in range(K)) for k in range(K)])
s = sum(f); c.append(s); a.append([v / s for v in f])
be = [[1.0] * K for _ in range(T)]
for t in range(T - 2, -1, -1):
be[t] = [sum(P[j][k] * B[t + 1][k] * be[t + 1][k] for k in range(K)) / c[t + 1]
for j in range(K)]
return a, be, c, B
def fit(x, mu, sd, max_iter=300, tol=1e-5):
K, T = len(mu), len(x); P = [[0.95 if j == k else 0.05 / max(K - 1, 1)
for k in range(K)] for j in range(K)]
pi, old = [1 / K] * K, -1e300
for _ in range(max_iter):
a, be, c, B = fwd_bwd(x, mu, sd, P, pi); ll = sum(math.log(v) for v in c)
g = [[a[t][k] * be[t][k] for k in range(K)] for t in range(T)]
xi = [[sum(a[t][j] * P[j][k] * B[t + 1][k] * be[t + 1][k] / c[t + 1]
for t in range(T - 1)) for k in range(K)] for j in range(K)]
P = [[v / sum(row) for v in row] for row in xi]; pi = g[0]
for k in range(K):
w = sum(r[k] for r in g)
mu[k] = sum(r[k] * v for r, v in zip(g, x)) / w
v2 = sum(r[k] * (v - mu[k]) ** 2 for r, v in zip(g, x)) / w
sd[k] = max(0.05, math.sqrt(v2))
if ll - old < tol:
break
old = ll
return mu, sd, P, pi, ll
def loglik(x, mu, sd, P, pi):
return sum(math.log(v) for v in fwd_bwd(x, mu, sd, P, pi)[2])
def lags(z, prob, thr=0.5):
out, t = [], 1
while t < len(z):
if z[t] == 1 and z[t - 1] == 0:
e = t
while e < len(z) and z[e] == 1:
e += 1
if e - t >= 10:
hit = next((u - t for u in range(t, e) if prob[u] > thr), None)
out.append(hit)
t = e
t += 1
return out
if __name__ == "__main__":
x, z = simulate(20000, 7)
a, be, c, _ = fwd_bwd(x, MU, SD, P, [1.0, 0.0])
filt = [r[1] for r in a]; smo = [r[1] * q[1] for r, q in zip(a, be)]
for name, pr in (("filtered", filt), ("smoothed", smo)):
L = lags(z, pr); found = sorted(v for v in L if v is not None)
fa = sum(1 for p, s in zip(pr, z) if p > 0.5 and s == 0) / z.count(0)
print(f"{name:8s}: {len(L)} stress spells >= 10 days;"
f" detected {len(found) / len(L):.0%};"
f" median lag {found[len(found) // 2]} days,")
print(f"{'':10s}90th percentile {found[int(0.9 * len(found))]} days;"
f" P > 0.5 on {fa:.1%} of calm days (false alarms)")
x, _ = simulate(5000, 11, df=4); ins, oos = x[:2500], x[2500:]
print("data: 2 regimes with Student-t(4) shocks; Gaussian HMMs fitted on days 1-2500")
print(" K params log L in-sample BIC log L days 2501-5000 (per day) sd")
starts = {1: ([0.0], [1.0]), 2: ([0.05, -0.05], [0.7, 1.8]),
3: ([0.05, 0.0, -0.1], [0.6, 1.0, 2.2])}
for K, (m0, s0) in starts.items():
mu, sd, Pk, pi, ll = fit(ins, m0[:], s0[:])
npar = 2 * K + K * (K - 1) + (K - 1)
bic = -2 * ll + npar * math.log(len(ins)); lo = loglik(oos, mu, sd, Pk, pi)
sds = ", ".join(f"{v:.2f}" for v in sd)
print(f" {K} {npar:5d} {ll:10.1f} {bic:9.1f} {lo:10.1f}"
f" ({lo / len(oos):+.4f})"
f" {sds}")
# filtered: 112 stress spells >= 10 days; detected 99%; median lag 2 days,
# 90th percentile 7 days; P > 0.5 on 3.5% of calm days (false alarms)
# smoothed: 112 stress spells >= 10 days; detected 99%; median lag 0 days,
# 90th percentile 3 days; P > 0.5 on 1.7% of calm days (false alarms)
# data: 2 regimes with Student-t(4) shocks; Gaussian HMMs fitted on days 1-2500
# K params log L in-sample BIC log L days 2501-5000 (per day) sd
# 1 2 -3688.8 7393.3 -3835.9 (-1.5343) 1.06
# 2 7 -3300.3 6655.3 -3333.5 (-1.3334) 0.65, 1.98
# 3 14 -3262.7 6634.9 -3310.6 (-1.3243) 0.53, 1.18, 1.96The first part of the program computes filtered and smoothed probabilities on 20,000 simulated days using the true parameters (the best case; in reality the parameters must also be estimated). For the 112 stressed episodes lasting at least 10 days, the median delay before the filtered probability exceeds 0.5 is 2 days and the 90th percentile 7 days; for the smoothed probability, 0 and 3 days. On calm days the filtered probability gives false alarms (above 0.5) 3.5 % of the time, the smoothed probability 1.7 %. Figure 1 shows the first 1,000 days: the smoothed probability (dashed) rises and falls cleanly at both ends of each stressed period, while the filtered probability (solid) catches up only after a regime has begun and shows scattered spikes in calm periods.
This is the mathematical meaning of “regimes are often clear only in hindsight”: the smoothed probability is clean because it borrows data from the future. Any rule that uses an HMM to make decisions in real time can use only the filtered probability; a backtest that uses smoothed probabilities systematically overstates the rule’s effect, because at every point it peeks at the future.
How many states? In the second part of the program the data come from two regimes, but the shocks within each regime are Student t with 4 degrees of freedom (fat-tailed); Gaussian HMMs with 1, 2 and 3 states are then fitted. The Bayesian information criterion (BIC, smaller is better) falls from 7393.3 to 6655.3 and then to 6634.9, and out-of-sample log-likelihood is also best with three states (−1.3243 against −1.3334 per day). Both the information criterion and the out-of-sample test thus “select” a third regime that does not exist in the data-generating process: the three-state model uses an intermediate state with volatility 1.18 to fit the fat tails within each regime. The number of states is not “discovered” in the data; it is jointly determined by the model’s specification (the shape of the within-regime distribution) and the data, and interpreting three states as three economically different market environments needs reasons from outside the model.
Note
Limits in summary. (1) The number of states and the within-regime distribution are modelling choices; change them and the number and meaning of the regimes change too. (2) Labels must be assigned by an explicit rule. (3) Baum–Welch guarantees only a local maximum, so several starts should be tried. (4) A good in-sample fit is not a good out-of-sample forecast; uncertainty about parameters such as durations and means is often large. (5) In real time only the filtered probability can be used, and it lags the smoothed probability and gives false alarms. (6) The Markov assumption makes durations memoryless; this is an assumption, not a fact.
Hypothesis · not established
The author’s hypothesis (not established): a rebalancing rule triggered by the HMM’s filtered probability (for example, halving the share of risky assets when the stress probability exceeds 0.5) earns a higher out-of-sample risk-adjusted return than fixed-proportion rebalancing. What would show it wrong: with an out-of-sample window and transaction-cost assumptions locked in advance, a confidence interval for the difference in Sharpe ratio or maximum drawdown between the two rules that includes zero or favours fixed proportions. Chapter 8 explains that such a test needs very many years of data, which is why the book chooses a different, better-powered test.
Chapter summary
A hidden Markov model describes the market as switching between a few unseen regimes. The forward–backward algorithm gives filtered and smoothed probabilities, Baum–Welch estimates the parameters and Viterbi gives the most likely path of states. Hamilton’s original 1989 model switched the mean of GNP growth; volatility regimes are an extension. The expected duration holds under the memoryless assumption, but durations estimated from ten years of data are very imprecise. After the fact, smoothed probabilities divide history into regimes quite clearly; in real time, filtered probabilities lag and give false alarms, and when returns are fat-tailed an information criterion selects a regime that does not exist. Regime identification is therefore an assumption-dependent probabilistic division, not a direct observation of the state of the market.
References
Fan, Y. (2026). Macro-Liquidity Regime Switching, Extreme Tail Dependence and Barbell Asset Allocation: Vine Copulas, Hidden Markov Models and Sovereign Buffer Dynamics [宏观流动性体制转换、极值尾部相依与杠铃资产配置] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23828049 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).