What does Regimes, Tail Dependence and Barbells conclude, in numbers, about “Hidden Markov Models and Volatility Regimes”?
Answer: A hidden Markov model describes the market as switching between a few unseen regimes. The forward–backward algorithm gives filtered and smoothed probabilities, Baum–Welch estimates the parameters and Viterbi gives the most likely path of states. Hamilton's original 1989 model switched the mean of GNP growth; volatility regimes are an extension. The expected duration \(1/(1-p_{ii})\) holds under the memoryless assumption, but durations estimated from ten years of data are very imprecise. After the fact, smoothed probabilities divide history into regimes quite clearly; in real time, filtered probabilities lag and give false alarms, and when returns are fat-tailed an information criterion selects a regime that does not exist. Regime identification is therefore an assumption-dependent probabilistic division, not a direct observation of the state of the market.
Scope of the numbers: When a crisis comes, do assets fall together? How large must reserves and safety cushions be? This book presents mathematical models for teaching and research and is not investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's models and hypotheses; all data in the book are fictional or simulated.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
simulated 2500 days; true share of stress days 0.142 start A (calm first): log L -3360.1 after 22 iterations state 0: mean +0.030 sd 0.801 P(stay) 0.9933 duration 148.8 days state 1: mean +0.022 sd 1.890 P(stay) 0.9579 duration 23.8 days start B (stress first): log L -3360.1 after 22 iterations state 0: mean +0.022 sd 1.890 P(stay) 0.9579 duration 23.8 days state 1: mean +0.030 sd 0.801 P(stay) 0.9933 duration 148.8 days start C (nearly equal): log L -3360.1 after 20 iterations state 0: mean +0.030 sd 0.801 P(stay) 0.9933 duration 148.8 days state 1: mean +0.022 sd 1.890 P(stay) 0.9579 duration 23.8 days Viterbi path agrees with the true states on 95.4% of days; smoothed P > 0.5 on 96.0% true durations: calm 100 days, stress 33.3 days
Key formula
Show the full program
# Two-state Gaussian HMM for daily returns (%), fitted by Baum-Welch (EM) with scaled
# forward-backward recursions and decoded by Viterbi. SIMULATED, ILLUSTRATIVE: calm
# (mean 0.04, sd 0.8), stress (mean -0.08, sd 2.0); P(stay) 0.99 and 0.97; 2,500 days.
import math, random
TRUE = dict(mu=[0.04, -0.08], sd=[0.8, 2.0], P=[[0.99, 0.01], [0.03, 0.97]], pi=[1, 0])
def simulate(T=2500, seed=1989, par=TRUE):
rng = random.Random(seed); s = 0 if rng.random() < par["pi"][0] else 1; x, z = [], []
for _ in range(T):
z.append(s); x.append(rng.gauss(par["mu"][s], par["sd"][s]))
s = s if rng.random() < par["P"][s][s] else 1 - s
return x, z
def dens(x, mu, sd):
return math.exp(-0.5 * ((x - mu) / sd) ** 2) / (sd * math.sqrt(2 * math.pi))
def forward(x, mu, sd, P, pi): # a[t][k] = P(state k | x_1..x_t)
K, a, c = len(mu), [], []
for t, xt in enumerate(x):
b = [dens(xt, mu[k], sd[k]) for k in range(K)]
f = ([pi[k] * b[k] for k in range(K)] if t == 0 else
[b[k] * sum(a[-1][j] * P[j][k] for j in range(K)) for k in range(K)])
s = sum(f); c.append(s); a.append([v / s for v in f])
return a, c # log L = sum of log c
def backward(x, mu, sd, P, c):
K, T = len(mu), len(x); be = [[1.0] * K for _ in range(T)]
for t in range(T - 2, -1, -1):
b = [dens(x[t + 1], mu[k], sd[k]) for k in range(K)]
be[t] = [sum(P[j][k] * b[k] * be[t + 1][k] for k in range(K)) / c[t + 1]
for j in range(K)]
return be
def fit(x, mu, sd, P, pi, tol=1e-7, max_iter=500):
K, T, old = len(mu), len(x), -1e300
for it in range(1, max_iter + 1):
a, c = forward(x, mu, sd, P, pi); be = backward(x, mu, sd, P, c)
ll = sum(math.log(v) for v in c)
g = [[a[t][k] * be[t][k] for k in range(K)] for t in range(T)]
xi = [[0.0] * K for _ in range(K)]
for t in range(T - 1):
b = [dens(x[t + 1], mu[k], sd[k]) for k in range(K)]
for j in range(K):
for k in range(K):
xi[j][k] += a[t][j] * P[j][k] * b[k] * be[t + 1][k] / c[t + 1]
P = [[xi[j][k] / sum(xi[j]) for k in range(K)] for j in range(K)]; pi = g[0][:]
for k in range(K):
w = sum(r[k] for r in g); mu[k] = sum(r[k] * v for r, v in zip(g, x)) / w
sd[k] = max(1e-3, math.sqrt(sum(r[k] * (v - mu[k]) ** 2
for r, v in zip(g, x)) / w))
if ll - old < tol:
break
old = ll
return dict(mu=mu, sd=sd, P=P, pi=pi, ll=ll, iters=it, gamma=g)
def viterbi(x, mu, sd, P, pi):
K = len(mu); lg = lambda v: math.log(max(v, 1e-300)); back = []
d = [lg(pi[k]) + lg(dens(x[0], mu[k], sd[k])) for k in range(K)]
for xt in x[1:]:
best = [max(range(K), key=lambda j: d[j] + lg(P[j][k])) for k in range(K)]
d = [d[best[k]] + lg(P[best[k]][k]) + lg(dens(xt, mu[k], sd[k]))
for k in range(K)]
back.append(best)
s = [max(range(K), key=lambda k: d[k])]
for best in reversed(back):
s.append(best[s[-1]])
return s[::-1]
if __name__ == "__main__":
x, z = simulate(); P0 = [[0.95, 0.05], [0.05, 0.95]]
print(f"simulated {len(x)} days; true share of stress days {sum(z) / len(z):.3f}")
for name, m0, s0 in (("A (calm first)", [0.0, 0.0], [0.5, 1.5]),
("B (stress first)", [0.0, 0.0], [1.5, 0.5]),
("C (nearly equal)", [0.01, -0.01], [1.0, 1.01])):
r = fit(x, m0, s0, P0, [0.5, 0.5])
print(f"start {name}: log L {r['ll']:.1f} after {r['iters']} iterations")
for k in range(2):
p = r["P"][k][k]
print(f" state {k}: mean {r['mu'][k]:+.3f} sd {r['sd'][k]:.3f}"
f" P(stay) {p:.4f} duration {1 / (1 - p):6.1f} days")
r = fit(x, [0.0, 0.0], [0.5, 1.5], P0, [0.5, 0.5])
path = viterbi(x, r["mu"], r["sd"], r["P"], r["pi"])
smooth = [1 if g[1] > 0.5 else 0 for g in r["gamma"]]
agree = lambda s: sum(p == q for p, q in zip(s, z)) / len(z)
print(f"Viterbi path agrees with the true states on {agree(path):.1%} of days;"
f" smoothed P > 0.5 on {agree(smooth):.1%}")
print("true durations: calm 100 days, stress 33.3 days")Single source and citation
Fan, Y. (2026). Macro-Liquidity Regime Switching, Extreme Tail Dependence and Barbell Asset Allocation (Version 1.0.0), Chapter 1: Hidden Markov Models and Volatility Regimes. 5D Library. https://doi.org/10.5281/zenodo.23828049
DOI: 10.5281/zenodo.23828049 · Read the monograph · 5DL-B-040 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026040,
author = {Fan, Yafu},
title = {Macro-Liquidity Regime Switching, Extreme Tail Dependence and Barbell Asset Allocation},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23828049},
url = {https://doi.org/10.5281/zenodo.23828049}
}Other chapter answers in this book
- Chapter 2 · Extreme Value Theory: From Tail Index to Expected Shortfall
- Chapter 3 · Copulas and Tail Dependence
- Chapter 4 · Correlation in Crises and Vine Copulas
- Chapter 5 · Barbell Allocation and Stochastic Dominance
- Chapter 6 · Sovereign Buffers and Liquidity Regimes
- Chapter 7 · Reading the Classics: the "Nine-Year Store" and the Ever-Normal Granary
- Chapter 8 · Practice and a Pre-Registered Test