What does Regimes, Tail Dependence and Barbells conclude, in numbers, about “Hidden Markov Models and Volatility Regimes”?

Answer: A hidden Markov model describes the market as switching between a few unseen regimes. The forward–backward algorithm gives filtered and smoothed probabilities, Baum–Welch estimates the parameters and Viterbi gives the most likely path of states. Hamilton's original 1989 model switched the mean of GNP growth; volatility regimes are an extension. The expected duration \(1/(1-p_{ii})\) holds under the memoryless assumption, but durations estimated from ten years of data are very imprecise. After the fact, smoothed probabilities divide history into regimes quite clearly; in real time, filtered probabilities lag and give false alarms, and when returns are fat-tailed an information criterion selects a regime that does not exist. Regime identification is therefore an assumption-dependent probabilistic division, not a direct observation of the state of the market.

Scope of the numbers: When a crisis comes, do assets fall together? How large must reserves and safety cushions be? This book presents mathematical models for teaching and research and is not investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's models and hypotheses; all data in the book are fictional or simulated.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

simulated 2500 days; true share of stress days 0.142
start A (calm first): log L -3360.1 after 22 iterations
   state 0: mean +0.030  sd 0.801  P(stay) 0.9933  duration  148.8 days
   state 1: mean +0.022  sd 1.890  P(stay) 0.9579  duration   23.8 days
start B (stress first): log L -3360.1 after 22 iterations
   state 0: mean +0.022  sd 1.890  P(stay) 0.9579  duration   23.8 days
   state 1: mean +0.030  sd 0.801  P(stay) 0.9933  duration  148.8 days
start C (nearly equal): log L -3360.1 after 20 iterations
   state 0: mean +0.030  sd 0.801  P(stay) 0.9933  duration  148.8 days
   state 1: mean +0.022  sd 1.890  P(stay) 0.9579  duration   23.8 days
Viterbi path agrees with the true states on 95.4% of days; smoothed P > 0.5 on 96.0%
true durations: calm 100 days, stress 33.3 days

Key formula

\[\alpha_t(k) \propto b_k(x_t) \sum_j \alpha_{t-1}(j)\,p_{jk}, \qquad b_k(x) = \varphi\!\left(\frac{x - \mu_k}{\sigma_k}\right)\Big/\sigma_k\]
Show the full program
# Two-state Gaussian HMM for daily returns (%), fitted by Baum-Welch (EM) with scaled
# forward-backward recursions and decoded by Viterbi. SIMULATED, ILLUSTRATIVE: calm
# (mean 0.04, sd 0.8), stress (mean -0.08, sd 2.0); P(stay) 0.99 and 0.97; 2,500 days.
import math, random

TRUE = dict(mu=[0.04, -0.08], sd=[0.8, 2.0], P=[[0.99, 0.01], [0.03, 0.97]], pi=[1, 0])

def simulate(T=2500, seed=1989, par=TRUE):
    rng = random.Random(seed); s = 0 if rng.random() < par["pi"][0] else 1; x, z = [], []
    for _ in range(T):
        z.append(s); x.append(rng.gauss(par["mu"][s], par["sd"][s]))
        s = s if rng.random() < par["P"][s][s] else 1 - s
    return x, z

def dens(x, mu, sd):
    return math.exp(-0.5 * ((x - mu) / sd) ** 2) / (sd * math.sqrt(2 * math.pi))

def forward(x, mu, sd, P, pi):               # a[t][k] = P(state k | x_1..x_t)
    K, a, c = len(mu), [], []
    for t, xt in enumerate(x):
        b = [dens(xt, mu[k], sd[k]) for k in range(K)]
        f = ([pi[k] * b[k] for k in range(K)] if t == 0 else
             [b[k] * sum(a[-1][j] * P[j][k] for j in range(K)) for k in range(K)])
        s = sum(f); c.append(s); a.append([v / s for v in f])
    return a, c                              # log L = sum of log c

def backward(x, mu, sd, P, c):
    K, T = len(mu), len(x); be = [[1.0] * K for _ in range(T)]
    for t in range(T - 2, -1, -1):
        b = [dens(x[t + 1], mu[k], sd[k]) for k in range(K)]
        be[t] = [sum(P[j][k] * b[k] * be[t + 1][k] for k in range(K)) / c[t + 1]
                 for j in range(K)]
    return be

def fit(x, mu, sd, P, pi, tol=1e-7, max_iter=500):
    K, T, old = len(mu), len(x), -1e300
    for it in range(1, max_iter + 1):
        a, c = forward(x, mu, sd, P, pi); be = backward(x, mu, sd, P, c)
        ll = sum(math.log(v) for v in c)
        g = [[a[t][k] * be[t][k] for k in range(K)] for t in range(T)]
        xi = [[0.0] * K for _ in range(K)]
        for t in range(T - 1):
            b = [dens(x[t + 1], mu[k], sd[k]) for k in range(K)]
            for j in range(K):
                for k in range(K):
                    xi[j][k] += a[t][j] * P[j][k] * b[k] * be[t + 1][k] / c[t + 1]
        P = [[xi[j][k] / sum(xi[j]) for k in range(K)] for j in range(K)]; pi = g[0][:]
        for k in range(K):
            w = sum(r[k] for r in g); mu[k] = sum(r[k] * v for r, v in zip(g, x)) / w
            sd[k] = max(1e-3, math.sqrt(sum(r[k] * (v - mu[k]) ** 2
                                            for r, v in zip(g, x)) / w))
        if ll - old < tol:
            break
        old = ll
    return dict(mu=mu, sd=sd, P=P, pi=pi, ll=ll, iters=it, gamma=g)

def viterbi(x, mu, sd, P, pi):
    K = len(mu); lg = lambda v: math.log(max(v, 1e-300)); back = []
    d = [lg(pi[k]) + lg(dens(x[0], mu[k], sd[k])) for k in range(K)]
    for xt in x[1:]:
        best = [max(range(K), key=lambda j: d[j] + lg(P[j][k])) for k in range(K)]
        d = [d[best[k]] + lg(P[best[k]][k]) + lg(dens(xt, mu[k], sd[k]))
             for k in range(K)]
        back.append(best)
    s = [max(range(K), key=lambda k: d[k])]
    for best in reversed(back):
        s.append(best[s[-1]])
    return s[::-1]

if __name__ == "__main__":
    x, z = simulate(); P0 = [[0.95, 0.05], [0.05, 0.95]]
    print(f"simulated {len(x)} days; true share of stress days {sum(z) / len(z):.3f}")
    for name, m0, s0 in (("A (calm first)", [0.0, 0.0], [0.5, 1.5]),
                         ("B (stress first)", [0.0, 0.0], [1.5, 0.5]),
                         ("C (nearly equal)", [0.01, -0.01], [1.0, 1.01])):
        r = fit(x, m0, s0, P0, [0.5, 0.5])
        print(f"start {name}: log L {r['ll']:.1f} after {r['iters']} iterations")
        for k in range(2):
            p = r["P"][k][k]
            print(f"   state {k}: mean {r['mu'][k]:+.3f}  sd {r['sd'][k]:.3f}"
                  f"  P(stay) {p:.4f}  duration {1 / (1 - p):6.1f} days")
    r = fit(x, [0.0, 0.0], [0.5, 1.5], P0, [0.5, 0.5])
    path = viterbi(x, r["mu"], r["sd"], r["P"], r["pi"])
    smooth = [1 if g[1] > 0.5 else 0 for g in r["gamma"]]
    agree = lambda s: sum(p == q for p, q in zip(s, z)) / len(z)
    print(f"Viterbi path agrees with the true states on {agree(path):.1%} of days;"
          f" smoothed P > 0.5 on {agree(smooth):.1%}")
    print("true durations: calm 100 days, stress 33.3 days")

Single source and citation

Fan, Y. (2026). Macro-Liquidity Regime Switching, Extreme Tail Dependence and Barbell Asset Allocation (Version 1.0.0), Chapter 1: Hidden Markov Models and Volatility Regimes. 5D Library. https://doi.org/10.5281/zenodo.23828049

DOI: 10.5281/zenodo.23828049 · Read the monograph · 5DL-B-040 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026040,
  author    = {Fan, Yafu},
  title     = {Macro-Liquidity Regime Switching, Extreme Tail Dependence and Barbell Asset Allocation},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.23828049},
  url       = {https://doi.org/10.5281/zenodo.23828049}
}

Other chapter answers in this book