5D Library › High-Frequency Limit Order Book Microstructure, Hawkes Self-Exciting Point Processes and Adverse-Selection Flow Toxicity · 中文
Chapter 3 Why Orders Arrive in Clusters: Hawkes Processes, the Signature Plot and the Epps Effect
What the Poisson process misses
The simplest model of arrivals is the Poisson process: orders arrive independently at a constant intensity, counts in non-overlapping intervals are independent, and the variance of a count equals its mean. High-frequency data almost never look like this: one trade is often quickly followed by another, and a burst of trading is followed by a relatively quiet spell. The standard tool for this “clustering” is the self-exciting point process introduced by Hawkes in 1971 (Hawkes 1971). The survey of Bacry, Mastromatteo and Muzy describes its many uses in finance: estimating volatility from transaction data, assessing market stability, modelling the contagion of systemic risk, designing optimal execution strategies and describing the dynamics of the whole order book (Bacry et al. 2015).
Definition, branching ratio and stationarity
The conditional intensity of a Hawkes process is
so every past event raises the current intensity a little, by an amount that decays with time. The intensity in the author’s framework, , is exactly the exponential-kernel case. In several dimensions, the intensity of events of type is excited by past events of all types (“mutual excitation”).
Hawkes and Oakes gave its branching (cluster) representation in 1974 (Hawkes and Oakes 1974): “immigrant” events arrive with Poisson intensity ; every event independently produces a number of “children”, Poisson-distributed with mean , each separated from its parent by a time with density . For the exponential kernel, and the delays are exponential with rate . is called the branching ratio: the average number of events directly triggered by one event.
Several results follow at once from the branching representation. The mean size of a cluster is , finite only if ; the mean intensity in the stationary state is ; and in windows much longer than the ratio of the variance of the count to its mean (the Fano factor) tends to . So the stationarity condition in the author’s framework, , is correct; when cluster sizes have no finite expectation and the process “explodes”.
Calculation
# Exponential Hawkes process lambda(t) = mu + sum_{t_i < t} alpha exp(-beta (t - t_i)),
# simulated through its cluster (branching) representation (Hawkes & Oakes 1974):
# immigrants arrive as Poisson(mu); every event has Poisson(eta) children
# (eta = alpha/beta), each after an Exp(beta) delay. Checks: mean rate mu/(1 - eta),
# cluster size 1/(1 - eta), Fano factor of counts in long windows -> 1/(1 - eta)^2;
# maximum likelihood; and a time-rescaling (residual) check of a Poisson fit against
# a Hawkes fit.
import math, random, statistics
def poisson(rng, m):
k, p, L = 0, rng.random(), math.exp(-m)
while p > L:
k += 1; p *= rng.random()
return k
def simulate(rng, mu, eta, beta, T):
ev, sizes, t = [], [], rng.expovariate(mu)
while t < T:
cluster, gen = [t], [t]
while gen:
nxt = []
for s in gen:
for _ in range(poisson(rng, eta)):
c = s + rng.expovariate(beta)
if c < T:
nxt.append(c)
cluster += nxt; gen = nxt
ev += cluster; sizes.append(len(cluster)); t += rng.expovariate(mu)
return sorted(ev), sizes
def loglik(th, ev, T):
mu, alpha, beta = th
if min(th) <= 0 or alpha >= beta:
return -1e18
ll, r, prev = 0.0, 0.0, None
for t in ev: # r = sum of exp(-beta (t - t_j)), t_j < t
if prev is not None:
r = (r + 1) * math.exp(-beta * (t - prev))
ll += math.log(mu + alpha * r); prev = t
comp = mu * T + alpha / beta * sum(1 - math.exp(-beta * (T - t)) for t in ev)
return ll - comp
def nelder_mead(f, x0, it=400):
n = len(x0)
pts = [list(x0)] + [[x * (1.3 if i == j else 1) for j, x in enumerate(x0)]
for i in range(n)]
for _ in range(it):
pts.sort(key=f); c = [sum(p[i] for p in pts[:-1]) / n for i in range(n)]
w = pts[-1]; r = [c[i] + (c[i] - w[i]) for i in range(n)]
if f(r) < f(pts[0]):
e = [c[i] + 2 * (c[i] - w[i]) for i in range(n)]
pts[-1] = e if f(e) < f(r) else r
elif f(r) < f(pts[-2]):
pts[-1] = r
else:
k = [c[i] + 0.5 * (w[i] - c[i]) for i in range(n)]
if f(k) < f(w):
pts[-1] = k
else:
pts = [pts[0]] + [[pts[0][i] + 0.5 * (p[i] - pts[0][i]) for i in range(n)]
for p in pts[1:]]
return min(pts, key=f)
def ks_exp(x): # KS distance to Exp(1)
x = sorted(x); n = len(x)
return max(max((i + 1) / n - (1 - math.exp(-v)), (1 - math.exp(-v)) - i / n)
for i, v in enumerate(x))
def rescaled(ev, mu, alpha, beta):
out, r, prev, Lprev = [], 0.0, None, 0.0
for t in ev: # compensator increments between events
if prev is None:
L = mu * t
else:
dec = 1 - math.exp(-beta * (t - prev))
L = Lprev + mu * (t - prev) + alpha / beta * (r + 1) * dec
r = (r + 1) * math.exp(-beta * (t - prev))
if prev is not None:
out.append(L - Lprev)
Lprev, prev = L, t
return out
if __name__ == "__main__":
rng = random.Random(1971); mu, beta, T, W = 0.5, 1.0, 40_000.0, 200.0
print("mu 0.5, beta 1.0, T 40,000 rate cluster size Fano (W=200)")
print(" eta simulated theory simulated theory simulated theory")
for eta in (0.3, 0.6, 0.8, 0.9):
ev, sz = simulate(rng, mu, eta, beta, T)
cnt = [0] * int(T / W)
for t in ev:
cnt[int(t / W)] += 1
fano = statistics.variance(cnt) / statistics.fmean(cnt)
print(f" {eta:.1f} {len(ev) / T:7.3f} {mu / (1 - eta):6.3f}"
f" {statistics.fmean(sz):7.2f} {1 / (1 - eta):5.2f}"
f" {fano:8.2f} {1 / (1 - eta) ** 2:6.2f}")
ev, _ = simulate(rng, 0.5, 0.8, 1.0, 5000.0)
th = nelder_mead(lambda p: -loglik(p, ev, 5000.0), [0.3, 0.3, 0.6])
print(f"MLE on T = 5,000 ({len(ev)} events; truth mu 0.5, alpha 0.8, beta 1.0):")
print(f" mu {th[0]:.3f}, alpha {th[1]:.3f}, beta {th[2]:.3f},"
f" branching ratio {th[1] / th[2]:.3f}")
lam = len(ev) / 5000.0
gaps = [lam * (b - a) for a, b in zip(ev, ev[1:])]
print(f"time-rescaling check (KS distance to Exp(1); 5 % critical value"
f" {1.358 / math.sqrt(len(gaps)):.3f}):")
ks_h = ks_exp(rescaled(ev, *th))
print(f" Poisson fit {ks_exp(gaps):.3f} Hawkes fit {ks_h:.3f}")
# mu 0.5, beta 1.0, T 40,000 rate cluster size Fano (W=200)
# eta simulated theory simulated theory simulated theory
# 0.3 0.713 0.714 1.43 1.43 1.90 2.04
# 0.6 1.270 1.250 2.51 2.50 5.88 6.25
# 0.8 2.529 2.500 5.04 5.00 22.24 25.00
# 0.9 5.102 5.000 10.19 10.00 97.05 100.00
# MLE on T = 5,000 (12292 events; truth mu 0.5, alpha 0.8, beta 1.0):
# mu 0.489, alpha 0.811, beta 1.013, branching ratio 0.801
# time-rescaling check (KS distance to Exp(1); 5 % critical value 0.012):
# Poisson fit 0.163 Hawkes fit 0.006The program simulates samples of length 40,000 with , and four branching ratios, using the branching representation. The mean intensity and cluster size agree with theory (for , a simulated intensity of 5.102 against 5.000 and cluster size 10.19 against 10.00). The Fano factor rises steeply with : 22.24 at (limit 25) and 97.05 at (limit 100). The simulated values are all slightly below the limits because the window length of 200 is finite and clusters at the window edges are cut. This has a practical consequence: near criticality the number of events in an interval can far exceed what a Poisson model would expect, even though the average arrival rate is unchanged.
The second half of the program fits a sample with (12,292 events) by maximum likelihood, obtaining , , and a branching ratio of 0.801. Whether the fit is adequate can be checked by time rescaling: if the model is right, the compensator increments between successive events, , should be independent exponential variables with mean 1 (Ogata 1988). The Kolmogorov–Smirnov distance is 0.163 for the Poisson model, far beyond the 5 % critical value of 0.012, and 0.006 for the Hawkes model, which passes. Book 39 discussed Ogata thinning and maximum-likelihood estimation in detail; they are not repeated here.
Evidence
Book 39 discussed the branching ratio of real markets; here it is only recapped. Filimonov and Sornette analysed E-mini S&P 500 futures with an exponential-kernel Hawkes model and estimated that the share of price changes driven by exogenous information fell from about 70 % in 1998 to less than 30 % after 2007, that is, the endogenous (self-excited) part grew (Filimonov and Sornette 2012). Hardiman, Bercot and Bouchaud used a power-law kernel on the same contract over 1998–2011 and found that the kernel integrated to about 1 in every period, concluding that the market “has always been” close to criticality (Hardiman et al. 2013). Filimonov and Sornette then pointed out that non-stationary baseline intensities, outliers and the choice of kernel shape all bias branching-ratio estimates towards 1 (Filimonov and Sornette 2015). The program in Book 39 also showed that data with no self-excitation, only switches in the baseline, can be fitted with a branching ratio of 0.92. “The market is at criticality” is therefore a disputed estimate, not a settled finding.
Microstructure noise and the signature plot
When realized variance (the sum of squared returns over intervals) is computed at higher and higher frequencies, it often rises with the sampling frequency instead of settling at a stable value. A plot of “realized variance divided by the sampling interval” against the sampling interval is the signature plot. Zhang, Mykland and Aït-Sahalia explained the phenomenon by “microstructure noise” and proposed an estimator using two time scales to remove the noise bias (Zhang et al. 2005).
Evidence
Bacry, Delattre, Hoffmann and Muzy proposed modelling the noise directly with mutually exciting point processes (Bacry et al. 2013): the price is the number of up-ticks minus the number of down-ticks, , where an up-tick raises the intensity of subsequent down-ticks and a down-tick that of subsequent up-ticks. This cross-excitation produces mean reversion at short time scales. For an exponential kernel , with , they obtained the signature plot in closed form,
equal to for small and tending to the diffusive variance for large . They extended the model to two assets in which moves of the two assets in the same direction excite each other, reproducing the Epps effect within the same framework.
The Epps effect: correlations vanish at high frequency
Epps found in 1979 that the correlation between stock returns falls as the interval over which returns are measured is shortened: two stocks that look highly correlated in daily returns look almost uncorrelated in returns over a few minutes (Epps 1979). This is an empirical phenomenon with at least two different explanations. The first is asynchronous trading: the two stocks trade at different moments, and taking “the last traded price” on a fixed grid counts information that one stock has already reflected, and the other has not yet had a trade to reflect, as uncorrelated. The second is microstructure dynamics: in the mutually exciting model above, for example, each asset’s own mean reversion “cancels out” common moves at short time scales. The author’s framework calls the Epps effect a “microstructure noise covariance” and links it only to Hawkes processes; more accurately, mutually exciting Hawkes processes are one model that explains the Epps effect, asynchronous trading is another, and each can produce the phenomenon on its own.
The program below has three parts. A simulates the signature plot with a mutually exciting process and compares it with the closed form; B computes correlations at different sampling intervals in the two-asset mutually exciting model; C uses no point-process structure at all: two Brownian motions with correlation 0.6 are observed at independent Poisson times (1 and 0.2 observations per unit time), and the previous-tick method is compared with the Hayashi–Yoshida estimator (Hayashi and Yoshida 2005), which sums products of returns over all overlapping pairs of trade intervals of the two assets and needs no synchronisation on a grid.
Calculation
# Microstructure noise and the Epps effect from point processes.
# A: Bacry et al. (2013): price X = N+ - N-, where an up-tick excites down-ticks and vice
# versa (kernel alpha exp(-beta t), norm G = alpha/beta). Their signature plot:
# C(tau) = L (k^2 + (1 - k^2)(1 - exp(-g tau))/(g tau)), L = 2mu/(1 - G),
# k = 1/(1 + G), g = alpha + beta. Compared with a simulation (cluster representation).
# B: two assets; same-direction ticks of the two assets excite each other (norm G13), plus
# each asset's own mean reversion (norm G12). Correlation of increments against scale.
# C: no point-process structure at all: two Brownian motions (rho 0.6) observed at
# independent Poisson times; previous-tick correlation against Hayashi-Yoshida (2005).
import bisect, math, random, statistics
def poisson(rng, m):
k, p, L = 0, rng.random(), math.exp(-m)
while p > L:
k += 1; p *= rng.random()
return k
def branching(rng, mu, G, beta, T):
"""G[k][j]: mean number of type-k children of a type-j event; returns (time, type)."""
n, ev = len(G), []
for j in range(n):
t = rng.expovariate(mu)
while t < T:
gen = [(t, j)]
while gen:
ev += gen; nxt = []
for s, a in gen:
for k in range(n):
for _ in range(poisson(rng, G[k][a])):
c = s + rng.expovariate(beta)
if c < T:
nxt.append((c, k))
gen = nxt
t += rng.expovariate(mu)
return sorted(ev)
def increments(ev, sign, tau, T):
"""increments of X = sum of signs over consecutive windows of length tau"""
ts = [e[0] for e in ev]; cum = [0]
for e in ev:
cum.append(cum[-1] + sign.get(e[1], 0))
x = [cum[bisect.bisect_right(ts, m * tau)] for m in range(int(T / tau) + 1)]
return [b - a for a, b in zip(x, x[1:])]
def incr(path, step):
return [b - a for a, b in zip(path[::step], path[step::step])]
if __name__ == "__main__":
rng = random.Random(2013)
mu, alpha, beta, T = 0.5, 0.6, 2.0, 400_000.0
G = alpha / beta; L = 2 * mu / (1 - G); kap = 1 / (1 + G); g = alpha + beta
ev = branching(rng, mu, [[0, G], [G, 0]], beta, T)
print(f"A signature plot, mu {mu}, alpha {alpha}, beta {beta} (norm {G}):"
f" {len(ev):,} ticks")
print(" tau 0.05 0.25 1 5 25 100")
sim = [statistics.fmean(d * d for d in increments(ev, {0: 1, 1: -1}, t, T)) / t
for t in (0.05, 0.25, 1, 5, 25, 100)]
th = [L * (kap ** 2 + (1 - kap ** 2) * (1 - math.exp(-g * t)) / (g * t))
for t in (0.05, 0.25, 1, 5, 25, 100)]
print(" simulated " + " ".join(f"{v:6.3f}" for v in sim))
print(" formula " + " ".join(f"{v:6.3f}" for v in th))
G12, G13, T2 = 0.3, 0.5, 60_000.0 # types: 1+, 1-, 2+, 2-
M = [[0, G12, G13, 0], [G12, 0, 0, G13], [G13, 0, 0, G12], [0, G13, G12, 0]]
ev = branching(rng, 0.5, M, 2.0, T2)
lim = 2 * G13 * (1 + G12) / (1 + G13 ** 2 + 2 * G12 + G12 ** 2)
print(f"B Epps effect, G12 {G12}, G13 {G13}: corr of increments (limit {lim:.3f})")
row = []
for t in (0.05, 0.5, 2, 10, 50):
a = increments(ev, {0: 1, 1: -1}, t, T2)
b = increments(ev, {2: 1, 3: -1}, t, T2)
row.append(statistics.correlation(a, b))
print(" tau 0.05 0.5 2 10 50")
print(" corr " + " ".join(f"{v:6.3f}" for v in row))
T3, rho = 200_000.0, 0.6
obs, t = [], 0.0 # observation times
for who, rate in ((0, 1.0), (1, 0.2)):
t = rng.expovariate(rate)
while t < T3:
obs.append((t, who)); t += rng.expovariate(rate)
obs.sort(); w1 = w2 = last = 0.0; tA, pA, tB, pB = [], [], [], []
for s_, who in obs: # exact Gaussian increments
z1, z2, d = rng.gauss(0, 1), rng.gauss(0, 1), math.sqrt(s_ - last)
w1 += d * z1; w2 += d * (rho * z1 + math.sqrt(1 - rho ** 2) * z2); last = s_
if who == 0:
tA.append(s_); pA.append(w1)
else:
tB.append(s_); pB.append(w2)
def prev_tick(times, vals, grid):
out, i = [], 0
for s in grid:
while i + 1 < len(times) and times[i + 1] <= s:
i += 1
out.append(vals[i])
return out
print(f"C asynchronous trading, true rho {rho}; ticks per unit time 1.0 and 0.2")
print(" tau 1 5 20 100")
row = []
for tau in (1, 5, 20, 100):
grid = [tau * m for m in range(1, int(T3 / tau))]
a, b = prev_tick(tA, pA, grid), prev_tick(tB, pB, grid)
row.append(statistics.correlation(incr(a, 1), incr(b, 1)))
print(" previous tick " + " ".join(f"{v:6.3f}" for v in row))
cov, j = 0.0, 0 # Hayashi-Yoshida estimator
for i in range(1, len(tA)):
a0, a1, dA = tA[i - 1], tA[i], pA[i] - pA[i - 1]
while j + 1 < len(tB) and tB[j + 1] <= a0:
j += 1
k = j
while k + 1 < len(tB) and tB[k] < a1:
if tB[k + 1] > a0:
cov += dA * (pB[k + 1] - pB[k])
k += 1
va = sum((b - a) ** 2 for a, b in zip(pA, pA[1:]))
vb = sum((b - a) ** 2 for a, b in zip(pB, pB[1:]))
print(f" Hayashi-Yoshida (all ticks): {cov / math.sqrt(va * vb):.3f}")
# A signature plot, mu 0.5, alpha 0.6, beta 2.0 (norm 0.3): 569,182 ticks
# tau 0.05 0.25 1 5 25 100
# simulated 1.388 1.273 1.050 0.887 0.868 0.860
# formula 1.392 1.274 1.053 0.890 0.854 0.848
# B Epps effect, G12 0.3, G13 0.5: corr of increments (limit 0.670)
# tau 0.05 0.5 2 10 50
# corr 0.041 0.302 0.545 0.643 0.677
# C asynchronous trading, true rho 0.6; ticks per unit time 1.0 and 0.2
# tau 1 5 20 100
# previous tick 0.084 0.264 0.474 0.575
# Hayashi-Yoshida (all ticks): 0.600In part A the simulation (569,182 price changes) matches the formula: 1.388 against 1.392 at a sampling interval of 0.05 and 0.860 against 0.848 at 100. The variance at short intervals is about 1.6 times that at long intervals, entirely because of the mean reversion produced by cross-excitation; there is no “measurement error” in the model. In part B the correlation between the two assets’ increments is only 0.041 at an interval of 0.05 and 0.677 at 50, close to the theoretical limit of 0.670. In part C the true correlation is 0.6; the previous-tick method gives only 0.084 at an interval of 1 and 0.575 at 100, while the Hayashi–Yoshida estimator using all trades gives 0.600. So, at least in this simulation, the Epps effect caused by asynchronous observation is removed completely by a suitable estimator, whereas the fall in correlation caused by the dynamics in part B is a property of the increments themselves, not a bias of the estimation method. Figure 3 plots the results of parts A and B.
Hypothesis · not established · Two sources of the Epps effect
The author’s hypothesis (not established): for pairs of liquid stocks, after asynchronous trading has been corrected with the Hayashi–Yoshida estimator, correlations at a one-second scale remain clearly below intraday low-frequency (say 30-minute) correlations; that is, part of the Epps effect comes from microstructure dynamics and not only from asynchronous observation. What would show it wrong: with pairs, periods and estimation methods fixed in advance, a confidence interval for the difference between the corrected one-second correlation and the 30-minute correlation that includes zero.
Chapter summary
Hawkes processes describe order clustering by letting past events raise the current intensity; the branching representation gives the mean intensity , the cluster size and the long-window Fano factor , and stationarity requires , as the framework states. Time-rescaling tests distinguish Poisson and Hawkes models clearly. Whether real markets have a branching ratio near 1 remains disputed (recapped from Book 39). Mutually exciting processes reproduce both the signature plot and the Epps effect; but the Epps effect can also come entirely from asynchronous trading, in which case the Hayashi–Yoshida estimator removes it. “Hawkes processes as the explanation of the Epps effect” in the author’s framework should be qualified to “Hawkes processes as one explanation”.
References
Fan, Y. (2026). High-Frequency Limit Order Book Microstructure, Hawkes Self-Exciting Point Processes and Adverse-Selection Flow Toxicity: From the Avellaneda–Stoikov Market-Making Equations to VPIN, Read Beside Classical Chinese Thought on Market Equilibrium [高频限价订单簿微观结构、霍克斯自激点过程与逆向选择毒流] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23828065 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).