What does Zero-Touch Sites, Payment Anti-Fraud and Cashflow Queues conclude, in numbers, about “Thirty Sites on One Kernel: Queues and Tail Latency”?

Answer: Chapter 4 shows by queueing theory and simulation that bursty arrivals push a single worker's T99 to tens of seconds; at a mean load of 0.80, bursts need 4 workers, or 3 with load shedding (refusing 0.0005 of requests).

Scope of the numbers: Marked boxes separate evidence, 16 runnable Python programs, historical sources and hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0005, 99

cT99 poisson (ms)T99 bursts (ms)
1173977741
23259253
3280514
4278285
5278275
6278274
7278274
8278274
smallest c for T99 < 500 ms: poisson 2, bursts 4
c = 3 with bursts, refuse when more than Q are waiting:
QrefusedT99 of served (ms)
50.0188289
100.0094312
200.0032376
400.0005470
Reading: a queue that is fine on average is not fine in a
burst; meeting the tail target takes more workers than the
mean load suggests. Refusing a small share of requests at
the door during bursts keeps the rest fast - a choice the
site must make explicitly, not discover in an outage.
Show the full program
# Chapter 4. [p08] How many workers for T99 < 500 ms, and load shedding.
# The 30 sites send 20 requests a second on average (the load of 0.80
# on one worker); service times as in p07 (lognormal, mean 40 ms, CV
# 1.5). Arrivals are Poisson or bursty (4 times the base rate for an
# exponential 10 s every 120 s on average, same long-run mean). c
# workers serve one FCFS queue (M/G/c). The program reports T99 for c =
# 1..8 and the smallest c meeting 500 ms. Then, with c = 3 and bursts,
# requests that find more than Q waiting are refused at once (HTTP
# 503): the refused share and the T99 of those served are reported.
import heapq
import math
import random

MEAN, CV, LAM = 0.040, 1.5, 20.0
SIG = math.sqrt(math.log(1 + CV * CV))
MU = math.log(MEAN) - SIG * SIG / 2

def arrivals(rng, n, burst):
    base = LAM / ((120 + 4 * 10) / 130) if burst else LAM
    t, out, hot = 0.0, [], False
    until = rng.expovariate(1 / 120) if burst else math.inf
    while len(out) < n:
        dt = rng.expovariate(base * (4 if hot else 1))
        if t + dt > until:
            t, hot = until, not hot
            until = t + rng.expovariate(1 / (10 if hot else 120))
            continue
        t += dt
        out.append((t, rng.lognormvariate(MU, SIG)))
    return out

def serve(jobs, c, qmax=None):
    free = [0.0] * c
    starts, resp, refused = [], [], 0
    for a, s in jobs:
        while starts and starts[0] <= a:
            heapq.heappop(starts)
        if qmax is not None and len(starts) > qmax:
            refused += 1
            continue
        f = heapq.heappop(free)
        st = max(a, f)
        heapq.heappush(free, st + s)
        if st > a:
            heapq.heappush(starts, st)
        resp.append(st + s - a)
    resp.sort()
    return resp[int(0.99 * len(resp))], refused / len(jobs)

if __name__ == "__main__":
    rng = random.Random(75008)
    pois = arrivals(rng, 300000, False)
    bur = arrivals(rng, 300000, True)
    print("    c   T99 poisson (ms)   T99 bursts (ms)")
    need = {}
    for c in range(1, 9):
        a, _ = serve(pois, c)
        b, _ = serve(bur, c)
        for k, v in (("poisson", a), ("burst", b)):
            if v < 0.5 and k not in need:
                need[k] = c
        print("   {:2d} {:18.0f} {:17.0f}".format(c, 1000 * a, 1000 * b))
    print("   smallest c for T99 < 500 ms: poisson {}, bursts {}".format(
        need.get("poisson"), need.get("burst")))
    print("   c = 3 with bursts, refuse when more than Q are waiting:")
    print("      Q   refused   T99 of served (ms)")
    for q in (5, 10, 20, 40):
        t99, ref = serve(bur, 3, q)
        print("     {:2d} {:9.4f} {:14.0f}".format(q, ref, 1000 * t99))
    print("\nReading: a queue that is fine on average is not fine in a")
    print("burst; meeting the tail target takes more workers than the")
    print("mean load suggests. Refusing a small share of requests at")
    print("the door during bursts keeps the rest fast - a choice the")
    print("site must make explicitly, not discover in an outage.")

Single source and citation

Fan, Y. (2026). Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing (Version 1.0.0), Chapter 4: Thirty Sites on One Kernel: Queues and Tail Latency. 5D Library. https://doi.org/10.5281/zenodo.24090201

DOI: 10.5281/zenodo.24090201 · Read the monograph · 5DL-B-075 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026075,
  author    = {Fan, Yafu},
  title     = {Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090201},
  url       = {https://doi.org/10.5281/zenodo.24090201}
}

Other chapter answers in this book