What does Zero-Touch Sites, Payment Anti-Fraud and Cashflow Queues conclude, in numbers, about “Thirty Sites on One Kernel: Queues and Tail Latency”?
Answer: Chapter 4 shows by queueing theory and simulation that bursty arrivals push a single worker's T99 to tens of seconds; at a mean load of 0.80, bursts need 4 workers, or 3 with load shedding (refusing 0.0005 of requests).
Scope of the numbers: Marked boxes separate evidence, 16 runnable Python programs, historical sources and hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0005, 99
| c | T99 poisson (ms) | T99 bursts (ms) |
| 1 | 1739 | 77741 |
| 2 | 325 | 9253 |
| 3 | 280 | 514 |
| 4 | 278 | 285 |
| 5 | 278 | 275 |
| 6 | 278 | 274 |
| 7 | 278 | 274 |
| 8 | 278 | 274 |
| smallest c for T99 < 500 ms: poisson 2, bursts 4 | ||
| c = 3 with bursts, refuse when more than Q are waiting: | ||
| Q | refused | T99 of served (ms) |
| 5 | 0.0188 | 289 |
| 10 | 0.0094 | 312 |
| 20 | 0.0032 | 376 |
| 40 | 0.0005 | 470 |
| Reading: a queue that is fine on average is not fine in a | ||
| burst; meeting the tail target takes more workers than the | ||
| mean load suggests. Refusing a small share of requests at | ||
| the door during bursts keeps the rest fast - a choice the | ||
| site must make explicitly, not discover in an outage. |
Show the full program
# Chapter 4. [p08] How many workers for T99 < 500 ms, and load shedding.
# The 30 sites send 20 requests a second on average (the load of 0.80
# on one worker); service times as in p07 (lognormal, mean 40 ms, CV
# 1.5). Arrivals are Poisson or bursty (4 times the base rate for an
# exponential 10 s every 120 s on average, same long-run mean). c
# workers serve one FCFS queue (M/G/c). The program reports T99 for c =
# 1..8 and the smallest c meeting 500 ms. Then, with c = 3 and bursts,
# requests that find more than Q waiting are refused at once (HTTP
# 503): the refused share and the T99 of those served are reported.
import heapq
import math
import random
MEAN, CV, LAM = 0.040, 1.5, 20.0
SIG = math.sqrt(math.log(1 + CV * CV))
MU = math.log(MEAN) - SIG * SIG / 2
def arrivals(rng, n, burst):
base = LAM / ((120 + 4 * 10) / 130) if burst else LAM
t, out, hot = 0.0, [], False
until = rng.expovariate(1 / 120) if burst else math.inf
while len(out) < n:
dt = rng.expovariate(base * (4 if hot else 1))
if t + dt > until:
t, hot = until, not hot
until = t + rng.expovariate(1 / (10 if hot else 120))
continue
t += dt
out.append((t, rng.lognormvariate(MU, SIG)))
return out
def serve(jobs, c, qmax=None):
free = [0.0] * c
starts, resp, refused = [], [], 0
for a, s in jobs:
while starts and starts[0] <= a:
heapq.heappop(starts)
if qmax is not None and len(starts) > qmax:
refused += 1
continue
f = heapq.heappop(free)
st = max(a, f)
heapq.heappush(free, st + s)
if st > a:
heapq.heappush(starts, st)
resp.append(st + s - a)
resp.sort()
return resp[int(0.99 * len(resp))], refused / len(jobs)
if __name__ == "__main__":
rng = random.Random(75008)
pois = arrivals(rng, 300000, False)
bur = arrivals(rng, 300000, True)
print(" c T99 poisson (ms) T99 bursts (ms)")
need = {}
for c in range(1, 9):
a, _ = serve(pois, c)
b, _ = serve(bur, c)
for k, v in (("poisson", a), ("burst", b)):
if v < 0.5 and k not in need:
need[k] = c
print(" {:2d} {:18.0f} {:17.0f}".format(c, 1000 * a, 1000 * b))
print(" smallest c for T99 < 500 ms: poisson {}, bursts {}".format(
need.get("poisson"), need.get("burst")))
print(" c = 3 with bursts, refuse when more than Q are waiting:")
print(" Q refused T99 of served (ms)")
for q in (5, 10, 20, 40):
t99, ref = serve(bur, 3, q)
print(" {:2d} {:9.4f} {:14.0f}".format(q, ref, 1000 * t99))
print("\nReading: a queue that is fine on average is not fine in a")
print("burst; meeting the tail target takes more workers than the")
print("mean load suggests. Refusing a small share of requests at")
print("the door during bursts keeps the rest fast - a choice the")
print("site must make explicitly, not discover in an outage.")Single source and citation
Fan, Y. (2026). Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing (Version 1.0.0), Chapter 4: Thirty Sites on One Kernel: Queues and Tail Latency. 5D Library. https://doi.org/10.5281/zenodo.24090201
DOI: 10.5281/zenodo.24090201 · Read the monograph · 5DL-B-075 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026075,
author = {Fan, Yafu},
title = {Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090201},
url = {https://doi.org/10.5281/zenodo.24090201}
}Other chapter answers in this book
- Chapter 1 · Transfer Receipts: Deduplication, Image Hashes and Release-Then-Audit
- Chapter 2 · Self-Referral and "Invite Friends to Unlock"
- Chapter 3 · The Order State Machine: Exhaustive Checking and the Idempotency Window
- Chapter 5 · Availability: Shared Kernels, Standby Nodes and Blue-Green Draining
- Chapter 6 · Allocating Traffic, Promoting Domains and Holding Cash
- Chapter 7 · Matched Tallies, Banknote Seals and Liu Yan's Canal
- Chapter 8 · A Nine-Flag Audit and a Pre-Registered Experiment