5D Library › Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing · 中文
Chapter 5. Availability: Shared Kernels, Standby Nodes and Blue-Green Draining
The cost of sharing: falling together
A node that fails 4 times a year and takes 131.4 minutes to repair each time is available 0.99900 of the time – the familiar “three nines”. For a single site that is about 525 minutes of downtime a year. Putting 30 sites on one shared node does not add downtime for any site, but they go down together; putting 30 sites on 30 nodes gives each site the same downtime, almost never at the same time, at the price of 30 times the operating work.
Besides hardware there are deploys: every new release may go wrong. Let there be 100 releases a year, 2% of them bad, each found and rolled back within 10 minutes. On a shared kernel a bad release breaks all 30 sites at once. A canary release goes first to one site and is rolled out only if that site is healthy; let it catch 90% of bad releases at the canary stage.
Calculation
# Chapter 5. [p09] Availability: one shared kernel or thirty, and deploys.
# A node fails 4 times a year and takes 131.4 minutes to repair, so it
# is up 0.999 of the time. Architectures for 30 sites:
# SOLO - one shared node; any failure takes all 30 sites down;
# EACH - 30 nodes, one per site (30 times the operating work);
# PAIR - a shared active node with a passive standby; a failure of
# the active node costs a 60 s switch-over; both down only
# if the standby fails during a repair.
# Bad deploys: 100 deploys a year, 2% of them bad, each found and
# rolled back in 10 minutes. On a shared kernel a bad deploy hits all
# 30 sites; with a canary (one site first) 90% are caught on the canary.
# Reported per year: minutes each site is down, and minutes during
# which all 30 sites are down together.
YEAR = 525600.0
F, MTTR, SWITCH = 4.0, 131.4, 1.0
DEPLOYS, BAD, FIX, CANARY = 100, 0.02, 10.0, 0.9
if __name__ == "__main__":
a = 1 - F * MTTR / YEAR
print(" node availability: {:.5f}".format(a))
solo = F * MTTR
pair_both = F * MTTR * (F * MTTR / YEAR)
pair = F * SWITCH + pair_both
bad = DEPLOYS * BAD * FIX
canary_site = bad * (1 - CANARY) + bad * CANARY / 30
canary_all = bad * (1 - CANARY)
rows = [
("SOLO", solo + bad, solo + bad),
("EACH", solo + bad / 30, 0.0),
("PAIR", pair + bad, pair + bad),
("PAIR+canary", pair + canary_site, pair + canary_all),
]
print(" architecture down min/site/yr all-30-down min/yr")
for name, site, alld in rows:
print(" {:14s} {:17.2f} {:20.2f}".format(name, site, alld))
print(" bad-deploy minutes on a shared kernel: {:.1f}".format(bad))
print("\nReading: a passive standby removes almost all hardware")
print("downtime, leaving bad deploys as the main risk of a shared")
print("kernel - they take every site down together. A one-site")
print("canary cuts that tenfold. Thirty separate nodes avoid joint")
print("outages but each site is down far longer than with the pair.")
# node availability: 0.99900
# architecture down min/site/yr all-30-down min/yr
# SOLO 545.60 545.60
# EACH 526.27 0.00
# PAIR 24.53 24.53
# PAIR+canary 7.13 6.53
# bad-deploy minutes on a shared kernel: 20.0
#
# Reading: a passive standby removes almost all hardware
# downtime, leaving bad deploys as the main risk of a shared
# kernel - they take every site down together. A one-site
# canary cuts that tenfold. Thirty separate nodes avoid joint
# outages but each site is down far longer than with the pair.A single shared node (SOLO) leaves each site down 545.60 minutes a year, all of it with the 30 sites down together. One node per site (EACH) leaves each site down 526.27 minutes, almost never together, but with 30 times the operating work. An active node with a standby (PAIR) cuts hardware downtime to a switch-over of about a minute per failure, bringing each site’s downtime to 24.53 minutes a year; the remaining risk is then mainly bad releases (20.0 minutes a year, all 30 sites together). With canary releases each site is down 7.13 minutes a year, and the time with all 30 down together falls to 6.53 minutes.
For a shared kernel, then, the main risk is not hardware but “one release affects every site”. A standby node solves the hardware problem and canary releases solve the release problem; both are needed.
How long must a blue-green switch drain?
A common way to deploy with “zero downtime” is a blue-green switch: once the new version (green) is up, new requests go to it, while the old version (blue) finishes what it holds and is stopped after draining for seconds. If requests arrive as a Poisson stream at rate and last , the expected number cut off at a switch is . The program sets 20 requests a second, 95% of them page loads (mean 40 ms) and 5% streamed AI readings (mean 30 s). An in-process hot reload follows the same arithmetic.
Calculation
# Chapter 5. [p10] Blue-green switching: how long must the old side drain?
# At a switch, new requests go to the green side while the blue side
# finishes what it holds and is stopped after a drain time D. With
# Poisson arrivals at rate lam, the expected number of requests still
# running after D (and so cut off) is lam E[(S - D)+], where S is the
# request's duration. Traffic: 20 requests a second; 95% are page loads
# (lognormal, mean 40 ms, CV 1.5) and 5% are AI-generated readings
# streamed to the browser (lognormal, mean 30 s, CV 1.0). E[(S - D)+]
# is computed from 400,000 sampled durations (fixed seed). 100 deploys
# a year. The same arithmetic holds for an in-process hot reload.
import math
import random
def ln_params(mean, cv):
s = math.sqrt(math.log(1 + cv * cv))
return math.log(mean) - s * s / 2, s
if __name__ == "__main__":
rng = random.Random(75010)
page, gen = ln_params(0.040, 1.5), ln_params(30.0, 1.0)
xs = [rng.lognormvariate(*(page if rng.random() < 0.95 else gen))
for _ in range(400000)]
print(" drain D (s) cut per switch cut per year share cut")
inflight = 20 * sum(xs) / len(xs)
for d in (0, 5, 30, 60, 120, 300):
cut = 20 * sum(max(0.0, x - d) for x in xs) / len(xs)
print(" {:11d} {:16.3f} {:14.1f} {:11.4f}".format(
d, cut, 100 * cut, cut / inflight))
print(" requests in flight at a switch: {:.2f}".format(inflight))
print("\nReading: page loads drain in a fraction of a second; the")
print("long streamed readings set the drain time. Stopping the old")
print("side at once cuts about thirty readings a switch; two")
print("minutes of drain still cuts dozens a year, five minutes a")
print("handful. Zero downtime is a drain time chosen from the tail of")
print("the longest request type, not a property of the switch.")
# drain D (s) cut per switch cut per year share cut
# 0 30.719 3071.9 1.0000
# 5 25.001 2500.1 0.8138
# 30 9.605 960.5 0.3127
# 60 3.788 378.8 0.1233
# 120 0.960 96.0 0.0313
# 300 0.037 3.7 0.0012
# requests in flight at a switch: 30.72
#
# Reading: page loads drain in a fraction of a second; the
# long streamed readings set the drain time. Stopping the old
# side at once cuts about thirty readings a switch; two
# minutes of drain still cuts dozens a year, five minutes a
# handful. Zero downtime is a drain time chosen from the tail of
# the longest request type, not a property of the switch.At a switch, 30.72 requests are in flight on average, almost all of them long streamed readings; page loads finish in a fraction of a second. Stopping the old side at once cuts 30.719 requests per switch; draining 30 s still cuts 9.605; draining 2 minutes cuts 0.960 per switch, 96.0 a year at 100 releases; draining 5 minutes, 3.7 a year. “Zero downtime” is in fact a drain time chosen from the tail of the longest kind of request, not a property of the switch itself.
Hot routing by host name
A shared kernel serves 30 sites by looking up the host name in the request header (Host) in a routing table: each name maps to a site’s configuration, templates and data. If the routing table is an object that can be swapped as a whole, sites can be added or changed without a restart – new requests read the new table while requests in progress keep the old one, the same principle as the drain of the previous section. The “promotion of a sub-path to its own domain” discussed in Chapter 6 then becomes one more row in the routing table plus a permanent redirect (HTTP 301) from the old path to the new domain, with no new program deployed. The routing table itself needs versions and checks, like an order: a wrong table breaks every site at once, so it too should take effect first on a canary site.
Hypothesis · not established
The author’s interpretation (not yet tested): in the author’s system, putting streamed AI readings and page serving in separate process pools – with only the page pool switching quickly and the streaming pool draining separately for 5 minutes – lets releases stop cutting off any reading being generated. What would show this wrong: if logs after launch showed streamed readings still interrupted at switches (because connections are closed at the load balancer, say), there would be paths besides the drain time to deal with.
Summary
Downtime on a shared kernel happens together: with a single node, each site is down 545.60 minutes a year, all at once. A standby node cuts hardware downtime to 24.53 minutes, leaving mostly bad releases; canary releases cut the time with all sites down together to 6.53 minutes. The drain time of a blue-green switch is set by the longest requests: stopping at once cuts 30.719 requests, draining 5 minutes only 3.7 a year.
Fan, Y. (2026). Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing: Receipt-Hash Deduplication, Self-Referral Blocking, Host-Header Routing and Availability Bounds [零人工多站点容错架构、支付防欺诈状态机与被动现金流排队论] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090201 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).