5D Library › 零人工多站点容错架构、支付防欺诈状态机与被动现金流排队论 · English
第五章 可用性:共享内核、主备切换与蓝绿排空
共享的代价:一起倒下
一个节点每年故障 4 次、每次修复 131.4 分钟,可用率就是 0.99900,也就是常说的「三个九」。对单个站点而言,每年停机约 525 分钟。把 30 个站点放在同一个共享节点上,停机时间对每个站点并没有变多,但它们会一起停机;把 30 个站点放在 30 个节点上,每个站点的停机时间相同,却几乎不会同时停机,代价是 30 倍的运维工作。
除了硬件,还有部署:每次发布新版本都有可能出错。设每年发布 100 次,其中 2% 有问题,每次在 10 分钟内发现并回滚。在共享内核上,一次坏的发布会让 30 个站点一起出错。金丝雀发布的做法是先只发布到一个站点,观察无误再推广;设它能在金丝雀阶段拦下 90% 的坏发布。
推导
# Chapter 5. [p09] Availability: one shared kernel or thirty, and deploys.
# A node fails 4 times a year and takes 131.4 minutes to repair, so it
# is up 0.999 of the time. Architectures for 30 sites:
# SOLO - one shared node; any failure takes all 30 sites down;
# EACH - 30 nodes, one per site (30 times the operating work);
# PAIR - a shared active node with a passive standby; a failure of
# the active node costs a 60 s switch-over; both down only
# if the standby fails during a repair.
# Bad deploys: 100 deploys a year, 2% of them bad, each found and
# rolled back in 10 minutes. On a shared kernel a bad deploy hits all
# 30 sites; with a canary (one site first) 90% are caught on the canary.
# Reported per year: minutes each site is down, and minutes during
# which all 30 sites are down together.
YEAR = 525600.0
F, MTTR, SWITCH = 4.0, 131.4, 1.0
DEPLOYS, BAD, FIX, CANARY = 100, 0.02, 10.0, 0.9
if __name__ == "__main__":
a = 1 - F * MTTR / YEAR
print(" node availability: {:.5f}".format(a))
solo = F * MTTR
pair_both = F * MTTR * (F * MTTR / YEAR)
pair = F * SWITCH + pair_both
bad = DEPLOYS * BAD * FIX
canary_site = bad * (1 - CANARY) + bad * CANARY / 30
canary_all = bad * (1 - CANARY)
rows = [
("SOLO", solo + bad, solo + bad),
("EACH", solo + bad / 30, 0.0),
("PAIR", pair + bad, pair + bad),
("PAIR+canary", pair + canary_site, pair + canary_all),
]
print(" architecture down min/site/yr all-30-down min/yr")
for name, site, alld in rows:
print(" {:14s} {:17.2f} {:20.2f}".format(name, site, alld))
print(" bad-deploy minutes on a shared kernel: {:.1f}".format(bad))
print("\nReading: a passive standby removes almost all hardware")
print("downtime, leaving bad deploys as the main risk of a shared")
print("kernel - they take every site down together. A one-site")
print("canary cuts that tenfold. Thirty separate nodes avoid joint")
print("outages but each site is down far longer than with the pair.")
# node availability: 0.99900
# architecture down min/site/yr all-30-down min/yr
# SOLO 545.60 545.60
# EACH 526.27 0.00
# PAIR 24.53 24.53
# PAIR+canary 7.13 6.53
# bad-deploy minutes on a shared kernel: 20.0
#
# Reading: a passive standby removes almost all hardware
# downtime, leaving bad deploys as the main risk of a shared
# kernel - they take every site down together. A one-site
# canary cuts that tenfold. Thirty separate nodes avoid joint
# outages but each site is down far longer than with the pair.单节点共享内核(SOLO)每个站点每年停机 545.60 分钟,而且全部是 30 个站点一起停。每站一节点(EACH)每个站点停机 526.27 分钟,几乎不会一起停,但运维量是 30 倍。主备双节点(PAIR)把硬件停机压到每次故障约 1 分钟的切换时间,每个站点每年停机降到 24.53 分钟;这时剩下的风险主要来自坏的发布(每年 20.0 分钟,30 个站点一起)。加上金丝雀发布,每个站点每年停机 7.13 分钟,30 个站点一起停机的时间降到 6.53 分钟。
这说明,对共享内核而言,最大的风险不是硬件,而是「一次发布影响所有站点」。主备节点解决硬件问题,金丝雀发布解决发布问题,两者缺一不可。
蓝绿切换要排空多久
「零停机」发布的常见做法是蓝绿切换:新版本(绿)启动后,新请求转到绿侧,旧版本(蓝)把手上的请求做完,排空 秒后关闭。若请求以速率 泊松到达、持续时间为 ,切换时被切断的请求数的期望是 。程序设每秒 20 个请求,其中 95% 是页面加载(平均 40 毫秒),5% 是流式输出的 AI 测算(平均 30 秒)。进程内的热重载适用同样的算式。
推导
# Chapter 5. [p10] Blue-green switching: how long must the old side drain?
# At a switch, new requests go to the green side while the blue side
# finishes what it holds and is stopped after a drain time D. With
# Poisson arrivals at rate lam, the expected number of requests still
# running after D (and so cut off) is lam E[(S - D)+], where S is the
# request's duration. Traffic: 20 requests a second; 95% are page loads
# (lognormal, mean 40 ms, CV 1.5) and 5% are AI-generated readings
# streamed to the browser (lognormal, mean 30 s, CV 1.0). E[(S - D)+]
# is computed from 400,000 sampled durations (fixed seed). 100 deploys
# a year. The same arithmetic holds for an in-process hot reload.
import math
import random
def ln_params(mean, cv):
s = math.sqrt(math.log(1 + cv * cv))
return math.log(mean) - s * s / 2, s
if __name__ == "__main__":
rng = random.Random(75010)
page, gen = ln_params(0.040, 1.5), ln_params(30.0, 1.0)
xs = [rng.lognormvariate(*(page if rng.random() < 0.95 else gen))
for _ in range(400000)]
print(" drain D (s) cut per switch cut per year share cut")
inflight = 20 * sum(xs) / len(xs)
for d in (0, 5, 30, 60, 120, 300):
cut = 20 * sum(max(0.0, x - d) for x in xs) / len(xs)
print(" {:11d} {:16.3f} {:14.1f} {:11.4f}".format(
d, cut, 100 * cut, cut / inflight))
print(" requests in flight at a switch: {:.2f}".format(inflight))
print("\nReading: page loads drain in a fraction of a second; the")
print("long streamed readings set the drain time. Stopping the old")
print("side at once cuts about thirty readings a switch; two")
print("minutes of drain still cuts dozens a year, five minutes a")
print("handful. Zero downtime is a drain time chosen from the tail of")
print("the longest request type, not a property of the switch.")
# drain D (s) cut per switch cut per year share cut
# 0 30.719 3071.9 1.0000
# 5 25.001 2500.1 0.8138
# 30 9.605 960.5 0.3127
# 60 3.788 378.8 0.1233
# 120 0.960 96.0 0.0313
# 300 0.037 3.7 0.0012
# requests in flight at a switch: 30.72
#
# Reading: page loads drain in a fraction of a second; the
# long streamed readings set the drain time. Stopping the old
# side at once cuts about thirty readings a switch; two
# minutes of drain still cuts dozens a year, five minutes a
# handful. Zero downtime is a drain time chosen from the tail of
# the longest request type, not a property of the switch.切换时在途的请求平均有 30.72 个,几乎全是长时间的流式测算;页面加载在几分之一秒内就做完了。立即关闭旧侧,每次切断 30.719 个;排空 30 秒仍切断 9.605 个;排空 2 分钟每次切断 0.960 个,按每年 100 次发布计一年 96.0 个;排空 5 分钟每年 3.7 个。所谓「零停机」,其实是按最长那类请求的尾部选定的排空时间,而不是切换本身的性质。
按域名热路由
共享内核要同时服务 30 个站点,靠的是按请求头中的域名(Host)查一张路由表:域名对应站点的配置、模板与数据。把路由表做成可以整体替换的对象,就能在不重启的情况下增加或修改站点——新请求读到新表,正在处理的请求继续用旧表,这与上一节的排空是同一个道理。第六章讨论的「把子路径升级为独立域名」,在这种架构下只是路由表里多一行,再加一条从旧路径到新域名的永久重定向(HTTP 301),不需要部署新程序。路由表本身也要像订单一样有版本与校验:一张写错的表会让所有站点一起出错,所以它也应当先在金丝雀站点上生效。
假说 · 尚未验证
作者的诠释(尚未验证): 在作者的系统中,把 AI 流式测算与页面服务放在不同的进程池里,只让页面池参与快速切换,流式池单独以 5 分钟排空,可以让发布不再切断任何正在生成的测算。什么会证明这一判断错:若上线后的日志显示,切换时仍有流式测算被中断(例如因为连接在负载均衡器一层被关闭),那么排空时间之外还有别的路径需要处理。
小结
共享内核的停机是「一起停」:单节点时每个站点每年停 545.60 分钟,全部同时发生。主备节点把硬件停机压到 24.53 分钟,剩下的主要是坏的发布;金丝雀发布再把全部站点一起停的时间降到 6.53 分钟。蓝绿切换的排空时间由最长的请求决定:立即关闭切断 30.719 个请求,排空 5 分钟一年只切断 3.7 个。
Fan, Y. (2026). Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing: Receipt-Hash Deduplication, Self-Referral Blocking, Host-Header Routing and Availability Bounds [零人工多站点容错架构、支付防欺诈状态机与被动现金流排队论] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090201 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).