5D Library › 零人工多站点容错架构、支付防欺诈状态机与被动现金流排队论 · English

第五章 可用性:共享内核、主备切换与蓝绿排空

共享的代价:一起倒下

一个节点每年故障 4 次、每次修复 131.4 分钟,可用率就是 0.99900,也就是常说的「三个九」。对单个站点而言,每年停机约 525 分钟。把 30 个站点放在同一个共享节点上,停机时间对每个站点并没有变多,但它们会一起停机;把 30 个站点放在 30 个节点上,每个站点的停机时间相同,却几乎不会同时停机,代价是 30 倍的运维工作。

除了硬件,还有部署:每次发布新版本都有可能出错。设每年发布 100 次,其中 2% 有问题,每次在 10 分钟内发现并回滚。在共享内核上,一次坏的发布会让 30 个站点一起出错。金丝雀发布的做法是先只发布到一个站点,观察无误再推广;设它能在金丝雀阶段拦下 90% 的坏发布。

推导

# Chapter 5. [p09] Availability: one shared kernel or thirty, and deploys.
# A node fails 4 times a year and takes 131.4 minutes to repair, so it
# is up 0.999 of the time. Architectures for 30 sites:
#   SOLO   - one shared node; any failure takes all 30 sites down;
#   EACH   - 30 nodes, one per site (30 times the operating work);
#   PAIR   - a shared active node with a passive standby; a failure of
#            the active node costs a 60 s switch-over; both down only
#            if the standby fails during a repair.
# Bad deploys: 100 deploys a year, 2% of them bad, each found and
# rolled back in 10 minutes. On a shared kernel a bad deploy hits all
# 30 sites; with a canary (one site first) 90% are caught on the canary.
# Reported per year: minutes each site is down, and minutes during
# which all 30 sites are down together.

YEAR = 525600.0
F, MTTR, SWITCH = 4.0, 131.4, 1.0
DEPLOYS, BAD, FIX, CANARY = 100, 0.02, 10.0, 0.9

if __name__ == "__main__":
    a = 1 - F * MTTR / YEAR
    print("   node availability: {:.5f}".format(a))
    solo = F * MTTR
    pair_both = F * MTTR * (F * MTTR / YEAR)
    pair = F * SWITCH + pair_both
    bad = DEPLOYS * BAD * FIX
    canary_site = bad * (1 - CANARY) + bad * CANARY / 30
    canary_all = bad * (1 - CANARY)
    rows = [
        ("SOLO", solo + bad, solo + bad),
        ("EACH", solo + bad / 30, 0.0),
        ("PAIR", pair + bad, pair + bad),
        ("PAIR+canary", pair + canary_site, pair + canary_all),
    ]
    print("   architecture    down min/site/yr   all-30-down min/yr")
    for name, site, alld in rows:
        print("   {:14s} {:17.2f} {:20.2f}".format(name, site, alld))
    print("   bad-deploy minutes on a shared kernel: {:.1f}".format(bad))
    print("\nReading: a passive standby removes almost all hardware")
    print("downtime, leaving bad deploys as the main risk of a shared")
    print("kernel - they take every site down together. A one-site")
    print("canary cuts that tenfold. Thirty separate nodes avoid joint")
    print("outages but each site is down far longer than with the pair.")
#    node availability: 0.99900
#    architecture    down min/site/yr   all-30-down min/yr
#    SOLO                      545.60               545.60
#    EACH                      526.27                 0.00
#    PAIR                       24.53                24.53
#    PAIR+canary                 7.13                 6.53
#    bad-deploy minutes on a shared kernel: 20.0
#
# Reading: a passive standby removes almost all hardware
# downtime, leaving bad deploys as the main risk of a shared
# kernel - they take every site down together. A one-site
# canary cuts that tenfold. Thirty separate nodes avoid joint
# outages but each site is down far longer than with the pair.

单节点共享内核(SOLO)每个站点每年停机 545.60 分钟,而且全部是 30 个站点一起停。每站一节点(EACH)每个站点停机 526.27 分钟,几乎不会一起停,但运维量是 30 倍。主备双节点(PAIR)把硬件停机压到每次故障约 1 分钟的切换时间,每个站点每年停机降到 24.53 分钟;这时剩下的风险主要来自坏的发布(每年 20.0 分钟,30 个站点一起)。加上金丝雀发布,每个站点每年停机 7.13 分钟,30 个站点一起停机的时间降到 6.53 分钟。

这说明,对共享内核而言,最大的风险不是硬件,而是「一次发布影响所有站点」。主备节点解决硬件问题,金丝雀发布解决发布问题,两者缺一不可。

蓝绿切换要排空多久

「零停机」发布的常见做法是蓝绿切换:新版本(绿)启动后,新请求转到绿侧,旧版本(蓝)把手上的请求做完,排空 DD 秒后关闭。若请求以速率 λ\lambda 泊松到达、持续时间为 SS,切换时被切断的请求数的期望是 λE[(S−D)+]\lambda E[(S-D)^+]。程序设每秒 20 个请求,其中 95% 是页面加载(平均 40 毫秒),5% 是流式输出的 AI 测算(平均 30 秒)。进程内的热重载适用同样的算式。

推导

# Chapter 5. [p10] Blue-green switching: how long must the old side drain?
# At a switch, new requests go to the green side while the blue side
# finishes what it holds and is stopped after a drain time D. With
# Poisson arrivals at rate lam, the expected number of requests still
# running after D (and so cut off) is lam E[(S - D)+], where S is the
# request's duration. Traffic: 20 requests a second; 95% are page loads
# (lognormal, mean 40 ms, CV 1.5) and 5% are AI-generated readings
# streamed to the browser (lognormal, mean 30 s, CV 1.0). E[(S - D)+]
# is computed from 400,000 sampled durations (fixed seed). 100 deploys
# a year. The same arithmetic holds for an in-process hot reload.
import math
import random

def ln_params(mean, cv):
    s = math.sqrt(math.log(1 + cv * cv))
    return math.log(mean) - s * s / 2, s

if __name__ == "__main__":
    rng = random.Random(75010)
    page, gen = ln_params(0.040, 1.5), ln_params(30.0, 1.0)
    xs = [rng.lognormvariate(*(page if rng.random() < 0.95 else gen))
          for _ in range(400000)]
    print("   drain D (s)   cut per switch   cut per year   share cut")
    inflight = 20 * sum(xs) / len(xs)
    for d in (0, 5, 30, 60, 120, 300):
        cut = 20 * sum(max(0.0, x - d) for x in xs) / len(xs)
        print("   {:11d} {:16.3f} {:14.1f} {:11.4f}".format(
            d, cut, 100 * cut, cut / inflight))
    print("   requests in flight at a switch: {:.2f}".format(inflight))
    print("\nReading: page loads drain in a fraction of a second; the")
    print("long streamed readings set the drain time. Stopping the old")
    print("side at once cuts about thirty readings a switch; two")
    print("minutes of drain still cuts dozens a year, five minutes a")
    print("handful. Zero downtime is a drain time chosen from the tail of")
    print("the longest request type, not a property of the switch.")
#    drain D (s)   cut per switch   cut per year   share cut
#              0           30.719         3071.9      1.0000
#              5           25.001         2500.1      0.8138
#             30            9.605          960.5      0.3127
#             60            3.788          378.8      0.1233
#            120            0.960           96.0      0.0313
#            300            0.037            3.7      0.0012
#    requests in flight at a switch: 30.72
#
# Reading: page loads drain in a fraction of a second; the
# long streamed readings set the drain time. Stopping the old
# side at once cuts about thirty readings a switch; two
# minutes of drain still cuts dozens a year, five minutes a
# handful. Zero downtime is a drain time chosen from the tail of
# the longest request type, not a property of the switch.
每次切换被切断的请求数随排空时间变化。纵轴为对数刻度。

切换时在途的请求平均有 30.72 个,几乎全是长时间的流式测算;页面加载在几分之一秒内就做完了。立即关闭旧侧,每次切断 30.719 个;排空 30 秒仍切断 9.605 个;排空 2 分钟每次切断 0.960 个,按每年 100 次发布计一年 96.0 个;排空 5 分钟每年 3.7 个。所谓「零停机」,其实是按最长那类请求的尾部选定的排空时间,而不是切换本身的性质。

按域名热路由

共享内核要同时服务 30 个站点,靠的是按请求头中的域名(Host)查一张路由表:域名对应站点的配置、模板与数据。把路由表做成可以整体替换的对象,就能在不重启的情况下增加或修改站点——新请求读到新表,正在处理的请求继续用旧表,这与上一节的排空是同一个道理。第六章讨论的「把子路径升级为独立域名」,在这种架构下只是路由表里多一行,再加一条从旧路径到新域名的永久重定向(HTTP 301),不需要部署新程序。路由表本身也要像订单一样有版本与校验:一张写错的表会让所有站点一起出错,所以它也应当先在金丝雀站点上生效。

假说 · 尚未验证

作者的诠释(尚未验证): 在作者的系统中,把 AI 流式测算与页面服务放在不同的进程池里,只让页面池参与快速切换,流式池单独以 5 分钟排空,可以让发布不再切断任何正在生成的测算。什么会证明这一判断错:若上线后的日志显示,切换时仍有流式测算被中断(例如因为连接在负载均衡器一层被关闭),那么排空时间之外还有别的路径需要处理。

小结

共享内核的停机是「一起停」:单节点时每个站点每年停 545.60 分钟,全部同时发生。主备节点把硬件停机压到 24.53 分钟,剩下的主要是坏的发布;金丝雀发布再把全部站点一起停的时间降到 6.53 分钟。蓝绿切换的排空时间由最长的请求决定:立即关闭切断 30.719 个请求,排空 5 分钟一年只切断 3.7 个。

Fan, Y. (2026). Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing: Receipt-Hash Deduplication, Self-Referral Blocking, Host-Header Routing and Availability Bounds [零人工多站点容错架构、支付防欺诈状态机与被动现金流排队论] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090201 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).