Normal accidents

In Normal Accidents, published in 1984, the sociologist Charles Perrow started from the 1979 accident at the Three Mile Island nuclear plant and advanced a disturbing thesis: in certain technological systems, serious accidents are not chance mistakes but a “normal” product of the system’s structure (Perrow 1984).

Perrow’s thesis has a corollary that is still disputed: in complex, tightly coupled systems, adding safety devices and redundancy may itself add complexity and new, unforeseen interactions; for some of the riskiest technologies he argued for abandonment or drastic restriction rather than ever more protection. In the language of Chapter 5, “tight coupling” means that failures pass between components faster than people can notice, and “complex interactions” mean connections that do not appear on the design drawings; when both are present, local faults turn most easily into systemic cascades.

High-reliability organisations: another view

The two theories do not simply refute each other. Normal accident theory is about structure: how much time for intervention and how much possibility of understanding interactive complexity and tight coupling leave to people. High-reliability theory is about people and organisation: within a given structure, how to notice and respond faster and how to let information flow to those who know. The first calls for changing the structure (loosening coupling, simplifying design, and if necessary giving the technology up), the second for changing the organisation; a particular system usually needs both, and neither is complete.

Modularity: a firebreak and its price

One basic way to loosen coupling is modularity: dividing a system into parts that are tightly connected inside and sparsely connected to one another. In 1962 Simon argued that complex systems are often “nearly decomposable” hierarchies and used the parable of two watchmakers to show that systems assembled step by step from stable intermediate components withstand interruption better (Simon 1962). May, Levin and Sugihara suggested drawing on ecological research on network structure and stability to understand systemic risk in banking (May et al. 2008).

# Modularity as a firebreak, and what it costs. 400 nodes in 10 modules of 40; 1,200 links
# (mean degree 6); a share mu of the links joins different modules. A failure starts at one
# random node and passes along each link once with probability beta = 0.3 (independent
# cascade, i.e. bond percolation). 400 runs per mu (20 networks x 20 seeds). Cost of modularity:
# mean shortest-path length between nodes (longer = slower help, slower information).
import random
from collections import deque
rng = random.Random(13)
N, M, LINKS, BETA = 400, 10, 1200, 0.3

def network(mu):
    adj = [set() for _ in range(N)]; size = N // M; n = 0
    while n < LINKS:
        a = rng.randrange(N); home = (a // size) * size
        if rng.random() < mu:                     # a random node in another module
            b = (home + size + rng.randrange(N - size)) % N
        else:                                     # a random node in the same module
            b = home + rng.randrange(size)
        if a != b and b not in adj[a]: adj[a].add(b); adj[b].add(a); n += 1
    return adj

def spread(adj, seed):
    hit, q = {seed}, deque([seed])
    while q:
        v = q.popleft()
        for w in adj[v]:
            if w not in hit and rng.random() < BETA: hit.add(w); q.append(w)
    return hit

def mean_path(adj, sources=25):
    tot = cnt = 0
    for s in rng.sample(range(N), sources):
        d = {s: 0}; q = deque([s])
        while q:
            v = q.popleft()
            for w in adj[v]:
                if w not in d: d[w] = d[v] + 1; q.append(w)
        tot += sum(d.values()); cnt += len(d) - 1
    return tot / cnt

print("  mu    mean failed  P(> 1 module hit)  modules hit  P(> 100 failed)  mean path length")
for mu in (0.01, 0.03, 0.05, 0.1, 0.2, 0.5, 0.9):
    sizes, multi, mods, paths = [], 0, 0, []
    for _ in range(20):
        adj = network(mu); paths.append(mean_path(adj))
        for _ in range(20):
            h = spread(adj, rng.randrange(N))
            m = len({v // (N // M) for v in h})
            sizes.append(len(h)); multi += m > 1; mods += m
    n = len(sizes)
    big, path = sum(s > 100 for s in sizes) / n, sum(paths) / len(paths)
    print(f"  {mu:4.2f}     {sum(sizes) / n:6.1f}          {multi / n:4.0%}"
          f"            {mods / n:4.1f}          {big:4.0%}            {path:4.2f}")
#   mu    mean failed  P(> 1 module hit)  modules hit  P(> 100 failed)  mean path length
#   0.01       34.9           35%             1.6            6%            7.64
#   0.03       84.0           60%             3.7           34%            5.50
#   0.05      136.1           68%             5.6           54%            4.89
#   0.10      199.3           76%             7.4           70%            4.29
#   0.20      201.4           76%             7.3           69%            3.87
#   0.50      212.1           80%             7.6           72%            3.58
#   0.90      218.6           84%             7.8           74%            3.55

The firebreak is real, but it is pronounced only when cross-module links are very few. With $\mu = 0.01$ (about 12 cross-module links in the whole network), a mean of 35 nodes fail, nearly two thirds (65 %) of failures stay within the starting module, and the chance that more than 100 nodes fail is only 6 %. With 5 % cross-module links, the mean rises to 136 failed nodes and the chance of more than 100 to 54 %; at 10 % the network differs little from a fully random one ($\mu = 0.9$). A handful of cross-module links is enough to carry the failure out.

The price is just as clear: the mean path length grows from 3.55 to 7.64, more than doubling. There is a further price: with $\mu = 0.01$ a mean of 35 nodes fail, close to the size of one module (40); at this transmission probability, once a failure starts, most of the starting module goes down. Modularity protects the whole, not the part that is hit.

Chapter summary

Perrow argued that interactively complex, tightly coupled systems “normally” have system accidents, and that adding safety devices may add complexity. High-reliability theory stresses that people and organisations can achieve very high reliability within a given structure, while Sagan’s test on nuclear weapons supported the more pessimistic view of normal accident theory. Modularity can keep a failure within one module, but only when cross-module links are very few, and at the cost of everyday efficiency and the near-total failure of the module that is hit. The next chapter reads some classical Chinese texts on “guarding against the small and stopping the gradual”.