A cup of tea and a hundred coins

Put a cup of hot tea on a desk and leave it. In half an hour it will be lukewarm; in two hours it will be at room temperature. No one has ever seen the reverse: a lukewarm cup that, left alone, draws heat from the room and begins to steam. The second law of thermodynamics names this one-way traffic. In an isolated system, a quantity called entropy never decreases, and in practice it increases until the system reaches equilibrium.

For most of the nineteenth century this was a law without an explanation. Ludwig Boltzmann supplied one. He argued that entropy counts the number of ways the invisible parts of a system, its molecules, can be arranged while the system looks the same from outside (Boltzmann 1877). The formula is carved on his grave in Vienna, in a later notation:

$$S = k_B \ln \Omega$$

Here $\Omega$ is the number of microscopic arrangements (microstates) compatible with what we see (the macrostate), and $k_B = 1.380649 \times 10^{-23}$ joules per kelvin is Boltzmann’s constant, which converts a pure count into the units of heat.

The idea is easiest to see with coins rather than molecules. Toss a hundred coins. The macrostate is the number of heads; a microstate is the full sequence, heads or tails for each coin.

from math import comb, log

k_B = 1.380649e-23   # J/K
N = 100              # coins

for heads in (100, 90, 75, 50):
    omega = comb(N, heads)
    print(f"{heads:3d} heads: Omega = {omega:9.3e}   S/k_B = ln(Omega) = {log(omega):5.1f}")

print(f"S for 50 heads = {k_B * log(comb(N, 50)):.2e} J/K")
total = 2 ** N
near_half = sum(comb(N, h) for h in range(40, 61))
print(f"share of all microstates with 40-60 heads: {near_half / total:.4f}")
print(f"chance of 90 or more heads: {sum(comb(N, h) for h in range(90, 101)) / total:.1e}")
# 100 heads: Omega = 1.000e+00   S/k_B = ln(Omega) =   0.0
#  90 heads: Omega = 1.731e+13   S/k_B = ln(Omega) =  30.5
#  75 heads: Omega = 2.425e+23   S/k_B = ln(Omega) =  53.8
#  50 heads: Omega = 1.009e+29   S/k_B = ln(Omega) =  66.8
# S for 50 heads = 9.22e-22 J/K
# share of all microstates with 40-60 heads: 0.9648
# chance of 90 or more heads: 1.5e-17

There is exactly one way to have a hundred heads and about $10^{29}$ ways to have fifty. Almost 97 percent of all possible sequences lie between forty and sixty heads. Nothing pushes the coins towards the middle; the middle is simply where almost all the possibilities are. Shake the coins in a box and they will find it, and a hundred heads, once lost, will not come back in the lifetime of the universe.

A cup of tea holds around $10^{25}$ molecules, not a hundred, and the counts grow accordingly. That is why the second law, which is “only” a statement about probability, behaves like an iron rule.

Open systems and negative entropy

The second law speaks of isolated systems, ones that exchange neither energy nor matter with anything else. Very little in our lives is isolated. A refrigerator makes its inside colder by doing work and dumping heat into the kitchen; the entropy of the kitchen rises by more than the entropy of the food falls, so the total still goes up. Local order can grow, provided something pays for it elsewhere.

Living things are the most striking example. A cell, a body and a forest are wildly improbable arrangements of matter, and they stay that way for years or centuries. They do it by taking in energy in an ordered form, sunlight or food, and giving it back in a disordered form, heat.

Keep this picture in mind, because the rest of the book leans on it. Order that lasts in an open system is not a possession. It is a process, a balance between what flows in and what leaks out. When the inflow stops, the order does not stay where it was; it relaxes, like the tea, towards the most probable state.

From heat to information

In 1948 Claude Shannon, an engineer at Bell Laboratories, was looking for a way to measure how much information a message carries. He arrived at a formula with the same shape as the entropy of statistical physics (Shannon 1948). If a source can produce outcomes with probabilities $p_1, p_2, \ldots$, its entropy in bits is

$$H = -\sum_i p_i \log_2 p_i .$$

$H$ is zero when one outcome is certain and largest when all outcomes are equally likely. It measures how spread out a probability distribution is, or equivalently how uncertain we are about what will happen next.

There is one place where information and heat meet directly. Rolf Landauer showed in 1961 that erasing one bit of information in a physical device must release at least $k_B T \ln 2$ of heat into the surroundings (Landauer 1961). Forgetting has a minimum energy price.

from math import log

k_B = 1.380649e-23        # J/K
T = 310.0                 # body temperature, K
landauer = k_B * T * log(2)
print(f"Landauer limit at 310 K: {landauer:.2e} J per erased bit")

rest_kcal_per_day = 1700  # a typical adult resting metabolic rate
body_W = rest_kcal_per_day * 4184 / 86400
brain_W = 0.20 * body_W   # the brain uses about a fifth of it
print(f"resting body: {body_W:4.1f} W   brain (20 %): {brain_W:4.1f} W")
print(f"brain power / Landauer limit: {brain_W / landauer:.1e} bit erasures per second")
# Landauer limit at 310 K: 2.97e-21 J per erased bit
# resting body: 82.3 W   brain (20 %): 16.5 W
# brain power / Landauer limit: 5.5e+21 bit erasures per second

The calculation shows that the brain works nowhere near the thermodynamic limit, and the evidence shows that concentrating does not cost much extra energy. So when people say that attention is “drained” or “used up”, they are not describing a literal shortage of joules. They are describing something else: attention is a limited capacity that must be allocated, and how it is allocated has consequences. That is where Shannon’s entropy becomes useful.

The entropy of an evening

Think of an evening at home as a probability distribution: the share of attention that goes to the person across the table, to the meal, to a phone, to work messages, to a television in the background. Shannon’s formula turns that distribution into a single number.

from math import log2

def H(p):
    return 0.0 - sum(x * log2(x) for x in p if x > 0)

evenings = {
    "one shared activity":        [1.0],
    "dinner, mostly together":    [0.7, 0.2, 0.1],
    "4 things, evenly":           [0.25] * 4,
    "16 apps and tabs, evenly":   [1 / 16] * 16,
}
for name, p in evenings.items():
    print(f"{name:26s} H = {H(p):4.2f} bits   (max for {len(p):2d} options: {log2(len(p)):4.2f})")
# one shared activity        H = 0.00 bits   (max for  1 options: 0.00)
# dinner, mostly together    H = 1.16 bits   (max for  3 options: 1.58)
# 4 things, evenly           H = 2.00 bits   (max for  4 options: 2.00)
# 16 apps and tabs, evenly   H = 4.00 bits   (max for 16 options: 4.00)

Each extra bit doubles the number of equally likely places attention might be. An evening at four bits is, in this precise sense, sixteen times as scattered as an evening devoted to one thing.

Higher entropy is not automatically worse. A party, a market or a brainstorming session is meant to spread attention widely. But attention that is always spread, by default rather than by choice, has measurable costs.

The classical Chinese tradition made a similar observation more than two thousand years ago, without the arithmetic.

The chapter does not say that colours, music and food are bad. It says that a mind pulled outward by every stimulus loses the capacity to register any of them; the eye that is shown everything sees nothing. “The belly” in this passage is usually read as what nourishes from within, as opposed to what dazzles from outside. In the language of this chapter, the sage chooses a low-entropy allocation of attention on purpose.

What cooling means, and what it does not

We can now state the first idea of the book carefully.