An old word for an old problem

Chapter 3 introduced the Buddhist āsrava, the “outflows” or “leaks” through which the mind’s energy runs out towards its objects. The tradition did not treat them as a minor nuisance. Their ending was the goal of the whole path.

“Great craving must cost much” is an energy account in one line. This chapter tries to write the account out, using the learning model of Chapter 2 and the drive $\mu = J – L – c$ of Chapter 3, and to ask why one modern object, the smartphone, is so good at making attention leak.

The schedule that never resolves

In the 1950s, Charles Ferster and B. F. Skinner studied how animals respond when rewards arrive on different schedules: after every response, after a fixed number of responses, after a fixed time, or after an unpredictable number of responses.

The learning rule of Chapter 2 shows why unpredictable rewards never “settle”. If a reward arrives with probability $p$, a learner’s prediction converges on $p$, and the prediction error is $1-p$ when the reward comes and $-p$ when it does not. Its average size is $2p(1-p)$, which is zero only when $p$ is 0 or 1.

import random

rng = random.Random(1)
alpha = 0.05
print("reward probability   learned value   mean |delta| (last 2000 trials)   2p(1-p)")
for p in (1.0, 0.9, 0.5, 0.25, 0.1):
    V, errs = 0.0, []
    for t in range(6000):
        r = 1.0 if rng.random() < p else 0.0
        d = r - V
        V += alpha * d
        if t >= 4000:
            errs.append(abs(d))
    print(f"{p:18.2f}   {V:13.2f}   {sum(errs) / len(errs):31.2f}   {2 * p * (1 - p):7.2f}")
# reward probability   learned value   mean |delta| (last 2000 trials)   2p(1-p)
#               1.00            1.00                              0.00      0.00
#               0.90            0.95                              0.19      0.18
#               0.50            0.55                              0.50      0.50
#               0.25            0.27                              0.37      0.38
#               0.10            0.08                              0.18      0.18

A reliable source of reward, $p = 1$, stops producing prediction errors once it is learned; this is the fading of Chapter 2. A source that pays off half the time never stops producing them. The errors are as large on the ten-thousandth trial as on the hundredth. A feed that sometimes shows something wonderful, a message that might be important, a notification that might be good news: each of these is a variable-ratio schedule, and each keeps the learning machinery running indefinitely.

How an uncertain reward captures an evening

Chapter 4 explained that a healthy mind values information: it is drawn to what it is uncertain about, because resolving uncertainty is useful. Reinforcement-learning models often build this in as an exploration bonus: an option’s attractiveness is its expected value plus a bonus that grows with how uncertain the learner is about it.

For most things, this bonus is self-limiting. Explore a new restaurant a few times and the uncertainty resolves; the bonus disappears. A variable-ratio source is different. Its uncertainty is not ignorance that can be reduced; it is built into the schedule. The bonus never disappears.

import math, random
from math import log2

def evening(phone_available, kappa, seed=3, minutes=180):
    """Each minute, attend to the partner or the phone. The partner is reliably rewarding (0.3 per minute);
    the phone pays 1.0 with probability 0.15 (mean 0.15). kappa = how strongly unresolved uncertainty attracts."""
    rng = random.Random(seed)
    Q = {"partner": 0.0, "phone": 0.0}       # learned value per minute
    U = {"partner": 0.0, "phone": 0.0}       # running mean of squared prediction error
    use = {"partner": 0, "phone": 0}
    switches, last = 0, None
    for day in range(30):                    # thirty evenings of learning; report the last one
        use = {"partner": 0, "phone": 0}
        switches, last = 0, None
        for m in range(minutes):
            options = ["partner", "phone"] if phone_available else ["partner"]
            w = [math.exp(8 * (Q[o] + kappa * U[o] ** 0.5)) for o in options]
            o = rng.choices(options, weights=w)[0]
            r = 0.3 if o == "partner" else (1.0 if rng.random() < 0.15 else 0.0)
            d = r - Q[o]
            Q[o] += 0.05 * d
            U[o] += 0.05 * (d * d - U[o])
            use[o] += 1
            switches += (last is not None and o != last)
            last = o
    p = [use[o] / minutes for o in use if use[o]]
    H = 0.0 - sum(x * log2(x) for x in p)
    return use["partner"], use["phone"], switches, H

print(f"{'scenario':38s} partner  phone  switches   H (bits)")
for name, avail, kappa in (("phone present, no pull of uncertainty", True, 0.0),
                           ("phone present, uncertainty attracts", True, 0.6),
                           ("phone in another room", False, 0.6)):
    a, b, s, H = evening(avail, kappa)
    print(f"{name:38s} {a:5d}  {b:5d}  {s:8d}   {H:8.2f}")
# scenario                               partner  phone  switches   H (bits)
# phone present, no pull of uncertainty    134     46        65       0.82
# phone present, uncertainty attracts       68    112        75       0.96
# phone in another room                    180      0         0       0.00

Even without any pull of uncertainty ($\kappa = 0$), some minutes go to the phone: a choice by probability always samples the alternatives. But once uncertainty attracts, the phone takes 112 of 180 minutes, although its average reward is half the partner’s. The partner has not become less rewarding. The partner has become known, and the phone never will be. When the phone is in another room, the leak is zero and the entropy of the evening falls to zero bits.

Evidence: phones at the table

The ledger

Combine the pieces into an account that a couple can actually keep. For one evening, write down:

The table is not a scoring system for love, and it should not be used to accuse a partner. It is a way to make an invisible flow visible. Most people underestimate their leaked minutes, for the same reason the learner in the simulation does not notice its drift: no single minute feels like a choice.

The Daodejing passage ends with “know when to stop, and you can last long”. Read through the ledger, it is not a call to renounce pleasure. It is the observation that a flow which leaks without limit cannot sustain any structure, and that the cheapest negentropy is often the leak one does not allow.