The monkey and the juice

In the 1980s and 1990s, Wolfram Schultz and his colleagues in Fribourg recorded single dopamine neurons in the midbrains of monkeys while the animals learned a simple task. A light or a sound came on, and a few seconds later a drop of fruit juice arrived.

What the neurons did surprised almost everyone. At first, before the animal had learned what the light meant, the neurons fired a burst when the juice arrived. Once the animal had learned that the light predicted juice, the burst moved: the neurons now fired when the light came on, and hardly responded to the juice itself. And if the light came on and the juice did not follow, the neurons fell silent, below their resting rate, at the moment the juice should have arrived.

The learning rule is called temporal-difference learning. At each moment $t$ the brain holds an estimate $V(s_t)$ of how much reward the present situation promises. When the next moment arrives, it computes

$$\delta_t = r_t + \gamma V(s_{t+1}) – V(s_t),$$

the reward $r_t$ that just arrived plus the (discounted) value of the new situation, minus what it had expected. Then it nudges the estimate towards the truth, $V(s_t) \leftarrow V(s_t) + \alpha\,\delta_t$, with a small learning rate $\alpha$ (Sutton and Barto 2018). The older Rescorla–Wagner rule from animal learning, $\Delta V = \alpha (r – V)$, is the same idea without the time steps (Rescorla and Wagner 1972).

alpha, gamma = 0.1, 1.0
T = 10                       # reward arrives 10 steps after the cue
V = [0.0] * (T + 1)          # value of "t steps after the cue"; V[T] is the end of the trial

def trial(reward=1.0, learn=True):
    """Return the prediction error at the cue and at the moment the reward is due."""
    d_cue = gamma * V[0] - 0.0             # the cue comes at an unpredictable time: before it, value 0
    for t in range(T):
        r = reward if t == T - 1 else 0.0
        d = r + gamma * V[t + 1] - V[t]
        if learn:
            V[t] += alpha * d
        if t == T - 1:
            d_reward = d
    return d_cue, d_reward

print("trial   delta at cue   delta at reward")
for n in range(1, 301):
    d_cue, d_rew = trial()
    if n in (1, 10, 50, 100, 300):
        print(f"{n:5d}   {d_cue:12.2f}   {d_rew:15.2f}")
d_cue, d_rew = trial(reward=0.0, learn=False)
print(f"omit    {d_cue:12.2f}   {d_rew:15.2f}")
# trial   delta at cue   delta at reward
#     1           0.00              1.00
#    10           0.00              0.39
#    50           0.02              0.01
#   100           0.54              0.00
#   300           1.00              0.00
# omit            1.00             -1.00

Three facts in this table matter for everything that follows.

A predicted reward produces no signal. By trial 300, the juice is just as sweet as it was on trial 1, and the simulated animal receives exactly the same reward. But the prediction error at that moment is zero. Nothing about the juice has changed; what has changed is that it is no longer news.

The signal moves to whatever predicts the reward. The excitement now belongs to the cue. In human terms, it belongs to the sound of a key in the door, the name on the screen, the evening that has been planned.

An expected reward that fails to arrive produces a negative signal. Before learning, no juice was simply no juice. After learning, no juice is a loss.

Wanting is not liking

It is tempting to call dopamine “the pleasure chemical”. The evidence says otherwise.

This distinction is the key to the rest of the chapter. Prediction errors drive learning and wanting. They are not a measure of how good something is. A long marriage can be full of liking, of comfort, warmth and trust, while producing very little of the dopamine signal that marked its first months. The two can come apart.

Love in the scanner

The second study is important. It shows that long-term love does not have to be only habit: in some couples, after decades, the reward system still responds to the partner. The question is what those couples do differently, and the prediction-error model suggests where to look.

Why the same pleasure fades

The learning rule explains why. If a good thing arrives day after day in the same form, expectation catches up with it, and the prediction error that accompanied it shrinks towards zero.

import random
from math import log

alpha = 0.05                       # how fast expectation catches up, per day
print(f"half-life of the surprise: {log(2) / -log(1 - alpha):.1f} days")

def run(days, sd, seed=7):
    rng = random.Random(seed)
    E, out = 0.0, []
    for day in range(days):
        r = 1.0 + rng.gauss(0, sd)       # what the day actually brings (mean 1)
        delta = r - E                    # prediction error
        E += alpha * delta
        out.append(delta)
    return out

for sd in (0.0, 0.5):
    d = run(365, sd)
    last = d[-90:]
    pos = sum(x for x in last if x > 0) / len(last)
    print(f"sd = {sd}: day 1 {d[0]:.2f}, day 30 {d[29]:.2f} | last 90 days: "
          f"mean delta {sum(last) / len(last):+.2f}, mean positive surprise {pos:.2f}")
# half-life of the surprise: 13.5 days
# sd = 0.0: day 1 1.00, day 30 0.23 | last 90 days: mean delta +0.00, mean positive surprise 0.00
# sd = 0.5: day 1 0.87, day 30 0.81 | last 90 days: mean delta +0.01, mean positive surprise 0.24

The numbers are illustrative; nobody knows the real learning rate of a human heart, and it surely differs between people and between pleasures. But the shape is general. When the days are identical, surprise decays to nothing. When the days vary, surprise never disappears. On average it is still zero, because a learner that is right on average cannot be pleasantly surprised on average, but positive surprises keep arriving, about a quarter of a “unit” per day here, balanced by negative ones.

This is the honest arithmetic of desire. Variety does not raise the average; it keeps the signal alive. And it cuts both ways: a life that keeps producing pleasant surprises also produces disappointments.

What the end of surprise means

Put the pieces together and the “cooling” of the first chapter appears in a new light.

The model also explains a familiar trap. If the signal of desire is the prediction error, there are two ways to get it back. One is to make the shared life richer and less predictable. The other is to look for a new source of reward that has not yet been learned: a new partner, a new screen, a new feed. The second way works, briefly, for exactly the reason the first partner stopped “working”: the new source has not yet been predicted. Then the same arithmetic catches up with it.

Chapter 3 asks the next question. If excitement fades and attention spreads out, what keeps an ordered system, a cell, a flame or a couple, in shape at all?