5D Library › Dynamical Systems of Neurotransmitters and Endocrinology · 中文
Chapter 1 Dopamine Is Not the Pleasure Molecule: Reward Prediction Error
An unexpected firing pattern
In the 1980s and 1990s Wolfram Schultz recorded the firing of dopamine neurons in the midbrain of monkeys. The monkeys learned that a few seconds after a light came on, they would receive a drop of juice. Early in learning, the juice itself caused a brief burst of firing; once the task was learned, the burst moved to the moment the light came on, and the juice no longer produced a response; and if the light came on but no juice followed, firing dropped below baseline at the moment the juice should have arrived.
Evidence
Schultz, Dayan and Montague pointed out that this pattern closely matches the “temporal-difference error” of reinforcement learning: the phasic firing of dopamine neurons reports not the reward itself but the difference between what was received and what was predicted, the reward prediction error (RPE) (Schultz et al. 1997). More than expected gives a positive error; exactly as expected gives zero; less than expected gives a negative error.
This finding changed the popular equation of dopamine with pleasure. A series of studies by Berridge and Robinson went further and separated “wanting” (incentive salience) from “liking” (hedonic experience): rats whose dopamine was blocked or depleted still showed hedonic reactions to sweetness (facial expressions such as lip licking), but no longer worked to obtain it (Berridge and Robinson 1998). Dopamine is closely tied to wanting and to learning what to expect, and much more weakly to liking itself.
Temporal-difference learning
In reinforcement learning the value of a state is the sum of future rewards expected from that state. At each step the learner computes the temporal-difference error (Sutton and Barto 2018)
and corrects the value: , where is the reward received at that step, the discount factor and the learning rate.
Calculation
Temporal-difference learning on a cue–reward trial. The cue appears at step 2 and the reward arrives at step 7. The moment before the cue cannot be predicted, so its value is fixed at 0.
# Temporal-difference learning on a cue -> reward trial (after Schultz, Dayan & Montague 1997).
# Time steps 0..9 within a trial; the cue appears at step 2, the reward (r = 1) at step 7.
# The state before the cue is unpredictable (its value is held at 0), so the cue's arrival is a surprise.
GAMMA, ALPHA, T, CUE, REW = 1.0, 0.2, 10, 2, 7
V = [0.0] * (T + 1) # value of each time step after the cue
def trial(reward=1.0, learn=True):
deltas = []
for t in range(T):
r = reward if t == REW else 0.0
v_now = V[t] if t >= CUE else 0.0
v_next = V[t + 1] if t + 1 >= CUE else 0.0
d = r + GAMMA * v_next - v_now
if learn and t >= CUE:
V[t] += ALPHA * d
deltas.append(d)
return deltas
def show(label, d):
print(f"{label:22s} delta at cue {d[CUE - 1]:+.2f}, at reward {d[REW]:+.2f}")
show("trial 1 (naive)", trial())
for _ in range(3):
trial()
show("trial 5", trial())
for _ in range(24):
trial()
show("trial 30", trial())
for _ in range(170):
trial()
show("trial 201 (learned)", trial())
show("reward omitted", trial(reward=0.0, learn=False))
show("reward doubled", trial(reward=2.0, learn=False))
# trial 1 (naive) delta at cue +0.00, at reward +1.00
# trial 5 delta at cue +0.00, at reward +0.41
# trial 30 delta at cue +0.54, at reward +0.00
# trial 201 (learned) delta at cue +1.00, at reward +0.00
# reward omitted delta at cue +1.00, at reward -1.00
# reward doubled delta at cue +1.00, at reward +1.00The program reproduces the three features of Schultz’s recordings: early in learning the error occurs at the reward; after learning it moves to the cue; and when an expected reward fails to arrive the error becomes negative. When the reward is doubled, a positive error appears at the reward again: the system is sensitive to “better than expected”, not to the absolute size of the reward.
Why the baseline gets raised
A common experience: after an evening of short videos, sitting down the next day to read, write or practise feels unusually dull. The popular phrase is that “the dopamine threshold has been raised”. There is a computational version of this idea that can be stated clearly, but it needs one more quantity than the classic temporal-difference model.
Evidence
Niv, Daw, Joel and Dayan proposed an average-reward model of reinforcement learning: besides phasic prediction errors, animals track the average rate of reward in the environment, , and use it to measure the opportunity cost of time. They suggested that tonic dopamine levels may encode this average rate and so set the vigour of responding (Niv et al. 2007). In this framework the error is written .
In the average-reward framework, whether a reward is good is judged against how much each step here usually brings. If intense rewards are frequent, rises, and an ordinary reward worth 1, minus the higher , becomes a negative error.
Calculation
The average-reward model. Sixty events a day; ordinary progress from effort is worth 1, an intense stimulus 6. First 7 ordinary days, then 14 days in which 30 % of events are intense stimuli, then 7 days with the intense stimuli removed. To isolate the role of , each event is treated as a one-step episode, so the value terms cancel.
# Average-reward TD (Niv, Daw, Joel & Dayan 2007): delta = r - rho + V(s') - V(s).
# rho is the running average reward rate (a model of "tonic" dopamine / opportunity cost).
# Here every event is a one-step episode, so V terms cancel and delta = r - rho.
# A simplified model, not a measurement of anyone's dopamine.
import random
random.seed(1)
BETA = 0.02 # learning rate of the average
EVENTS_PER_DAY = 60
def day(rho, intense_share, log):
for _ in range(EVENTS_PER_DAY):
intense = random.random() < intense_share
r = 6.0 if intense else 1.0 # ordinary effortful progress = 1, intense stimulus = 6
delta = r - rho
rho += BETA * delta
if not intense:
log.append(delta)
return rho
rho = 1.0
for phase, days, share in (("ordinary days", 7, 0.0), ("plus frequent intense stimuli", 14, 0.30),
("intense stimuli removed", 7, 0.0)):
for d in range(days):
log = []
rho = day(rho, share, log)
if d in (0, days - 1):
neg = sum(1 for x in log if x < -0.05) / len(log)
mean = sum(log) / len(log)
print(f"{phase:30s} day {d + 1:2d}: rho {rho:4.2f}, ordinary event delta {mean:+.2f}, "
f"below -0.05 in {100 * neg:3.0f} %")
# how many ordinary events until rho is back within 10 % of 1.0, starting from the inflated value
rho, n = 2.5, 0
while rho > 1.1:
rho += BETA * (1.0 - rho); n += 1
print(f"from rho 2.5, events of ordinary reward until rho < 1.1: {n} (about {n / EVENTS_PER_DAY:.1f} days at {EVENTS_PER_DAY}/day)")
# ordinary days day 1: rho 1.00, ordinary event delta +0.00, below -0.05 in 0 %
# ordinary days day 7: rho 1.00, ordinary event delta +0.00, below -0.05 in 0 %
# plus frequent intense stimuli day 1: rho 2.28, ordinary event delta -0.68, below -0.05 in 95 %
# plus frequent intense stimuli day 14: rho 2.13, ordinary event delta -1.30, below -0.05 in 100 %
# intense stimuli removed day 1: rho 1.34, ordinary event delta -0.66, below -0.05 in 100 %
# intense stimuli removed day 7: rho 1.00, ordinary event delta -0.00, below -0.05 in 0 %
# from rho 2.5, events of ordinary reward until rho < 1.1: 135 (about 2.2 days at 60/day)
Note
This is a simplified model, not a measurement. No study has directly measured the prediction errors of midbrain dopamine neurons in people watching short videos, nor shown that dopamine errors for ordinary effort become negative afterwards. The model shows that if the brain judges rewards against an average reward rate, frequent intense stimulation will make ordinary rewards seem worse than expected; it also shows that the effect should reverse once the intense stimulation stops, at a speed set by the update rate of the average, whose value in humans is unknown. The “two days or so” of recovery in the program comes from an assumed parameter and should not be read as advice to “abstain for a few days”.
Opponent processes
There is an older description too. In 1974 Solomon and Corbit proposed the opponent-process theory: a strong emotional stimulus triggers a fast “a-process” and also a slow “b-process” of opposite sign; the emotion felt is the difference. The b-process strengthens with repeated exposure, so the peak produced by the stimulus becomes lower and the opposite trough after it becomes deeper and longer (Solomon and Corbit 1974).
Calculation
A simplified opponent-process model. The a-process has a time constant of 1 minute and the b-process 8 minutes; the gain of b grows a little after each exposure and decays by 3 % a day once exposure stops.
# Opponent-process model of affect (after Solomon & Corbit 1974), as a simplified model:
# a-process follows the stimulus quickly; b-process is slow and opposes it; its gain g grows a little
# with every exposure and decays slowly between exposures. Felt affect = a - b.
def exposure(g, minutes=20, total=120, dt=0.1):
a = b = 0.0
peak, trough, t_neg, t = 0.0, 0.0, 0.0, 0.0
while t < total:
s = 1.0 if t < minutes else 0.0
a += dt * (s - a) / 1.0 # fast: time constant 1 min
b += dt * (g * a - b) / 8.0 # slow: time constant 8 min, driven by a
f = a - b
peak, trough = max(peak, f), min(trough, f)
if t > minutes and f < -0.05:
t_neg += dt
t += dt
return peak, trough, t_neg
g = 0.3
print("exposure b-gain peak affect after-dip minutes below -0.05")
for n in range(1, 61):
if n in (1, 10, 30, 60):
p, tr, tn = exposure(g)
print(f"{n:8d} {g:6.2f} {p:11.2f} {tr:9.2f} {tn:8.0f}")
g = g + 0.05 * (1.6 - g) # strengthening with use, towards a ceiling of 1.6
for days in (7, 30):
g_rest = 0.3 + (g - 0.3) * 0.97 ** days # slow decay of the gain during abstinence (3 % per day)
p, tr, tn = exposure(g_rest)
print(f"after {days:2d} days without exposure: b-gain {g_rest:.2f}, peak {p:.2f}, after-dip {tr:.2f}")
# exposure b-gain peak affect after-dip minutes below -0.05
# 1 0.30 0.89 -0.17 13
# 10 0.78 0.78 -0.51 22
# 30 1.31 0.70 -0.91 26
# 60 1.54 0.67 -1.09 28
# after 7 days without exposure: b-gain 1.30, peak 0.70, after-dip -0.91
# after 30 days without exposure: b-gain 0.80, peak 0.78, after-dip -0.53The model gives three testable directions: after repeated exposure the peak falls a little while the after-trough becomes deeper and longer; once exposure stops the trough becomes shallower again, but more slowly than it deepened. Opponent-process theory was first used to describe parachuting, blood donation and addiction; how it maps onto dopamine circuits is still being studied, and this book uses it only as a model of the time course of feelings.
Hypothesis · not established
Not established. In adults who spend much of each day on short videos and other high-frequency, intense-feedback media, cutting such use to under 30 minutes a day for 14 days will raise their self-rated interest in, and time spent on, a pre-chosen ordinary task (such as 20 minutes of reading), with most of the rise between days 3 and 10, in line with the recovery curve of the average-reward model. If, in a randomised controlled trial, the 95 % confidence interval for the difference in task time between the reduction and no-reduction groups on day 14 lies entirely within ±5 minutes, the hypothesis is refuted.
Summary
The phasic dopamine signal is better described as a report of “better or worse than expected” than as pleasure itself. The intuition of a raised baseline has a clear form in the average-reward model, but it is a model, not a measurement. The next chapter turns to another loop: the negative feedback of the stress hormones.
References
Fan, Y. (2026). Dynamical Systems of Neurotransmitters and Endocrinology: Dopamine Reward Prediction Error, Serotonergic Homeostasis, HPA-Axis Allostatic Load and the Challenge Hypothesis, Read Beside East Asian Classics of Mind Regulation [神经递质与内分泌的动力系统] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23738782 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).