Chapter 5 The Auditory-Motor Loop: The Pitch-Shift Reflex, DIVA and Delayed State Estimation

A slow loop

Chapter 3 of Volume 10 distinguished feedforward pitch control (learned commands that land on the note before it is heard) from feedback (correcting the difference between what is heard and what was intended), and showed with a 120 ms delay model, $\dot x(t) = -Gx(t-\tau)$ , that once $G\tau$ exceeds $\pi/2$ harder correction makes the pitch wobble. This chapter asks what the feedback response looks like in experiments, how neural models organise the two, and how far an “internal model” can solve the problem of delay.

The pitch-shift reflex

Evidence

When people sustain a note while the pitch of their own voice in their headphones is suddenly shifted up or down, their fundamental frequency mostly moves in the direction opposite to the shift, and the response usually compensates for only part of it; with larger shifts, “following” responses in the same direction as the shift become more frequent (Burnett et al. 1998). The latency of these responses is often summarised as about 100–150 ms, although mean values reported for different studies and response types run to about 200 ms (Burnett et al. 1998). Hain and colleagues asked participants to respond voluntarily when they heard a shift and found that the response has two components: an earlier, relatively automatic one and a later one whose latency is long enough to be produced voluntarily; instructions changed the timing and size of both (Hain et al. 2000). Zarate and Zatorre used functional MRI to compare trained singers with non-musicians: when asked to compensate, the two groups compensated to the same degree; when asked to ignore a two-semitone shift of feedback, the singers suppressed their compensatory responses, their F0 resembling that with unaltered feedback, whereas the non-musicians could not (Zarate and Zatorre 2008).

The program below writes the pitch-shift reflex as a delayed “leaky integrator” loop: the headphones return the actual pitch plus an experimental shift (+100 cents for 200 ms), the correction acts on the error heard 120 ms earlier, and a little motor noise is added. Latency is read as experimenters read it: the moment at which the average of 40 trials leaves a band of ±2 standard deviations around the baseline and stays out for 20 ms.

Calculation

# The pitch-shift reflex as a delayed, leaky feedback loop (a sketch, not a fitted model).
# y: sung pitch (cents re target). Headphones return h = y + s, s = +100 cents, 0.5-0.7 s.
# Correction acts on what was heard tau earlier: c' = -G h(t - tau) - c / T_leak; y = c +
# small motor noise; tau = 120 ms (assumed, inside the 100-150 ms range). Latency is read
# as experimenters do: when the mean of 40 trials leaves +/- 2 SD of the pre-shift
# baseline and stays out for 20 ms.
import math, random

DT, TAU, T_LEAK = 0.001, 0.120, 0.5

def trial(G, rnd, shift=100.0):
    n, lag = 1300, int(TAU / DT)
    y, h, c = [0.0] * n, [0.0] * n, 0.0
    for i in range(n):
        t = i * DT
        c += (-G * (h[i - lag] if i >= lag else 0.0) - c / T_LEAK) * DT
        y[i] = c + rnd.gauss(0, 2.0)
        h[i] = y[i] + (shift if 0.5 <= t < 0.7 else 0.0)
    return y

def average(G, k=40, seed=7):
    rnd = random.Random(seed)
    runs = [trial(G, rnd) for _ in range(k)]
    return [sum(r[i] for r in runs) / k for i in range(len(runs[0]))]

print("gain G (1/s) | latency (ms after shift) | largest opposing change | share")
for G in (1.0, 2.0, 4.0):
    m = average(G)
    base = m[300:500]
    mu = sum(base) / len(base)
    sd = math.sqrt(sum((v - mu) ** 2 for v in base) / (len(base) - 1))
    onset = next(i for i in range(500, 1300) if all(v < mu - 2 * sd for v in m[i:i + 20]))
    peak = min(m[500:1000])
    print(f"    {G:4.1f}     | {onset - 500:24d} | {peak:+17.1f} cents |"
          f" {-peak / 100:4.0%}")
print(f"loop delay assumed: {TAU * 1000:.0f} ms; responses oppose the shift,"
      " as in experiments")
# gain G (1/s) | latency (ms after shift) | largest opposing change | share
#      1.0     |                      129 |             -16.5 cents |  17%
#      2.0     |                      122 |             -32.1 cents |  32%
#      4.0     |                      122 |             -61.5 cents |  62%
# loop delay assumed: 120 ms; responses oppose the shift, as in experiments

The measured latency is 122–129 ms, slightly longer than the assumed 120 ms loop delay: the response has to grow above the noise before it can be “seen”, and the smaller the gain, the later that happens. The response always opposes the shift, and its size is proportional to the gain: at gains of 1, 2 and 4 per second the largest opposing change is 17 %, 32 % and 62 % of the shift. The partial compensation usually seen in experiments corresponds in this model to a modest gain, and Volume 10 has already shown that the gain cannot be raised without limit.

DIVA: feedforward, feedback and learning

Evidence

Guenther’s DIVA model (Directions Into Velocities of Articulators) describes speech production as a set of interacting cortical and subcortical regions: a feedforward control subsystem issues learned motor commands directly, while auditory feedback and somatosensory feedback control subsystems compare the actual sensory consequences with target regions and generate corrective commands when errors arise (Guenther 2006). Tourville and Guenther developed the model into a neural theory of speech acquisition and production: it first learns, in a babbling-like phase, the mapping between motor commands and their sensory consequences, then learns the sensory target of each speech sound by imitation, and feedback corrections are gradually absorbed into the feedforward commands, so that skilled speakers rely mainly on feedforward control and feedback becomes visible only when something perturbs it (Tourville and Guenther 2011). Another family of theories casts speech control as state feedback control: the brain uses an “efference copy” of its motor commands and an internal model to predict the current state, corrects that prediction with delayed, noisy sensory feedback, and acts on the estimated state (Houde and Nagarajan 2011). This accords with the view in motor-control research that internal models serve sensorimotor integration (Wolpert et al. 1995); its mathematical prototype is the Kalman filter (Kalman 1960).

State feedback goes straight at the weakness of Volume 10’s model: multiplying an error heard 120 ms ago by a large gain corrects “the past”; adding the commands issued since then to the delayed measurement estimates “the present” (the form of a Smith predictor), with a Kalman filter supplying the best estimate in noise.

The program: delayed feedback and an internal model

The program below works in steps of 10 ms. Pitch error drifts like a random walk and is moved by the motor command; hearing reports the pitch of 12 steps (120 ms) earlier, with sensory noise. Two controllers are compared: direct delayed feedback, and “internal model + Kalman estimate + correction of the estimate”.

Calculation

# Holding a note with a 120 ms auditory delay, 10 ms steps. Pitch error y (cents) drifts:
#   y[k+1] = y[k] + b u[k] + w, w ~ N(0, 1.5^2);  hearing z[k] = y[k-D] + v, v ~ N(0, 8^2)
# (a) direct feedback u = -g z.  (b) forward model + Kalman: estimate y[k-D] from z, add
# the commands sent since (efference copy) to predict y[k], u = -L x prediction
# (Smith-predictor form, after state-feedback accounts of speech). Noise levels and gains
# are illustrative.
import math, random

D, SW, SV = 12, 1.5, 8.0

def simulate(mode, gain, b_true=1.0, steps=2000, seed=0, trace=False):
    rnd = random.Random(seed)
    y, ys, us, e, P, err2, out = 0.0, [0.0] * (D + 1), [0.0] * (D + 1), 0.0, 25.0, 0.0, []
    for k in range(steps):
        z = ys[-D - 1] + rnd.gauss(0, SV)                  # heard now: y from D steps ago
        if mode == "direct":
            u = -gain * z
        else:
            # internal-model step for y[k-D]
            e, P = e + us[-D - 1], P + SW ** 2
            K = P / (P + SV ** 2)
            # Kalman update with the late sample
            e, P = e + K * (z - e), (1 - K) * P
            u = -gain * (e + sum(us[-D:]))                 # + commands sent since then
        y = y + b_true * u + rnd.gauss(0, SW)
        ys, us = ys[1:] + [y], us[1:] + [u]
        err2 += y * y
        out.append(y)
    return (math.sqrt(err2 / steps), out) if trace else math.sqrt(err2 / steps)

def rms(mode, gain, **kw):                       # average over 10 noise realisations
    return sum(simulate(mode, gain, seed=s, **kw) for s in range(10)) / 10

if __name__ == "__main__":
    print(f"no correction (feedforward only): RMS error {rms('direct', 0.0):6.1f} cents")
    print("direct delayed feedback, gain per 10 ms step (G = gain / 0.01 s):")
    for g in (0.02, 0.05, 0.08, 0.11, 0.13, 0.15):
        r = rms("direct", g)
        txt = (f"{r:6.1f} cents" if r < 60 else
               f"{r:6.0f} cents  <- oscillates" if r < 1000 else
               "  > 1000 cents  <- oscillation grows")
        print(f"   g = {g:.2f} (G = {g * 100:4.0f}/s, G*tau = {g * D:.2f}): RMS {txt}")
    print("forward model + Kalman estimate, then the same kind of correction:")
    for L in (0.2, 0.5, 0.9):
        print(f"   L = {L:.1f}: RMS {rms('kalman', L):6.1f} cents")
    print("forward model wrong by 40 % (true command effect 1.4 x the model's), L = 0.9:")
    print(f"   RMS {rms('kalman', 0.9, b_true=1.4):6.1f} cents")
    print(f"stability limit, direct feedback: G*tau below about pi/2 = {math.pi / 2:.2f}")
    print(f"floor no controller can beat: drift over the delay and one step, sqrt(D + 1) x 1.5"
          f" = {math.sqrt(D + 1) * SW:.1f} cents")
# no correction (feedforward only): RMS error   30.4 cents
# direct delayed feedback, gain per 10 ms step (G = gain / 0.01 s):
#    g = 0.02 (G =    2/s, G*tau = 0.24): RMS    8.6 cents
#    g = 0.05 (G =    5/s, G*tau = 0.60): RMS    7.0 cents
#    g = 0.08 (G =    8/s, G*tau = 0.96): RMS    7.6 cents
#    g = 0.11 (G =   11/s, G*tau = 1.32): RMS   11.9 cents
#    g = 0.13 (G =   13/s, G*tau = 1.56): RMS    274 cents  <- oscillates
#    g = 0.15 (G =   15/s, G*tau = 1.80): RMS   > 1000 cents  <- oscillation grows
# forward model + Kalman estimate, then the same kind of correction:
#    L = 0.2: RMS    6.8 cents
#    L = 0.5: RMS    6.5 cents
#    L = 0.9: RMS    6.4 cents
# forward model wrong by 40 % (true command effect 1.4 x the model's), L = 0.9:
#    RMS    6.8 cents
# stability limit, direct feedback: G*tau below about pi/2 = 1.57
# floor no controller can beat: drift over the delay and one step, sqrt(D + 1) x 1.5 = 5.4 cents

Three controllers holding a pitch with a 120 ms auditory delay, on the same noise realisation (first 4 s). With low direct-feedback gain (teal) the error drifts within a few cents; raising the gain to 13 per second (dark red) sets the loop oscillating; an internal model with a Kalman estimate (blue) can use a much larger correction gain without oscillating. Parameters are illustrative.

The results come in four parts.

First, with no feedback at all (feedforward only), drift accumulates and the root-mean-square error over 20 s reaches 30.4 cents. Second, direct delayed feedback has a narrow “useful” range: the error is smallest at a gain of 5 per second (7.0 cents), rises again to 11.9 at 11 per second, and at 13 per second ( $G\tau = 1.56$ , just short of the continuous-time limit $\pi/2$ ) the loop already oscillates, with an RMS error of 274 cents; at 15 per second the oscillation grows faster. The reason is that the discrete-time stability limit lies slightly below the continuous $\pi/2$ (about 12.6 per second for this model, with 10 ms steps and a 12-step delay), so 13 per second is already just beyond it and the oscillation grows slowly over the 20 s. Third, with an internal model the correction gain can be far larger (0.2, 0.5 and 0.9 per step) without oscillation, and the error falls to 6.4–6.8 cents; even when the internal model underestimates the effect of a command by 40 %, the error is only 6.8. Fourth, and easiest to overlook: no controller can go below a floor of about 5.4 cents, because the 13 steps of drift between the heard sample and the next step (the 120 ms delay plus the current step) are something the brain cannot yet know about. An internal model solves the problem of stability, not the problem of information.

Note

Model and brain. The “internal model + Kalman filter” here is only the simplest implementation of the idea of state feedback control, used to show why a delayed system needs prediction. It does not show that the brain contains a Kalman filter, and it leaves out DIVA’s somatosensory feedback, learning and target regions. Real singers’ feedforward commands are so accurate that most of the time feedback needs to make only small corrections, which is the background against which Volume 10 discussed research on external focus of attention.

“Attending to bone conduction and resonance”: as a hypothesis

The framework states: “Attending to bone conduction and resonance sensations stabilises pitch instantly.” First, “instantly” conflicts with the calculation above: drift during the 100–150 ms delay cannot be removed at once by any strategy. Second, bone-conducted sound passes through the same cochlea and auditory pathway as air-conducted sound, with no shorter delay, and Chapter 6 shows that outside noise masks it too. Third, what may be truly independent of hearing are the somatosensory and vibrotactile sensations of larynx and chest, whose resolution for pitch is coarse. In the language of state estimation, the claim becomes a testable hypothesis: under heavy masking, attending to these sensations might shift weight towards them and reduce large pitch excursions; in quiet it should bring no benefit.

Hypothesis · not established

Somatosensory weighting under masking hypothesis (not yet established). Under loud masking accompaniment (the singer’s own voice at the ear below the accompaniment, a ratio under 0 dB), singers instructed to “attend to the vibration in your throat and chest” show a smaller mean absolute pitch error on sustained notes than when instructed to “listen carefully to your own voice”; in quiet the two instructions do not differ, or the first is worse. Testable prediction: in a randomised crossover study with singers as their own controls, an instruction × masking interaction appears, and the confidence interval for the difference under masking lies entirely below zero. If the instructions do not differ under masking either, the hypothesis is refuted. (The author has an interest in the outcome of tests of this hypothesis; see the disclosure in the preface.)

Chapter summary

The pitch-shift reflex shows that people monitor themselves continuously while phonating: a shift of pitch in the headphones produces an opposing, partial compensation after about 100–150 ms, and trained singers can suppress it on instruction. The DIVA model organises speech and song as feedforward control plus auditory and somatosensory feedback, with feedback corrections gradually absorbed into feedforward commands; state feedback control theory emphasises internal models and prediction. The programs show that direct delayed feedback has only a narrow range of useful gain and oscillates when the gain is too high; with an internal model and a Kalman estimate a large gain stays stable, but drift during the delay sets a floor on the error that cannot be broken. “Attending to bone conduction and resonance stabilises pitch instantly” is rewritten as a hypothesis about somatosensory weighting under masking. The next chapter looks at how noise and ear plugs change this loop.

References

Burnett, Theresa A., M. B. Freedland, Charles R. Larson, and Timothy C. Hain. 1998. “Voice F0 Responses to Manipulations in Pitch Feedback.” Journal of the Acoustical Society of America 103 (6): 3153–61.

Guenther, Frank H. 2006. “Cortical Interactions Underlying the Production of Speech Sounds.” Journal of Communication Disorders 39 (5): 350–65.

Hain, Timothy C., Theresa A. Burnett, Swathi Kiran, Charles R. Larson, Shajila A. Singh, and M. K. Kenney. 2000. “Instructing Subjects to Make a Voluntary Response Reveals the Presence of Two Components to the Audio-Vocal Reflex.” Experimental Brain Research 130 (2): 133–41. https://doi.org/10.1007/s002219900237.

Houde, John F., and Srikantan S. Nagarajan. 2011. “Speech Production as State Feedback Control.” Frontiers in Human Neuroscience 5: 82. https://doi.org/10.3389/fnhum.2011.00082.

Kalman, Rudolf E. 1960. “A New Approach to Linear Filtering and Prediction Problems.” Journal of Basic Engineering 82 (1): 35–45. https://doi.org/10.1115/1.3662552.

Tourville, Jason A., and Frank H. Guenther. 2011. “The DIVA Model: A Neural Theory of Speech Acquisition and Production.” Language and Cognitive Processes 26 (7): 952–81.

Wolpert, Daniel M., Zoubin Ghahramani, and Michael I. Jordan. 1995. “An Internal Model for Sensorimotor Integration.” Science 269 (5232): 1880–82. https://doi.org/10.1126/science.7569931.

Zarate, Jean Mary, and Robert J. Zatorre. 2008. “Experience-Dependent Neural Substrates Involved in Vocal Pitch Regulation During Singing.” NeuroImage 40 (4): 1871–87. https://doi.org/10.1016/j.neuroimage.2008.01.026.

Fan, Y. (2026). Vocal Biomechanics, Non-Linear Source-Filter Coupling and Auditory-Motor Feedback: From Titze's Self-Sustained Vocal Fold Oscillation and Vocal-Tract Inertance to the Lombard Effect, Read Beside Classical Chinese Vocal Aesthetics [嗓音生物力学、非线性源-滤波耦合与听觉-运动反馈的数学模型] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23770084 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).