To combine or not to combine

Optimal integration in the last chapter had a premise: the two signals share a source. In real life this is not always so. A presenter is talking on television while a car passes outside; we do not take the car for the presenter’s voice. A flash and a beep that are close in space and nearly simultaneous are treated by the brain as one event; far apart, they are treated separately. The ventriloquist effect vanishes when voice and puppet are far apart.

The brain therefore faces two questions: do the two signals have the same source (the causal structure)? And if so, where is it (the estimate)? Körding and colleagues (2007) wrote this as a hierarchical Bayesian model (Körding et al. 2007). Let $C = 1$ denote one common source and $C = 2$ two independent sources, with prior probability $P(C = 1) = p_c$. Source positions have a prior distribution $N(0, \sigma_P^2)$; the visual and auditory measurements $x_V$ and $x_A$ carry noise with standard deviations $\sigma_V$ and $\sigma_A$. By Bayes’s theorem,

$$P(C = 1 \mid x_V, x_A) = \frac{p_c\, p(x_V, x_A \mid C = 1)}{p_c\, p(x_V, x_A \mid C = 1) + (1 – p_c)\, p(x_V, x_A \mid C = 2)} .$$

Under $C = 1$ both measurements come from one position, and the likelihood falls off quickly with their difference $(x_V – x_A)^2$; under $C = 2$ they are independent. The farther apart the signals, the lower the posterior probability of a common source. The final estimate of the sound’s position averages the estimates under the two structures (combined with vision as in the last chapter if $C = 1$, hearing alone plus the prior if $C = 2$), weighted by their posterior probabilities.

The program below uses the four fitted values. The sound is fixed at the centre of the prior (0°) and the light at $+D$; for each separation it simulates 20,000 trials and reports the mean posterior probability of a common source and the average share by which the heard position is pulled toward the light. For comparison it also computes forced fusion ($p_c = 1$), complete segregation ($p_c = 0$) and a hypothetical observer more inclined to fuse ($p_c = 0.8$).

# Bayesian causal inference for a flash and a beep (Koerding et al. 2007). C = 1: one
# source; C = 2: two. Group-fitted values from the paper: sd_V 2.14, sd_A 9.2 deg, prior
# on position N(0, 12.3^2), p_common 0.28. Beep at 0 (prior centre), flash at +D. Bias of
# the heard position = mean estimate / D, in % of the way to the flash.
import math, random
SV, SA, SP, PC = 2.14, 9.2, 12.3, 0.28

def post_common(xv, xa, sv=SV, sa=SA, sp=SP, pc=PC):
    v1, va, vp = sv * sv, sa * sa, sp * sp
    k = v1 * va + v1 * vp + va * vp
    l1 = math.exp(-((xv - xa) ** 2 * vp + xv ** 2 * va + xa ** 2 * v1) / (2 * k)) \
        / (2 * math.pi * math.sqrt(k))
    l2 = math.exp(-xv ** 2 / (2 * (v1 + vp)) - xa ** 2 / (2 * (va + vp))) \
        / (2 * math.pi * math.sqrt((v1 + vp) * (va + vp)))
    return pc * l1 / (pc * l1 + (1 - pc) * l2)

def heard(xv, xa, pc=PC):
    p1 = post_common(xv, xa, pc=pc)
    fused = (xv / SV ** 2 + xa / SA ** 2) / (1 / SV ** 2 + 1 / SA ** 2 + 1 / SP ** 2)
    alone = (xa / SA ** 2) / (1 / SA ** 2 + 1 / SP ** 2)
    return p1 * fused + (1 - p1) * alone, p1

if __name__ == "__main__":
    rng = random.Random(2007)
    print("D (deg)  P(one source)   bias of heard beep:  causal   forced   none"
          "   causal, p_c 0.8")
    for d in (0, 5, 10, 15, 20, 30, 40):
        res = [(rng.gauss(d, SV), rng.gauss(0.0, SA)) for _ in range(20000)]
        p1 = sum(post_common(xv, xa) for xv, xa in res) / len(res)
        if d == 0:
            print(f"{d:6.0f} {p1:12.2f}          (no conflict)")
            continue
        b = [round(100 * sum(heard(xv, xa, pc)[0] for xv, xa in res) / len(res) / d)
             for pc in (PC, 1.0, 0.0, 0.8)]
        print(f"{d:6.0f} {p1:12.2f} {b[0]:23d}% {b[1]:7d}% {b[2]:5d}% {b[3]:11d}%")
# D (deg)  P(one source)   bias of heard beep:  causal   forced   none   causal, p_c 0.8
#      0         0.32          (no conflict)
#      5         0.31                      20%      93%     1%          62%
#     10         0.26                      17%      92%     0%          55%
#     15         0.20                      12%      92%     0%          44%
#     20         0.14                       9%      92%     0%          33%
#     30         0.05                       3%      92%     0%          13%
#     40         0.01                       1%      92%     0%           3%

With the fitted parameters, the heard position is pulled 20 % of the way toward the light at $D = 5°$, 17 % at 10°, 9 % at 20° and only 1 % at 40°. The forced-fusion model pulls the sound 92 % to 93 % of the way at every separation (short of 100 % because the positional prior pulls estimates slightly back to the centre), and complete segregation pulls it almost not at all. This is the “transition from fusion to segregation” of the framework: not a switch, but a curve that falls gradually with separation (Figure 3, right).

Note that with these parameters the mean posterior probability of a common source is only 0.32 even when light and sound coincide. The reason is that auditory localisation is very noisy (9.2°), so the measurements “happening to be close” is weak evidence, and the prior is only 0.28. A hypothetical observer with a prior of 0.8 shows a sharper transition: a 62 % pull at 5°, 33 % at 20° and 3 % at 40°. The fitted values average over participants; individuals may differ widely in prior and noise.

The McGurk effect

The McGurk effect is often held up as the classic case of vision changing hearing. In this chapter’s language: when the visual and auditory signals are treated as having one source, the perceptual system finds an interpretation compatible with both of two different phonemes. Magnotti and Beauchamp (2017) applied the causal-inference model to speech and proposed that fusion occurs only when the brain infers that the two syllables come from the same talker; the same model also accounts for the perception of other incongruent audiovisual speech (Magnotti and Beauchamp 2017).

But “often perceived as /da/” needs qualifying. Later studies found that the strength of the effect varies enormously between people and between stimuli.

These differences have direct consequences for study design. The program below illustrates them with a simple measurement model: each person has a “susceptibility”, each stimulus a “strength”, and the probability of fusion depends on their sum; a person’s susceptibility also wobbles a little between test sessions. The population fusion rates of the 12 stimuli are set to span the 17 %–58 % range above; the distribution across people is assumed.

# How much of "the" McGurk effect is the person and how much the clip? Illustrative
# measurement model: P(fusion) = logistic(person + day + clip). Person ~ N(0, 2.5) and
# day-to-day wobble ~ N(0, 0.7) on the logit scale (both ASSUMED); 12 clips tuned so the
# population fusion rate runs from 17 % to 58 %, the range reported by Basu Mallick et
# al. (2015). 10 trials per clip per session.
import math, random
from statistics import correlation
logistic = lambda z: 1 / (1 + math.exp(-z))

def tune(target, people):
    lo, hi = -10.0, 10.0
    for _ in range(60):
        mid = (lo + hi) / 2
        rate = sum(logistic(p + mid) for p in people) / len(people)
        lo, hi = (mid, hi) if rate < target else (lo, mid)
    return hi

if __name__ == "__main__":
    rng = random.Random(1976)
    people = [rng.gauss(0, 2.5) for _ in range(2000)]
    clips = [tune(0.17 + (0.58 - 0.17) * j / 11, people) for j in range(12)]
    def session():
        out = []
        for p in people:
            q = p + rng.gauss(0, 0.7)
            out.append([sum(rng.random() < logistic(q + c) for _ in range(10)) / 10
                        for c in clips])
        return out
    s1, s2 = session(), session()
    each = [sum(r) / 12 for r in s1]
    rates = [sum(r[j] for r in s1) / len(s1) for j in range(12)]
    print("clip fusion rates (population):", " ".join(f"{x:.0%}" for x in rates))
    print(f"people, all 12 clips: {sum(e < 0.1 for e in each) / len(each):.0%} fuse on"
          f" <10 % of trials, {sum(e > 0.9 for e in each) / len(each):.0%} on >90 %,"
          f" range {min(each):.0%} to {max(each):.0%}")
    for name, j in (("weakest clip", 0), ("strongest clip", 11)):
        share = sum(r[j] >= 0.5 for r in s1) / len(s1)
        print(f"'has the McGurk effect' (>= 5 of 10 fusions) on the {name}: {share:.0%}")
    r = correlation(each, [sum(x) / 12 for x in s2])
    r1 = correlation([x[0] for x in s1], [x[0] for x in s2])
    print(f"test-retest r: mean over 12 clips {r:.2f}; one weak clip alone {r1:.2f}")
# clip fusion rates (population): 18% 21% 25% 29% 32% 36% 40% 43% 47% 51% 55% 58%
# people, all 12 clips: 27% fuse on <10 % of trials, 7% on >90 %, range 0% to 100%
# 'has the McGurk effect' (>= 5 of 10 fusions) on the weakest clip: 16%
# 'has the McGurk effect' (>= 5 of 10 fusions) on the strongest clip: 62%
# test-retest r: mean over 12 clips 0.91; one weak clip alone 0.78

In this model, averaging over the 12 stimuli, 27 % of people fuse on fewer than 10 % of trials and 7 % on more than 90 %, with a range from 0 to 100 %. If a study used only the weakest stimulus and counted “at least 5 fusions in 10” as having the effect, only 16 % of people would “have the McGurk effect”; with only the strongest, 62 %. Same people, conclusions nearly four times apart. Averaging the 12 stimuli, the correlation between two test sessions is 0.91; with a single weak stimulus it falls to 0.78. “How many people experience the McGurk effect?” has no single answer; studies should use several stimuli and report distributions rather than one proportion.

A “common source” in time

Space is not the only cue. If sound and picture are too far apart in time, they too are treated as separate events.

There is a physical reason for the asymmetry: light travels far faster than sound, so in nature sound always arrives after the picture (about 30 ms later from 10 m away) and almost never before. The Chapter 8 checklist uses this asymmetry.

Chapter summary

The causal-inference model makes “whether to combine” itself an object of inference: the farther apart two signals are, the lower the posterior probability of a common source and the weaker the integration. With the values fitted by Körding and colleagues, the pull of a sound toward a light falls gradually from 20 % at 5° to 1 % at 40°, which the forced-fusion model cannot reproduce. The McGurk effect can be understood in the same way, but fusion rates range from 17 % to 58 % across stimuli and from 0 to 100 % across people. Audiovisual speech is integrated within an asymmetric window, from about 30 ms with sound leading to about 170 ms with sound lagging.