How two noisy estimates become one

Hold an object between finger and thumb while looking at it and you have two estimates of its size, one from touch and one from vision. Both are noisy, and their noise differs: vision is precise in good light and imprecise in dim light or when the object is partly hidden. Suppose the two estimates $\hat s_V$ and $\hat s_H$ are unbiased, with independent normal noise of standard deviations $\sigma_V$ and $\sigma_H$. Given only these two pieces of information, and given that they come from the same object, the maximum-likelihood estimate is a weighted average:

$$\hat s = w_V\hat s_V + w_H\hat s_H,\qquad w_V = \frac{1/\sigma_V^2}{1/\sigma_V^2 + 1/\sigma_H^2},\quad w_H = 1 – w_V .$$

The weights are proportional to each estimate’s precision (inverse variance). The variance of the combined estimate is

$$\sigma_{VH}^2 = \frac{\sigma_V^2\sigma_H^2}{\sigma_V^2 + \sigma_H^2} \le \min(\sigma_V^2, \sigma_H^2).$$

The combined estimate is therefore always more precise than the better of the two; when the two senses are equally reliable, the standard deviation shrinks to $1/\sqrt 2$, about 0.71, of a single sense. Adding a prior over object size gives the mean of the Bayesian posterior, of the same form. Volume 29, Chapter 6 used the same inverse-variance weighting to combine hearing with laryngeal body sense into an estimate of pitch, and that is not repeated here.

The model makes two testable predictions. First, when two senses disagree, the percept should lie closer to the more reliable one, by an amount set by the two variances. Second, the discrimination threshold with both senses should be lower than with either alone, by an amount that can be calculated.

Judging height by sight and touch

The program below uses the reported values to compute the weights, the combined thresholds and the perceived height in a conflict. The paper reports visual thresholds only at the two ends; the values at 67 % and 133 % are assumed, filled in by the author by linear interpolation. The conflict puts vision 3 mm above the standard and touch 3 mm below. The program also checks the combined spread with 200,000 simulations and compares four ways a person might use the two cues.

# Maximum-likelihood combination of vision and touch for the height of a ridge, after
# Ernst & Banks (2002): standard 55 mm, haptic threshold ~0.085 x height, visual 0.04 x
# height at 0 % noise and 0.2 x at 200 %. Values at 67 % and 133 % are LINEAR
# INTERPOLATIONS (assumed, not reported). T = discrimination threshold (84 % point).
import math, random

def combine(tv, th):
    wv = th ** 2 / (tv ** 2 + th ** 2)
    return wv, math.sqrt(tv ** 2 * th ** 2 / (tv ** 2 + th ** 2))

if __name__ == "__main__":
    h = 55.0
    th = 0.085 * h
    noise = (0, 67, 133, 200)
    tvs = [h * (0.04 + 0.16 * n / 200) for n in noise]
    print(f"haptic threshold {th:.2f} mm")
    print("noise   T_V    w_V   T_VH   best single  felt height (V +3, H -3 mm)")
    for n, tv in zip(noise, tvs):
        wv, tvh = combine(tv, th)
        shift = (2 * wv - 1) * 6 / 2                          # visual +3, haptic -3
        print(f"{n:4d}% {tv:6.2f} {wv:6.2f} {tvh:6.2f} {min(tv, th):9.2f}"
              f" {shift:+20.2f} mm")
    rng = random.Random(2002)                                 # check by simulation
    tv = tvs[1]
    wv, tvh = combine(tv, th)
    est = [wv * rng.gauss(0, tv) + (1 - wv) * rng.gauss(0, th) for _ in range(200000)]
    sd = math.sqrt(sum(e * e for e in est) / len(est))
    print(f"simulated combined spread at 67 %: {sd:.2f} (formula {tvh:.2f})")
    print("three ways a perceiver might use two cues (no conflict), threshold in mm:")
    print("noise   optimal   touch only   alternate 50:50   fixed 50:50 average")
    for n, tv in zip(noise, tvs):
        alt = math.sqrt(0.5 * tv ** 2 + 0.5 * th ** 2)
        avg = math.sqrt(0.25 * tv ** 2 + 0.25 * th ** 2)
        print(f"{n:4d}% {combine(tv, th)[1]:9.2f} {th:12.2f} {alt:17.2f} {avg:15.2f}")
# haptic threshold 4.68 mm
# noise   T_V    w_V   T_VH   best single  felt height (V +3, H -3 mm)
#    0%   2.20   0.82   1.99      2.20                +1.91 mm
#   67%   5.15   0.45   3.46      4.68                -0.29 mm
#  133%   8.05   0.25   4.04      4.68                -1.49 mm
#  200%  11.00   0.15   4.30      4.68                -2.08 mm
# simulated combined spread at 67 %: 3.46 (formula 3.46)
# three ways a perceiver might use two cues (no conflict), threshold in mm:
# noise   optimal   touch only   alternate 50:50   fixed 50:50 average
#    0%      1.99         4.68              3.65            2.58
#   67%      3.46         4.68              4.92            3.48
#  133%      4.04         4.68              6.58            4.66
#  200%      4.30         4.68              8.45            5.98

The weight on vision falls from 0.82 without noise to 0.15 at 200 % noise, and the perceived height in the conflict moves from 1.91 mm toward vision to 2.08 mm toward touch. This is what “visual dominance” amounts to: not an innate privilege of vision, but the fact that vision is usually more reliable. The combined threshold is always below the better single threshold, but the gain varies: without noise, 1.99 against 2.20 mm, only about 10 % lower; at 67 % noise vision and touch are about equally reliable (5.15 and 4.68 mm) and the combined threshold of 3.46 mm is 26 % below the better one. Integration pays most when the two senses are equally reliable.

The second table compares four strategies. “Touch only” and “each sense half the time (alternation)” are both possible. Alternation gives no gain, and when the reliabilities differ widely it is much worse than using the better sense alone (8.45 against 4.68 mm at 200 % noise). A fixed half-and-half average comes close to optimal when the senses are equally reliable (3.48 against 3.46) but not otherwise. To distinguish these strategies in data, the key questions are whether the combined threshold falls below the better single threshold and whether the weights in a conflict shift with noise.

The ventriloquist effect: when vision is blurred, sound takes over

A ventriloquist speaks without moving the lips, and the audience hears the voice coming from the puppet’s mouth. This “ventriloquist effect” is usually described as vision capturing auditory localisation. The maximum-likelihood model offers another account: vision usually localises far more precisely than hearing, and so gets a large weight; make vision blurred enough, and sound should capture vision instead.

The program below uses these values (the middle level is the reported 6°; the other two are rounded) to compute the weights and the perceived position for three blob widths, with vision displaced by +5° and sound by −5°, and then shows why individual observers often fall short of a $\sqrt 2$ improvement.

# The ventriloquist effect as reliability weighting, after Alais & Burr (2004). Sound:
# clicks localised with sd ~6 deg; vision: Gaussian blobs, sd ~1 deg (small blob) to ~20
# deg (64 deg blob). Middle value 6 deg (blob ~32 deg) as reported; others ROUNDED.
# Conflict: vision displaced +D, sound -D, D = 5 deg. Positive shift = towards vision.
import math

def mle(sv, sa, rho=0.0, extra=0.0):
    """weight on vision, combined sd; rho = correlated noise, extra = late added noise"""
    cov = rho * sv * sa
    wv = (sa ** 2 - cov) / (sv ** 2 + sa ** 2 - 2 * cov)
    var = (sv ** 2 * sa ** 2 - cov ** 2) / (sv ** 2 + sa ** 2 - 2 * cov)
    return wv, math.sqrt(var + extra ** 2)

if __name__ == "__main__":
    sa, d = 6.0, 5.0
    print("blob   sd_V   w_V    felt place (deg)   combined sd   gain on best single")
    for blob, sv in ((4, 1.0), (32, 6.0), (64, 20.0)):
        wv, s = mle(sv, sa)
        pos = wv * d + (1 - wv) * (-d)
        print(f"{blob:4d} {sv:6.1f} {wv:6.2f} {pos:+14.2f} {s:16.2f}"
              f" {min(sv, sa) / s:14.3f}")
    print("equal reliabilities (sd 6 and 6): predicted gain sqrt(2) = 1.414;"
          " reported average 1.425")
    print("why single observers often miss the predicted gain (sd 6 and 6):")
    for rho, extra in ((0.0, 0.0), (0.3, 0.0), (0.0, 2.0), (0.3, 2.0)):
        s = mle(6.0, 6.0, rho, extra)[1]
        print(f"  noise correlation {rho:.1f}, late noise {extra:.0f} deg ->"
              f" combined sd {s:.2f}, gain {6.0 / s:.3f}")
# blob   sd_V   w_V    felt place (deg)   combined sd   gain on best single
#    4    1.0   0.97          +4.73             0.99          1.014
#   32    6.0   0.50          +0.00             4.24          1.414
#   64   20.0   0.08          -4.17             5.75          1.044
# equal reliabilities (sd 6 and 6): predicted gain sqrt(2) = 1.414; reported average 1.425
# why single observers often miss the predicted gain (sd 6 and 6):
#   noise correlation 0.0, late noise 0 deg -> combined sd 4.24, gain 1.414
#   noise correlation 0.3, late noise 0 deg -> combined sd 4.84, gain 1.240
#   noise correlation 0.0, late noise 2 deg -> combined sd 4.69, gain 1.279
#   noise correlation 0.3, late noise 2 deg -> combined sd 5.23, gain 1.146

With the small blob the visual weight is 0.97 and the perceived position is +4.73°, almost entirely drawn to the light; with the large blob the visual weight falls to 0.08 and the perceived position is −4.17°: sound has captured the light. Only when the two are equally reliable does combining them improve precision by 1.414 times; when the reliabilities differ widely the improvement is only 1.014 or 1.044 times, almost impossible to measure.

The final part of the program shows that if the noises of the two senses are partly correlated (correlation 0.3), or if 2° of “late” noise is added after combination, the possible improvement falls to 1.24 or 1.28 times, and to 1.15 times with both. These are among the possible reasons the original authors discussed. A test of optimal integration needs enough trials to estimate each person’s single-sense thresholds, and should state in advance how “close to optimal” will be judged.

Chapter summary

Maximum-likelihood integration weights each sense by its precision; the combined estimate is more precise than any single sense, and the gain is largest, a standard deviation shrunk to $1/\sqrt 2$, when the senses are equally reliable. The visual–haptic experiment of Ernst and Banks and the audiovisual localisation experiment of Alais and Burr both came close to the model’s predictions, and the latter showed that sound takes over the judgement of position when vision is blurred enough. But these experiments had very few observers (four and six), individual observers often fall short of the predicted gain, children do not integrate optimally before about 8 to 10 years, and adult experiments have also found over-reliance on vision and departures from maximum-likelihood weights. “Optimal” holds only within the model’s assumptions, the most important of which is that the two signals share a source.