A meeting point

The last two chapters were about behaviour: how people combine estimates from two senses. This chapter looks at neurons. The superior colliculus in the midbrain helps turn the eyes and head toward external stimuli, and its deep layers contain many neurons that respond to visual, auditory and somatosensory stimuli alike. In the 1980s Meredith and Stein recorded such neurons in anaesthetised cats and found that when two senses were stimulated together within a neuron’s receptive fields, the response could far exceed the response to either alone, sometimes even their sum (Meredith and Stein 1983).

In their 1993 book Stein and Meredith summarised these observations as three “principles” (Stein and Meredith 1993): the spatial principle (enhancement when the two stimuli fall at the same place, none or even suppression when they are far apart), the temporal principle (enhancement when the two stimuli are close in time), and the principle of inverse effectiveness (the weaker the single-sense stimuli, the larger the relative enhancement from combining them). The size of the enhancement is commonly expressed by an index,

$$\text{ME} = \frac{CM – SM_{\max}}{SM_{\max}} \times 100\%,$$

where $CM$ is the mean response to the multisensory combination and $SM_{\max}$ the mean response to the stronger single sense. Another common comparison sets the combined response against the sum of the two single responses: a ratio above 1 is called superadditive, equal to 1 additive, below 1 subadditive.

One smooth nonlinearity is enough

Superadditivity and inverse effectiveness sound like special “integration rules”. But both appear as soon as a neuron adds its two inputs and passes the sum through an S-shaped nonlinearity, accelerating for weak inputs and saturating for strong ones. The program below simulates such a neuron: the drives of the two inputs add, and the firing rate is an S-shaped function of the drive (exponent 2.5, half-saturation at 1, maximum about 62 spikes/s, spontaneous rate 2 spikes/s); each condition is recorded on 30 trials of 0.5 s of Poisson spikes. The program computes the enhancement index, the additivity ratio (after removing spontaneous firing), and a $z$ value testing whether the combined response exceeds the sum of the single responses. All parameters are illustrative.

# Superadditivity and inverse effectiveness from one smooth nonlinearity (illustrative).
# A neuron sums visual and auditory drive, E = v + a, then fires as a Poisson process at
# rate 2 + 60 E^2.5 / (E^2.5 + 1) spikes/s (half-saturation at E = 1). 30 trials of 0.5 s.
# ME = (VA - best single) / best single (Meredith & Stein's index); additivity = VA /
# (V + A) after removing the 2 spikes/s baseline; z tests VA against V + A.
import math, random

def rate(e):
    return 2 + 60 * e ** 2.5 / (e ** 2.5 + 1)

def counts(rng, e, trials=30, t=0.5):
    out = []
    for _ in range(trials):                         # Poisson counts from exponential gaps
        n, clock = 0, rng.expovariate(rate(e))
        while clock < t:
            n, clock = n + 1, clock + rng.expovariate(rate(e))
        out.append(n)
    return out

def stats(x):
    m = sum(x) / len(x)
    return m, sum((v - m) ** 2 for v in x) / (len(x) - 1) / len(x)

if __name__ == "__main__":
    rng = random.Random(1983)
    base = 2 * 0.5
    print("drive each   V      A      VA    ME     additivity   z (VA vs V + A)")
    for s in (0.15, 0.3, 0.5, 1.0, 2.0, 4.0):
        (mv, vv), (ma, va), (mb, vb), (m0, v0) = (stats(counts(rng, e)) for e in
                                                  (s, s, 2 * s, 0.0))
        me = (mb - max(mv, ma)) / max(mv, ma)
        add = (mb - base) / (mv + ma - 2 * base)
        z = (mb - (mv + ma - m0)) / math.sqrt(vb + vv + va + v0)
        print(f"{s:9.2f} {mv:6.1f} {ma:6.1f} {mb:6.1f} {me:+6.0%} {add:9.2f} {z:12.1f}")
    for s in (0.15, 1.0):
        exact = (rate(2 * s) - 2) / (2 * (rate(s) - 2))
        print(f"noise-free additivity at drive {s}: {exact:.2f}")
# drive each   V      A      VA    ME     additivity   z (VA vs V + A)
#      0.15    1.2    1.0    2.3   +89%      6.33          1.8
#      0.30    2.5    2.5    8.6  +241%      2.52          7.4
#      0.50    5.7    5.8   14.4  +150%      1.41          4.2
#      1.00   15.9   16.0   25.9   +62%      0.83         -3.6
#      2.00   26.1   25.9   29.7   +14%      0.57        -13.7
#      4.00   30.2   30.9   31.1    +1%      0.51        -15.7
# noise-free additivity at drive 0.15: 2.72
# noise-free additivity at drive 1.0: 0.85

At a drive of 0.3 the combined response is 2.52 times the sum of the single responses, an enhancement of +241 %, with $z = 7.4$; at a drive of 1 the additivity ratio falls to 0.83, subadditive; at a drive of 4 the enhancement is only +1 % and the neuron is saturated. This is inverse effectiveness: the same neuron, the same rule, superadditive for weak stimuli and subadditive for strong ones.

Two points deserve attention. First, at the weakest level (0.15) the true additivity ratio is 2.72 (the last line of the program, computed without noise), yet with 30 trials the $z$ value is only 1.8, short of the usual criterion for significance, and the enhancement index of +89 % is smaller than at the next level. The reason is that both single responses are close to spontaneous firing, so the denominator of the ratio is small and its estimate unstable. Second, the estimated additivity ratio of 6.33 is far above the true 2.72, again because the denominator is close to zero. Near the floor of the response, superadditivity is both hard to detect and easy to overestimate.

The normalisation model

Ohshiro, Angelaki and DeAngelis (2011) proposed a more general explanation (Ohshiro et al. 2011). Each neuron first adds its two inputs linearly with its own “dominance weights”, $E = d_1 I_1 + d_2 I_2$, passes the result through an expansive nonlinearity with exponent $n$, and then divides by the mean activity of a pool of $N$ neurons:

$$R = \frac{E^{\,n}}{\alpha^{\,n} + \frac{1}{N}\sum_{j=1}^{N} E_j^{\,n}} .$$

This divisive normalisation has long been studied in visual cortex. Applied to multisensory integration it makes three predictions: first, all model neurons show inverse effectiveness, but superadditivity appears only when responses are weak; second, the spatial principle emerges naturally; third, an input that is itself excitatory but non-optimal for a given neuron can suppress that neuron’s response, because it strongly drives other neurons in the normalisation pool. The authors regarded the third as the key prediction distinguishing this model from others.

The program below implements the model at one receptive-field location: 25 neuron types, with dominance weights for each input of 0, 0.25, 0.5, 0.75 and 1, $n = 2$ and $\alpha = 1$. As in the original, the denominator uses the mean activity of the normalisation pool; the original pool also includes neurons with receptive fields at other locations, so taking a single location here is a simplification.

# Divisive normalisation model of multisensory integration (Ohshiro, Angelaki & DeAngelis
# 2011), one receptive-field location. 25 neuron types with dominance weights d1, d2 in
# {0, .25, .5, .75, 1}; drive E = d1*I1 + d2*I2; response R = E^n / (a^n + mean_pool E^n)
# with n = 2, a = 1; as in the paper, the pool term is the MEAN over the N neuron types.
W = (0.0, 0.25, 0.5, 0.75, 1.0)
POOL = [(d1, d2) for d1 in W for d2 in W]

def resp(d1, d2, i1, i2, n=2.0, a=1.0):
    pool = sum((p1 * i1 + p2 * i2) ** n for p1, p2 in POOL) / len(POOL)
    return (d1 * i1 + d2 * i2) ** n / (a ** n + pool)

if __name__ == "__main__":
    print("balanced neuron (d1 = d2 = 1), both inputs at intensity I:")
    print("    I     R(1 only)  R(2 only)  R(both)   additivity   ME")
    for i in (0.05, 0.2, 0.5, 1, 2, 5, 20):
        r1, r2, rb = resp(1, 1, i, 0), resp(1, 1, 0, i), resp(1, 1, i, i)
        print(f"{i:6.2f} {r1:11.4f} {r2:10.4f} {rb:9.4f} {rb / (r1 + r2):11.2f}"
              f" {(rb - max(r1, r2)) / max(r1, r2):+7.0%}")
    print("neuron dominated by input 1 (d1 = 1, d2 = 0.25 or 0); input 1 fixed at 5,")
    print("adding a second input of strength I2 (response relative to input 1 alone):")
    for d2 in (0.25, 0.0):
        alone = resp(1, d2, 5, 0)
        row = "  ".join(f"I2 = {i2:g}: {resp(1, d2, 5, i2) / alone:.2f}"
                        for i2 in (1, 5, 20))
        print(f"  d2 = {d2:4.2f}:  {row}")
    print("same neuron, weak input 1 (0.3):",
          "  ".join(f"I2 = {i2:g}: {resp(1, 0.25, 0.3, i2) / resp(1, 0.25, 0.3, 0):.2f}"
                    for i2 in (0.3, 1)))
# balanced neuron (d1 = d2 = 1), both inputs at intensity I:
#     I     R(1 only)  R(2 only)  R(both)   additivity   ME
#   0.05      0.0025     0.0025    0.0100        2.00   +299%
#   0.20      0.0394     0.0394    0.1524        1.93   +287%
#   0.50      0.2286     0.2286    0.7619        1.67   +233%
#   1.00      0.7273     0.7273    1.7778        1.22   +144%
#   2.00      1.6000     1.6000    2.6667        0.83    +67%
#   5.00      2.4096     2.4096    3.1008        0.64    +29%
#  20.00      2.6490     2.6490    3.1936        0.60    +21%
# neuron dominated by input 1 (d1 = 1, d2 = 0.25 or 0); input 1 fixed at 5,
# adding a second input of strength I2 (response relative to input 1 alone):
#   d2 = 0.25:  I2 = 1: 0.86  I2 = 5: 0.50  I2 = 20: 0.20
#   d2 = 0.00:  I2 = 1: 0.78  I2 = 5: 0.32  I2 = 20: 0.05
# same neuron, weak input 1 (0.3): I2 = 0.3: 1.45  I2 = 1: 2.23

For a neuron that weights both inputs equally, the combined response at intensity 0.05 is 2.00 times the sum of the single responses (an enhancement of +299 %), 1.22 times at intensity 1, and only 0.60 times at intensity 20 (an enhancement of +21 %). The second part shows suppression: for a neuron driven mainly by input 1 ($d_2 = 0.25$), adding an equally strong input 2 when input 1 is strong (5) lowers the response to 0.50 of its value with input 1 alone, and a stronger input 2 (20) lowers it to 0.20, even though input 2 is excitatory for that neuron. When input 1 is weak (0.3), the same input 2 raises the response to 1.45 and 2.23 times. The same input enhances against a weak background and suppresses against a strong one.

Statistical criticisms of inverse effectiveness

Holmes (2007, 2009) argued that inverse effectiveness may be partly a statistical artefact of how data are analysed (Holmes 2007, 2009). He raised three points: first, if neurons or participants are grouped after the fact by their measured single-sense responses, regression to the mean by itself produces a “the weaker, the more enhanced” trend, and the remedy is to design discrete levels of stimulus intensity in advance; second, floor and ceiling effects limit how much enhancement and suppression can be seen; third, whether enhancement is measured in relative (percentage) or absolute terms affects whether inverse effectiveness appears. He concluded that, once these issues are taken into account, inverse effectiveness may disappear or even reverse in a given data set.

The program below turns the first point into a simulated behavioural experiment: 600 participants differ in accuracy with sight alone, but every one of them gains exactly +0.10 when sound is added; 40 trials per condition.

# Inverse effectiveness from regression to the mean (after Holmes 2007, 2009). 600
# simulated participants; true accuracy with sight alone p_V ~ U(0.45, 0.80); with sight
# and sound the gain is the SAME for everyone: p_VA = p_V + 0.10. 40 trials per condition
# per session. Participants are then sorted into thirds by unisensory accuracy.
import random

def mean(x):
    return sum(x) / len(x)

def score(rng, p, n=40):
    return sum(rng.random() < p for _ in range(n)) / n

if __name__ == "__main__":
    rng = random.Random(2007)
    people = []
    for _ in range(600):
        p = rng.uniform(0.45, 0.80)
        people.append({"p": p, "V1": score(rng, p), "V2": score(rng, p),
                       "VA": score(rng, p + 0.10)})
    def thirds(key, gain):
        s = sorted(people, key=key)
        return "   ".join(f"{mean([gain(q) for q in s[k * 200:(k + 1) * 200]]):+.3f}"
                           for k in range(3))
    print("true gain is +0.100 for everyone. Mean gain by third (weakest -> strongest):")
    rows = (("sorted by V1, gain = VA - V1 (same data):   ", "V1", "V1"),
            ("sorted by V1, gain = VA - V2 (fresh V data):", "V1", "V2"),
            ("sorted by the true p_V (design):            ", "p", "V1"))
    for label, k, base in rows:
        print("  " + label, thirds(lambda q: q[k], lambda q: q["VA"] - q[base]))
    v1, v2 = [q["V1"] for q in people], [q["V2"] for q in people]
    m1, m2 = mean(v1), mean(v2)
    cov = mean([(a - m1) * (b - m2) for a, b in zip(v1, v2)])
    var = (mean([(a - m1) ** 2 for a in v1]) * mean([(b - m2) ** 2 for b in v2])) ** 0.5
    print(f"test-retest reliability of the unisensory score: r = {cov / var:.2f}")
# true gain is +0.100 for everyone. Mean gain by third (weakest -> strongest):
#   sorted by V1, gain = VA - V1 (same data):    +0.149   +0.086   +0.052
#   sorted by V1, gain = VA - V2 (fresh V data): +0.096   +0.083   +0.094
#   sorted by the true p_V (design):             +0.095   +0.100   +0.091
# test-retest reliability of the unisensory score: r = 0.63

Grouped into thirds by single-sense scores from the same data, the weakest third gains 0.149 and the strongest only 0.052: textbook inverse effectiveness. But everyone’s true gain is 0.10. Computing the gain against an independent set of single-sense scores (while still grouping by the first set) gives 0.096, 0.083 and 0.094; grouping by the designed true ability gives 0.095, 0.100 and 0.091. In both cases the trend vanishes. The test–retest reliability of the single-sense score is only 0.63, exactly the condition under which regression to the mean does its work.

Chapter summary

The combined response of multisensory neurons in the superior colliculus can exceed the sum of the single responses, but such superadditivity appears mainly for very weak stimuli; with stronger stimuli additive and subadditive responses are more common. A smooth S-shaped nonlinearity, or the divisive normalisation of Ohshiro and colleagues, is enough to produce superadditivity, inverse effectiveness and subadditivity together, and the normalisation model also predicts suppression by an excitatory but non-optimal input. Near the floor of the response, superadditivity is hard to detect and easy to overestimate; grouping after the fact by single-sense responses can manufacture inverse effectiveness out of regression to the mean. In behaviour, the gain for audiovisual speech is largest at intermediate signal-to-noise ratios. Data from animal neurons cannot be carried directly over to human learning.