“Seeing” with the skin

If the brain assembles perception as flexibly as the previous chapters suggest, a natural question follows: can one sense take over the work of another? In 1969 Bach-y-Rita and colleagues gave the first systematic answer.

The framework puts “tongue electrode arrays” together with the 1969 experiment. Checking shows they must be distinguished: the 1969 device stimulated the skin of the back; electrotactile arrays on the tongue were developed later, and a 1998 technical note describes 49 electrodes (7 × 7) (Bach-y-Rita et al. 1998). The tongue’s advantages are that it is moist, conducts well and has fine tactile resolution, so lower voltages and currents suffice.

Other devices replace vision with sound. The vOICe system described by Meijer (1992) scans a camera image column by column from left to right: each column is a time slice, a pixel’s height in the column sets the pitch of a sound and its brightness the loudness; the original system handled images of 64 × 64 pixels with 16 grey levels (Meijer 1992), and one scan of an image typically takes about a second.

The visual cortex recruited by other senses

The framework says that “the visual cortex of blind people is activated by tactile or auditory input”. On the whole there is evidence for this, but two kinds of result must be kept apart: the studies of Sadato and Cohen concern primary visual cortex and Braille reading; the study of Amedi and colleagues concerns a shape-processing region of the lateral occipital cortex, and its participants included sighted people. Neither amounts to “restoring vision through sound or touch”.

What device parameters does such acuity require? The program below draws up a “resolution budget”. Snellen acuity 20/X means that the smallest resolvable detail (the gap in a letter E) subtends X/20 minutes of arc; a device that covers a camera field of $F$ degrees with $C$ columns has pixels $60F/C$ minutes of arc wide. Optimistically, one pixel can represent one detail; more realistically, two pixels are needed.

# The resolution budget of a sensory-substitution device. Snellen 20/X means the smallest
# resolvable detail (gap of a letter E) subtends X/20 arcmin. A device that samples the
# camera image with C columns over a horizontal field F deg has pixels of 60 F / C arcmin.
# Optimistic limit: detail = 1 pixel; realistic: 2 pixels (sampling). Devices: the vOICe
# (64 columns, Meijer 1992); a 20 x 20 tactile array (Bach-y-Rita et al. 1969).
WHO_BLIND = 400                                     # worse than 20/400: blindness (WHO)

def snellen(field_deg, columns, px_per_detail):
    return 20 * px_per_detail * 60 * field_deg / columns

if __name__ == "__main__":
    print("device          field  pixel (arcmin)  best 20/X: 1 px   2 px   beats 20/400?")
    for name, cols in (("vOICe 64 cols", 64), ("tactile 20x20", 20)):
        for field in (60, 30, 15, 7.5):
            x1, x2 = snellen(field, cols, 1), snellen(field, cols, 2)
            ok = "yes" if x2 < WHO_BLIND else ("only at 1 px" if x1 < WHO_BLIND else "no")
            print(f"{name:14s} {field:6.1f} {60 * field / cols:11.1f} {x1:15.0f}"
                  f" {x2:7.0f}   {ok}")
    need = 200 / 20 / 60 * 64 / 2                   # field: 20/200 at 2 px per detail
    print(f"field for 20/200 with 64 columns, 2 px per detail: {need:.1f} deg"
          f" ({60 / need:.1f}x zoom from 60 deg)")
    print("price of zoom: the 64 columns cover less of the scene, so finding an object")
    print("  takes more head or camera movements and more 1-second scans")
# device          field  pixel (arcmin)  best 20/X: 1 px   2 px   beats 20/400?
# vOICe 64 cols    60.0        56.2            1125    2250   no
# vOICe 64 cols    30.0        28.1             562    1125   no
# vOICe 64 cols    15.0        14.1             281     562   only at 1 px
# vOICe 64 cols     7.5         7.0             141     281   yes
# tactile 20x20    60.0       180.0            3600    7200   no
# tactile 20x20    30.0        90.0            1800    3600   no
# tactile 20x20    15.0        45.0             900    1800   no
# tactile 20x20     7.5        22.5             450     900   no
# field for 20/200 with 64 columns, 2 px per detail: 5.3 deg (11.2x zoom from 60 deg)
# price of zoom: the 64 columns cover less of the scene, so finding an object
#   takes more head or camera movements and more 1-second scans

With a 60° camera field, the 64 columns of the vOICe give pixels of about 56 minutes of arc; even in the best case this corresponds only to 20/1125, far from the blindness line of 20/400. To reach 20/200 with two pixels per detail, the field must shrink to about 5.3°, an 11-fold zoom of a 60° lens. The 20 × 20 tactile array never reaches 20/400 in any field in the table. Acuity with a substitution device is thus largely bought with field of view and time: with the lens zoomed in, the same 64 columns cover only a small patch of the scene, finding an object takes more turns of the head or the camera, and each scan takes about a second. The chart scores are real, but they are not the same thing as normal vision’s ability to take in a wide field at high resolution in a single glance.

Active movement: who moves the camera?

In his 1972 book Bach-y-Rita emphasised an observation: with the camera fixed, subjects could discriminate the stimuli only to a limited extent and felt them as vibrations on the skin; when subjects could move the camera themselves (left and right, up and down, zooming), their ability to recognise shapes improved greatly and they gradually came to experience objects as located in external space (Bach-y-Rita 1972).

Why does active movement matter so much? The program below is a toy model. The user must learn a parameter $g$: how far the sound or touch image moves for each degree the camera turns. On each trial the camera turns $m$ degrees and the image moves $y = gm$ plus noise. The active user turned the camera and knows $m$ (to within 2°); the passive user’s camera was turned by someone else, and their guess of $m$ is off by 15°. Both estimate $g$ by least squares from the $m$ they believe and the $y$ they observe.

# Why moving the camera yourself helps (toy model). To use a substitution device the
# user must learn the gain g (image shift per degree of camera turn). Each trial the
# camera turns m deg (sd 15) and the sound or touch image shifts y = g m + noise (sd 3).
# ACTIVE: the user turned it, so m is known to within 2 deg (motor command + body sense).
# PASSIVE: someone else turned it; the user's only guess of m has sd 15 deg.
# The user fits y = g m by least squares on the m they believe. True g = 1.
import random

def learn(rng, n, sd_known, g=1.0):
    num = den = 0.0
    for _ in range(n):
        m = rng.gauss(0, 15)
        y = g * m + rng.gauss(0, 3)
        m_seen = m + rng.gauss(0, sd_known)
        num += m_seen * y; den += m_seen * m_seen
    return num / den

if __name__ == "__main__":
    rng = random.Random(1963)
    print("trials   active: mean g   spread   |   passive: mean g   spread")
    for n in (10, 40, 160, 640):
        a = [learn(rng, n, 2.0) for _ in range(2000)]
        p = [learn(rng, n, 15.0) for _ in range(2000)]
        sd = lambda x: (sum((v - sum(x) / len(x)) ** 2 for v in x) / (len(x) - 1)) ** 0.5
        print(f"{n:6d} {sum(a) / len(a):14.3f} {sd(a):8.3f}   | {sum(p) / len(p):15.3f}"
              f" {sd(p):8.3f}")
    print(f"theory for passive learning: g x 15^2 / (15^2 + 15^2) = {225 / 450:.3f};"
          f" active: {225 / (225 + 4):.3f}")
    err = lambda g: abs(g - 1.0) * 20
    print(f"error pointing at an object 20 deg away: active {err(225 / 229):.1f} deg,"
          f" passive {err(0.5):.1f} deg")
# trials   active: mean g   spread   |   passive: mean g   spread
#     10          0.983    0.082   |           0.502    0.184
#     40          0.982    0.039   |           0.500    0.084
#    160          0.983    0.019   |           0.498    0.041
#    640          0.983    0.009   |           0.500    0.021
# theory for passive learning: g x 15^2 / (15^2 + 15^2) = 0.500; active: 0.983
# error pointing at an object 20 deg away: active 0.3 deg, passive 10.0 deg

The active user’s estimate averages 0.983, close to the true value of 1, and steadies as trials accumulate (standard deviation from 0.082 down to 0.009). The passive user’s estimate stays at 0.50, and no number of trials improves it: regressing on a noisy predictor systematically flattens the slope (attenuation bias, in statistical terms), by the ratio of the true variance of $m$ to its total variance, here one half. Having learned only half of $g$, the user points 10° off at an object 20° away. The model is of course too simple: a passive learner who knew the size of their own error could correct for it, and real learning is not the estimation of a single parameter. It makes just one point: self-generated movement supplies a known input, which lets the mapping between action and sensation be learned accurately.

Chapter summary

The tactile vision-substitution device of Bach-y-Rita in 1969 stimulated the skin of the back; electrotactile arrays on the tongue came later; the auditory device vOICe codes height as pitch and brightness as loudness. The visual cortex of blind people can be activated by Braille reading and contributes functionally to tactile tasks, and a shape-processing region of the lateral occipital cortex can be activated by sound-coded shapes. With a substitution device, congenitally blind people can score better than the blindness line on an acuity chart, but only by narrowing the field and lengthening the scan. Active control of the camera matters for learning the mapping between action and sensation. All these studies are small, and claims of “really seeing” must be qualified. The next chapter turns to ancient texts on the senses.