Chapter 05: Sensation and Perception

IN HOSPITALS ACROSS THE COUNTRY, Neurology Grand Rounds is a weekly event. There, staff neurologists, internists, and residents gather to review the most puzzling and unusual cases being treated on the ward. In Portland, Oregon, the head of neurology presented such a case. He was not puzzled about what had caused his patient’s problem. That was clear. The patient, P.T., had suffered a cerebral vascular accident, commonly known as a stroke. In fact, he had sustained two strokes. The first, suffered 6 years previously, had been a left hemisphere stroke. The patient had shown a nearly complete recovery from that stroke. P.T. had suffered a second stroke a few months before, however, and the CT scan showed that the damage was in the right hemisphere. This finding was consistent with the patient’s experience of left-sided weakness, although the weakness had mostly subsided after a month.

The unusual aspect of P.T.’s case was the collection of symptoms he continued to experience 4 months later. As he tried to resume the daily routines required on his small family farm, P.T. had particular difficulty recognizing familiar places and objects. While working on a stretch of fence, for example, he might look out over the hills and suddenly realize that he did not know the landscape. It was hard for him to pick out individual dairy cows—a matter of concern lest he attempt to milk a bull! Disturbing as this was, it was not the worst of his problems. Most troubling of all, he no longer recognized the people around him, including his wife. He had no trouble seeing her and could accurately describe her actions, but when it came to identifying her, he was at a complete loss. She was completely unrecognizable to him! He knew that her parts—body, legs, arms, and head—formed a person, but P.T. failed to see these parts as belonging to a specific individual. This deficit was not limited to P.T.’s wife; he had the same problem with other members of his family and friends from his small town, a place he had lived for 66 years.

A striking feature of P.T.’s impairment was that his inability to recognize objects and people was limited to the visual modality. As soon as his wife spoke, he immediately recognized her voice. Indeed, he claimed that, on hearing her voice, the visual percept of her would “fall into place.” The shape in front of him would suddenly morph into his wife. In a similar fashion, he could recognize specific objects by touching, smelling, or tasting them.

The overarching reason why you are sitting here reading this book today is that you had ancestors who successfully survived their environment and reproduced. One reason they were able to do this was their ability to sense and perceive things that could be threatening to their survival and then act on those perceptions. Pretty obvious, right? Less obvious is that most of these perceptions and behavioral responses never even reach people’s conscious awareness, and what does reach our awareness is not an exact replica of the stimulus. This latter phenomenon becomes more evident when we are presented with optical illusions (as we see later in the chapter). Perception begins with a stimulus from the environment, such as sound or light, which stimulates one of the sense organs such as the ear or eye. The input from the sound or light wave is transduced into neural activity by the sense organ and sent to the brain for processing. Sensation refers to the early processing that goes on. The mental representation of that original stimulus, which results from the various processing events, whether it accurately reflects the stimulus or not, is called a percept. Thus, perception is the process of constructing the percept.

Our senses are our physiological capacities to provide input from the environment to our neurological system. Hence, our sense of sight is our capacity to capture light waves on the retina, convert them into electrical signals, and ship them on for further processing. We tend to give most of the credit for our survival to our sense of sight, but it does not operate alone. For instance, the classic “we don’t have eyes in the back of our head” problem means we can’t see the bear sneaking up behind us. Instead, the rustling of branches or the snap of a twig warns us. We do not see particularly well in the dark either, as many people know after stubbing a toe when groping about to find the light switch. And though the milk may look fine, one sniff tells you to dump it down the drain. Although these examples illustrate the interplay of senses on the conscious level, neuroimaging studies have helped to reveal that extensive interaction takes place between the sensory modalities much earlier in the processing pathways than was previously imagined.

In normal perception, all of the senses are critical. Effectively and safely driving a car down a busy highway requires the successful integration of seeing, touch, hearing, and perhaps even smell (warning, for example, that you have been riding the brakes down a hill). Enjoying a meal also involves the interplay of the senses. We cannot enjoy food intensely without smelling its fragrance. The sense of touch is an essential part of our gastronomic experience also, even if we don’t think much about it. It gives us an appreciation for the texture of the food: the creamy smoothness of whipped cream or the satisfying crunch of an apple. Even visual cues enhance our gustatory experience—a salad of green, red, and orange hues is much more enticing than one that is brown and black.

In this chapter, we begin with an overview of sensation and perception and then turn to a description of what is known about the anatomy and function of the individual senses. Next we tackle the issue of how information from our different sensory systems is integrated to produce a coherent representation of the world. We end by discussing the interesting phenomenon of synesthesia—what happens when sensory information is more integrated than is usual.

Before dealing with each sense individually, let’s look at the anatomical and processing features that the sensory systems have in common. Each system begins with some sort of anatomical structure for collecting, filtering, and amplifying information from the environment. For instance, the outer ear, the ear canal, and inner ear concentrate and amplify sound. In vision, the muscles of the eye direct the gaze, the pupil size is adjusted to filter the amount of light, and the cornea and lens refract light to focus it on the retina. Each system has specialized receptor cells that transduce the environmental stimulus, such as sound waves or light waves or chemicals, into neural signals. These neural signals are passed along their specific sensory nerve pathways: the olfactory signals via the olfactory nerve (first cranial nerve); visual signals via the optic nerve (second cranial nerve); auditory signals via the cochlear nerve (also called the auditory nerve, which joins with the vestibular nerve to form the eighth cranial nerve); taste via the facial and glossopharyngeal nerves (seventh and ninth cranial nerves); facial sensation via the trigeminal nerve (fifth cranial nerve); and sensation for the rest of the body via the sensory nerves that synapse in the dorsal roots of the spinal cord.

The sensory nerves from the body travel up the spinal cord and enter the brain through the medulla, where the glossopharyngeal and vestibulocochlear nerves also enter. The facial nerve enters the brainstem at the pontomedullary junction. The trigeminal nerve enters at the level of the pons. These nerves all terminate in different parts of the thalamus (Figure 5.1). The optic nerve travels from the eye socket to the optic chiasm, where fibers from the nasal visual fields cross to the opposite side of the brain, and most (not all) of the newly combined fibers terminate in the thalamus. From the thalamus, neural connections from each of these pathways travel first to what are known as primary sensory cortex, and then to secondary sensory cortex (Figure 5.1). The olfactory nerve is a bit of a rogue. It is the shortest cranial nerve and follows a different course. It terminates in the olfactory bulb, and axons extending from here course directly to the primary and secondary olfactory cortices without going through the brainstem or the thalamus.

Across the senses, receptor cells share a few general properties. Receptor cells are limited in the range of stimuli that they respond to, and as part of this limitation, their capability to transmit information has only a certain degree of precision. Receptor cells do not become active until the stimulus exceeds some minimum intensity level. They are not fixed entities, but rather adapt as the environment changes.

Range Each sensory modality responds to a limited range of stimuli. Most people’s impression is that human color vision is unlimited. However, there are many colors, or parts of the electromagnetic spectrum, that we cannot see (Figure 5.2). Our vision is limited to a small region of this spectrum, wavelengths of light in the range of 400 to 700 nanometers (nm). Individual receptor cells respond to just a portion of this range. This range is not the same for all species. For example, birds and insects have receptors that are sensitive to shorter wavelengths and thus, can see ultraviolet light (Figure 5.2b, right). Some bird species actually exhibit sexual dichromatism (the male and female have different coloration) that is not visible to humans. Similar range differences are found in audition. We are reminded of this when we blow a dog whistle (invented by Francis Galton, Charles Darwin’s cousin). We immediately have the dog’s attention, but we cannot hear the high-pitched sound ourselves. Dogs can hear sound-wave frequencies of up to about 60 kilohertz (kHz), but we hear only sounds below about 20 kHz. Although a dog has better night vision than we do, we see more colors. Dogs cannot see the red–green spectrum. As limited as our receptor cells may be, we do respond to a wide range of stimulus intensities. The threshold stimulus value is the minimum stimulus that will activate a percept.

Adaptation Adaptation refers to how sensory systems stay fine-tuned. It is the adjusting of the sensitivity of the sensory system to the current environment and to important changes in the environment. You will come to see that perception is mainly concerned with changes in sensation. This makes good survival sense. Adaptation happens quickly in the olfactory system. You smell the baking bread when you walk into the bakery, but the fragrance seems to evaporate quickly. Our auditory system also adapts rather quickly. When we first turn the key to start a car, the sound waves from the motor hit our ears, activating sensory neurons. But this activity soon stops, even though the stimulus continues as we drive along the highway. Some neurons continue to fire as long as the stimulus continues, but their rate of firing slows down: the longer the stimulus continues, the less frequent the action potentials are. The noise of the computer drops into the background, and we have “adapted” to it.

Visual system adaptation also occurs for changes in the light intensity in the environment. We frequently move between areas with different light intensities, for instance, when walking from a shaded area into the bright sunlight. It takes some time for the eyes to reset to the ambient light conditions, especially when going from bright light into darkness. When you go camping for the first time with veteran campers, one of the first things you are going to be told is not to shine your flashlight into someone’s eyes. It would take about 20–30 minutes for that person to regain her “night vision,” that is, to regain sensitivity to the low level of ambient light after being exposed to the bright light. We discuss how this works later, in the Vision section.

Acuity Our sensory systems are tuned to respond to different sources of information in the environment. Light activates receptors in the retina, pressure waves produce mechanical and electrical changes in the eardrum, and odor molecules are absorbed by receptors in the nose. How good we are at distinguishing among stimuli within a sensory modality, or what we would call acuity, depends on a couple of factors. One is simply the design of the stimulus collection system. Dogs can adjust the position of their two ears independently to better capture sound waves. This design contributes to their ability to hear sounds that are up to four times farther away than humans are capable of hearing. Another factor is the number and distribution of the receptors. For instance, for touch, we have many more receptors on our fingers than we do on our back; thus, we can discern stimuli better with our fingers. Our visual acuity is better than that of most animals, but not better than an eagle. Our acuity is best in the center of our visual field, because the central region of the retina, the fovea, is packed with photoreceptors. The farther away from the fovea, the fewer the receptors. The same is true for the eagle, but he has two foveas.

In general, if a sensory system devotes more receptors to certain types of information (e.g., as in the sensory receptors of the hands), there is a corresponding increase in cortical representation of that information (see, for example, Figure 5.16). This finding is interesting, because many creatures carry out exquisite perception without a cortex. So what is our cortex doing with all of the sensory information? The expanded sensory capabilities in humans, and mammals in general, are probably not for better sensation per se; rather, they allow that information to support flexible behavior, due to greatly increased memory capacity and pathways linking that information to our action and attention systems.

Sensory Stimuli Share an Uncertain Fate The physical stimulus is transduced into neural activity (i.e., electrochemical signals) by the receptors and sent through subcortical and cortical regions of the brain to be processed. Sometimes a stimulus may produce subjective sensory awareness. When that happens, the stimulus is not the only factor contributing to the end product. Each level of processing—including attention, memory, and emotional systems—contributes as well. Even with all of this activity going on, most of the sensory stimulation never reaches the level of consciousness. No doubt if you close your eyes right now, you will not be able to describe everything that is in front of you, although it has all been recorded on your retina.

Connective Similarities Most people typically think of sensory processing as working in one direction; that is, information moves from the sensor organs to the brain. Neural activity, however, is really a two-way street. At all levels of the sensory pathways, neural connections are going in both directions. This feature is especially pronounced at the interface between the subcortex and cortex. Sensory signals from the visual, auditory, somatosensory, and gustatory (taste) systems all synapse within the thalamus before projecting onto specific regions within the cortex. The visual pathway passes through the lateral geniculate nucleus (LGN) of the thalamus, the auditory system through the medial geniculate nucleus (MGN), the somatic pathway through the ventral posterior nuclear complex and the gustatory pathway through the ventral posteromedial nucleus. Just exactly what is going on in the thalamus is unclear. It appears to be more than just a relay station. Not only are there projections from these nuclei to the cortex, but the thalamic nuclei are interconnected, providing an opportunity for multisensory integration, an issue we turn to later in the chapter. The thalamus also receives descending, or feedback, connections from primary sensory regions of the cortex as well as other areas of the cortex, such as the frontal lobe. These connections appear to provide a way for the cortex to control, to some degree, the flow of information from the sensory systems (see Chapter 7).

Now that we have a general idea of what is similar about the anatomy of the various sensory systems and processing of sensory stimuli, let’s take a closer look at the individual sensory systems.

Imagine you are out walking to your car late at night, and you hear a rustling sound. Your ears (and heart!) are working on overdrive, trying to determine what is making the sound (or more troubling, who) and where the sound is coming from. Is it merely a tree branch blowing in the breeze, or is someone sneaking up behind you? The sense of hearing, or audition, plays an important role in our daily lives. Sounds can be essential for survival—we want to avoid possible attacks and injury—but audition also is fundamental for communication. How does the brain process sound? What happens as sound waves enter the ear? And how does our brain interpret these signals? More specifically, how does the nervous system figure out the what and the where of sound sources?

Figure 5.3 presents an overview of the auditory pathways. The complex structures of the inner ear provide the mechanisms for transforming sounds (variations in sound pressure) into neural signals. This is how hearing works: Sound waves arriving at the ear enter the auditory canal. Within the canal, the sound waves are amplified, similar to what happens when you honk your car’s horn in a tunnel. The waves travel to the far end of the canal, where they hit the tympanic membrane, or eardrum, and make it vibrate. These low-pressure vibrations then travel through the air-filled middle ear and rattle three tiny bones, the malleus, incus, and stapes, which cause a second membrane, the oval window, to vibrate.

The oval window is the “door” to the fluid-filled cochlea, the critical auditory structure of the inner ear. Within the cochlea are tiny hair cells located along the inner surface of the basilar membrane. The hair cells are the sensory receptors of the auditory system. Hair cells are composed of up to 200 tiny filaments known as stereocilia that float in the fluid. The vibrations at the oval window produce tiny waves in the fluid that move the basilar membrane, deflecting the stereocilia. The location of a hair cell on the basilar membrane determines its frequency tuning, the sound frequency that it responds to. This is because the thickness (and thus, the stiffness) of the basilar membrane varies along its length from the oval window to the apex of the cochlea. The thickness constrains how the membrane will move in response to the fluid waves. Near the oval window, the membrane is thick and stiff. Hair cells attached here can respond to high-frequency vibrations in the waves. At the other end, the apex of the cochlea, the membrane is thinner and less stiff. Hair cells attached here will respond only to low frequencies. This spatial arrangement of the sound receptors is known as tonotopy, and the arrangement of the hair cells along the cochlear canal form a tonotopic map. Thus, even at this early stage of the auditory system, information about the sound source can be discerned.

The hair cells act as mechanoreceptors. When deflected by the membrane, mechanically gated ion channels open in the hair cells, allowing positively charged ions of potassium and calcium to flow into the cell. If the cell is sufficiently depolarized, it will release transmitter into a synapse between the base of the hair cell and an afferent nerve fiber. In this way, a mechanical event, the deflections of the hair cells, is converted into a neural signal (Figure 5.4).

Natural sounds like music or speech are made up of complex frequencies. Thus, a natural sound will activate a broad range of hair cells. Although we can hear sounds up to 20,000 hertz (Hz), our auditory system is most sensitive to sounds in the range of 1000 to 4000 Hz, a range that carries much of the information critical for human communication, such as speech or the cries of a hungry infant. Other species have sensitivity to very different frequencies. Elephants can hear very low-frequency sounds, allowing them to communicate over long distances (since such sounds are only slowly distorted by distance); mice communicate at frequencies well outside our hearing system. These species-specific differences likely reflect evolutionary pressures that arose from the capabilities of different animals to produce sounds. Our speech apparatus has evolved to produce changes in sound frequencies in the range of our highest sensitivity.

The auditory system contains several synapses between the hair cells and the cortex. The cochlear nerve, also called the auditory nerve, projects to the cochlear nucleus in the medulla. Axons from this nucleus travel up to the pons and split to innervate the left and right olivary nucleus, providing the first point within the auditory pathways where information is shared from both ears. Axons from the cochlear and olivary nuclei project to the inferior colliculus, higher up in the midbrain. At this stage, the auditory signals can access motor structures; for example, motor neurons in the colliculus can orient the head toward a sound. Some of the axons coursing through the pons branch off to the nucleus of the lateral lemniscus in the midbrain, where another important characteristic of sound, timing, is processed. From the midbrain, auditory information ascends to the MGN in the thalamus, which in turn projects to the primary auditory cortex (A1) in the superior part of the temporal lobe.

Neurons throughout the auditory pathway continue to have frequency tuning and maintain their tonotopic arrangement as they travel up to the cortex. As described in Chapter 2 (p. 56), the primary auditory cortex contains a tonotopic map, an orderly correspondence between the location of the neurons and their specific frequency tuning. Cells in the rostral part of A1 tend to be responsive to low-frequency sounds; cells in the caudal part of A1 are more responsive to high-frequency sounds. The tonotopic organization is evident in studies using single-cell recording methods, and thanks to the resolution provided by fMRI, it can also be seen in humans (Figure 5.5). Tonotopic maps are also found in secondary auditory areas of the cortex.

As Figure 5.6 shows, the tuning curves for auditory cells can be quite broad. The finding that individual cells do not give precise frequency information but provide only coarse coding may seem puzzling, because animals can differentiate between very small differences in sound frequencies. Interestingly, the tuning of individual neurons becomes sharper as we move through the auditory system. A neuron in the cat’s cochlear nucleus that responds maximally to a pure tone of 5000 Hz may also respond to tones ranging from 2000 to 10,000 Hz. A comparable neuron in the cat auditory cortex responds to a much narrower range of frequencies. The same principle is observed in humans. In one study, electrodes were placed in the auditory cortex of epileptic patients to monitor for seizure activity (Bitterman et al., 2008). Individual cells were exquisitely tuned, showing a strong response to, say, a tone at 1010 Hz but no response, or even a slight inhibition to tones just 20 Hz different. This fine resolution is essential for making the precise discriminations for perceiving sounds, including speech. Indeed, it appears that human auditory tuning is sharper than that of all other species except for the bat.

While A1 is, at a gross level, tonotopically organized, more recent studies using high-resolution imaging methods in the mouse suggest that, at a finer level of resolution, organization may be much more messy. At this level, adjacent cells frequently show very different tuning. Thus, there is a large-scale tonotopic organization but with considerable heterogeneity at the local level (Bandyopadhyay et al., 2010; Rothchild et al., 2010). This mixture may reflect the fact that natural sounds contain information across a broad range of frequencies and that the local organization arises from experience with these sounds.

Frequency data are essential for deciphering a sound. Sound-producing objects have unique resonant properties that provide a characteristic signature. The same note played on a clarinet and a trumpet will sound differently, because the resonant properties of each instrument will produce considerable differences in the note’s harmonic structure. Yet, we are able to identify a “G” from different instruments as the same note. This is because the notes share the same base frequency. In a similar way, we produce our range of speech sounds by varying the resonant properties of the vocal tract. Movements of our lips, tongue, and jaw change the frequency content of the acoustic stream produced during speech. Frequency variation is essential for a listener to identify words or music.

Auditory perception does not merely identify the content of an acoustic stimulus. A second important function of audition is to localize sounds in space. Consider the bat, which hunts by echolocation. High-pitched sounds are emitted by the bat and bounce back, as echoes from the environment. From these echoes, the bat’s brain creates an auditory image of the environment and the objects within it—preferably a tasty moth. But knowing that a moth (“what”) is present will not lead to a successful hunt. The bat also has to determine the moth’s precise location (“where”). Some very elegant work in the neuroscience of audition has focused on the “where” problem. In solving the “where” problem, the auditory system relies on integrating information from the two ears.

In developing animal models to study auditory perception, neuroscientists select animals with well-developed hearing. A favorite species for this work has been the barn owl, a nocturnal creature. Barn owls have excellent scotopia (night vision), which guides them to their prey. Barn owls, however, also must use an exquisitely tuned sense of hearing to locate food, because visual information can be unreliable at night. The low levels of illumination provided by the moon and stars fluctuate with the lunar cycle and clouds. Sound, such as the patter of a mouse scurrying across a field, offers a more reliable stimulus. Indeed, barn owls have little trouble finding prey in a completely dark laboratory.

Barn owls rely on two cues to localize sounds: the difference in when a sound reaches each of the two ears, the interaural time, and the difference in the sound’s intensity at the two ears. Both cues exist because the sound reaching two ears is not identical. Unless the sound source is directly parallel to the head’s orientation, the sound will reach one ear before the other. Moreover, because the intensity of a sound wave becomes attenuated over time, the magnitude of the signal at the two ears will not be identical. The time and intensity differences are minuscule. For example, if the stimulus is located at a 45° angle to the line of sight, the interaural time difference will be approximately 1/10,000 of a second. The intensity differences resulting from sound attenuation are even smaller—indistinguishable from variations due to “noise.” However, these small differences are amplified by a unique asymmetry of owl anatomy: The left ear is higher than eye level and points downward, and the right ear is lower than eye level and points upward. Because of this asymmetry, sounds coming from below are louder in the left ear than the right. Humans do not have this asymmetry, but the complex structure of the human outer ear, or pinna, amplifies the intensity difference between a sound heard at the two ears (Figure 5.7).

Interaural time and intensity differences provide independent cues for sound localization. To show this, researchers use little owl headphones. Stimuli are presented over headphones, and the owl is trained to turn its head in the perceived direction of the sound. The headphones allow the experimenter to manipulate each cue separately. When amplitude is held constant, asynchronies in presentation times prompt the owl to shift its head in the horizontal plane. Variations in amplitude produce vertical head movements. Combining the two cues by fusing the inputs from the two ears provides the owl with a complete representation of three-dimensional space. If one ear is plugged, the owl’s response indicates that a sound has been detected, but it cannot localize the source.

Mark Konishi of the California Institute of Technology has provided a well-specified neural model of how neurons in the brainstem of the owl code interaural time differences by operating as coincidence detectors (M. Konishi, 1993). To be activated, these neurons must simultaneously receive input from each ear. In computer science terms, these neurons act as AND operators. An input from either ear alone or in succession is not sufficient; the neurons will fire only if an input arrives at the same time from both ears.

To see how this model works, look at Figure 5.8. In Figure 5.8a, the sound source is directly in front of the animal. In this situation the coincidence detector in the middle is activated, because the stimulus arrives at each ear at the same time. In Figure 5.8b, the sound source is to the animal’s left. This gives the axon from the left ear a slight head start. Simultaneous activation now occurs in a coincidence detector to the left of center. This simple arrangement provides the owl with a complete representation of the horizontal position of the sound source.

FIGURE 5.7 Variation in pinnae.
The shape of the pinnae help filter sounds and can amplify differences in the stimulus at the two ears. Considerable variation is seen across species. (a) Great Horned Owl, (b) Fennec Fox, and (c) human.

FIGURE 5.8 Slight asymmetries in the arrival times at the two ears can be used to locate the lateral position of a stimulus.
(a) When the sound source is directly in front of the owl, the stimulus will reach the two ears at the same time. As activation is transmitted across the delay lines, the coincidence detector representing the central location will be activated simultaneously from both ears. (b) When the sound source is located to the left, the sound reaches the left ear first. Now a coincidence detector offset to the opposite side receives simultaneous activation from the two ears.

A different coding scheme represents interaural intensities. For this stimulus dimension, the neural code is based on the input’s firing rate. The stronger the input signal, the more strongly the cell fires. Neurons sum the combined intensity signals from both ears to pinpoint the vertical position of the source.

In Konishi’s model, the problem of sound localization by the barn owl is solved at the level of the brainstem. To date, this theory has not explained higher stages of processing, such as in the auditory cortex. Perhaps cortical processing is essential for converting location information into action. The owl does not want to attack every sound it hears; it must decide if the sound is generated by potential prey. Another way of thinking about this is to reconsider the issues surrounding the computational goals of audition. Konishi’s brainstem system provides the owl with a way to solve “where” problems but has not addressed the “what” question. The owl needs a more detailed analysis of the sound frequencies to determine whether a stimulus results from the movement of a mouse or a deer.

We have the greatest awareness of our senses of sight, sound, taste, and touch. Yet the more primitive sense of smell is, in many ways, equally essential for our survival. Although the baleen whale probably does not smell the tons of plankton it ingests, the sense of smell is essential for terrestrial mammals, helping them to recognize foods that are nutritious and safe. Olfaction may have evolved primarily as a mechanism for evaluating whether a potential food is edible, but it serves other important roles as well—for instance, in avoiding hazards, such as fire or airborne toxins. Olfaction also plays an important role in social communication. Pheromones are excreted or secreted chemicals perceived by the olfactory system that trigger a social response in another individual of the same species. Pheromones are well documented in some insects, reptiles, and mammals. It also appears that they play an important role in human social interactions. Odors generated by women appear to vary across the menstrual cycle, and we are all familiar with the strong smells generated by people coming back from a long run. The physiological responses to such smells may be triggered by pheromones. To date, however, no compounds or receptors have been identified in humans. Before discussing the functions of olfaction, let’s review the neural pathways of the brain that respond to odors.

Smell is the sensory experience that results from the transduction of neural signals triggered by odor molecules, or odorants. These molecules enter the nasal cavity, either during the course of normal breathing or when we sniff. They will also flow into the nose passively, because air pressure in the nasal cavity is typically lower than in the outside environment, creating a pressure gradient. Odorants can also enter the system through the mouth, traveling back up into the nasal cavity (e.g., during consumption of food).

How olfactory receptors actually “read” odor molecules is unknown. One popular hypothesis is that odorants attach to odor receptors, which are embedded in the mucous membrane of the roof of the nasal cavity, called the olfactory epithelium. There are over 1,000 types of receptors, and most of these respond to only a limited number of odorants, though a single odorant can bind to more than one type of receptor. Another hypothesis is that the molecular vibrations of groups of odorant molecules contribute to odor recognition (Franco et al., 2011; Turin, 1996). This model predicts that odorants with similar vibrational spectra should elicit similar olfactory responses, and it explains why similarly shaped molecules, but with dissimilar vibrations, have very different fragrances. For example, alcohols and thiols have almost exactly the same structure, but alcohols have a fragrance of, well, alcohol, and thiols smell like rotten eggs.

Figure 5.9 details the olfactory pathway. The olfactory receptor is called a bipolar neuron because appendages extend from opposite sides of its cell body. When an odorant triggers the neuron, whether by shape or vibration, a signal is sent to the neurons in the olfactory bulbs, called the glomeruli. Tremendous convergence and divergence take place in the olfactory bulb. One bipolar neuron may activate over 8,000 glomeruli, and each glomerulus, in turn, receives input from up to 750 receptors. The axons from the glomeruli then exit laterally from the olfactory bulb, forming the olfactory nerve. Their destination is the primary olfactory cortex, or pyriform cortex, located at the ventral junction of the frontal and temporal cortices. The olfactory pathway to the brain is unique in two ways. First, most of the axons of the olfactory nerve project to the ipsilateral cortex. Only a small number cross over to innervate the contralateral hemisphere. Second, unlike the other sensory nerves, the olfactory nerve arrives at the primary olfactory cortex without going through the thalamus. The primary olfactory cortex projects to a secondary olfactory area within the orbitofrontal cortex, as well as making connections with other brain regions including the thalamus, hypothalamus, hippocampus, and amygdala. With these wide-ranging connections, it appears that odor cues influence autonomic behavior, attention, memory, and emotions—something that we all know from experience.

FIGURE 5.10 Sniffing and smelling.
(a) This special device was constructed to deliver controlled odors during fMRI scanning. (b, top) Regions activated during sniffing. The circled region includes the primary olfactory cortex and a posteromedial region of the orbitofrontal cortex. (b, bottom) Regions more active during sniffing when an odor was present compared to when the odor was absent.

Olfaction has gotten short shrift from cognitive neuroscientists. This neglect reflects, in part, our failure to appreciate the importance of olfaction in people’s lives: We have handed the sniffing crown over to bloodhounds and their ilk. In addition, some thorny technical challenges must be overcome to apply tools such as fMRI to study the human olfactory system. First is the problem of delivering odors to a participant in a controlled manner (Figure 5.10a). Nonmagnetic systems must be constructed to allow the odorized air to be directed at the participant’s nostrils while he is in the fMRI magnet. Second, it is hard to determine when an odor is no longer present. The chemicals that carry the odor can linger in the air for a long time. Third, although some odors overwhelm our senses, most are quite subtle, requiring exploration through the act of sniffing to detect and identify. Whereas it is almost impossible to ignore a sound, we can exert considerable control over the intensity of our olfactory experience.

Noam Sobel of the Weizmann Institute in Israel developed methods to overcome these challenges, conducting neuroimaging studies of olfaction that have revealed an intimate relationship between smelling and sniffing (Mainland & Sobel, 2006; Sobel et al., 1998). Participants were scanned while being exposed to either nonodorized, clean air or one of two chemicals: vanillin or decanoic acid. The former has a fragrance like vanilla, the latter, like crayons. The odor-absent and odor-present conditions alternated every 40 seconds. Throughout the scanning session, the instruction, “Sniff and respond, is there an odor?” was presented every 8 seconds. In this manner, the researchers sought to identify areas in which brain activity was correlated with sniffing versus smelling (Figure 5.10b).

Surprisingly, smelling failed to produce consistent activation in the primary olfactory cortex. Instead, the presence of the odor produced a consistent increase in the fMRI response in lateral parts of the orbitofrontal cortex, a region typically thought to be a secondary olfactory area. Activity in the primary olfactory cortex was closely linked to the rate of sniffing. Each time the person took a sniff, the fMRI signal increased regardless of whether the odor was present. These results seemed quite puzzling and suggested that the primary olfactory cortex might be more a part of the motor system for olfaction.

Upon further study, however, the lack of activation in the primary olfactory cortex became clear. Neurophysiological studies of the primary olfactory cortex in the rat had shown that these neurons habituate (adapt) quickly. It was suggested that perhaps the primary olfactory cortex lacks a smell-related response because the hemodynamic response measured by fMRI exhibits a similar habituation. To test this idea, Sobel’s group modeled the fMRI signal by assuming a sharp increase followed by an extended drop after the presentation of an odor—an elegant example of how single-cell results can be used to interpret imaging data. When analyzed in this manner, the hemodynamic response in the primary olfactory cortex was found to be related to smell as well as to sniffing. These results suggest that the role of the primary olfactory cortex might be essential for detecting a change in the external odor and that the secondary olfactory cortex plays a critical role in identifying the odor itself. Each sniff represents an active sampling of the olfactory environment, and the primary olfactory cortex plays a critical role in determining if a new odor is present.

The importance of sniffing for olfactory perception is underscored by the fact that our ability to smell is continually being modulated by changes in the size of the nasal passages. In fact, the two nostrils appear to switch back and forth—one is larger than the other for a number of hours, and then the reverse. These changes have a profound effect on how smell is processed (Figure 5.11). Why might the nose behave this way?

The olfactory percept depends not only on how intense the odor is but also on how efficiently we sample it (Mozell et al., 1991). The presence of two nostrils of slightly different sizes provides the brain with slightly different images of the olfactory environment. To test the importance of this asymmetry, Sobel monitored which nostril was allowing high airflow and which nostril was allowing low airflow, while presenting odors with both high and low absorption rates to each nostril. As predicted (see Figure 5.11), when sniffed through the high-airflow nostril, the odorant with a high absorption rate was judged to be more intense compared to when the same odorant was presented to the low-airflow nostril. The opposite was true for the odorant with a low absorption rate; here, the odor with a low rate of absorption was judged to be more intense when sniffed through the low-airflow nostril. Some of the participants were monitored when the flow rate of their nostrils reversed. The perception of the odorant presented to the same nostril reversed with the change in airflow.

As we saw in Chapter 4, asymmetrical representations are the rule in human cognition, perhaps providing a more efficient manner of processing complex information. With the ancient sense of olfaction, this asymmetry appears to be introduced at the peripheral level by modulation of the rate of airflow through the nostrils.

FIGURE 5.11 Human nostrils have asymmetric flow rates.
Although the same odorants enter each nostril, the response across the epithelium will be different for the two nostrils because of variation in flow rates. One nostril always has a greater input airflow than the other, and the nostrils switch between the two rates every few hours. This system of having one lowflow and one high-flow nostril has evolved to give the nose optimal accuracy in perceiving odorants that have both high and low rates of absorption.

The sense of taste depends greatly on the sense of smell. Indeed, the two senses are often grouped together because they both begin with a chemical stimulus. Because these two senses interpret the environment by discriminating between different chemicals, they are referred to as the chemical senses.

Gustation begins with the tongue. Strewn across the surface of the tongue in specific locations are different types of papillae, the little bumps you can feel on the surface. Papillae serve multiple functions. Some are concerned with gustation, others with sensation, and some with the secretion of lingual lipase, an enzyme that helps break down fats. The papillae in the anterior region and along the sides of the tongue contains several taste buds; those types found predominantly in the center of the tongue do not have taste buds. Taste pores are the conduits that lead from the surface of the tongue to the taste buds. Each taste bud contains many taste cells (Figure 5.12). Taste buds are also found in the cheeks and parts of the roof of the mouth. There are five basic tastes: salty, sour, bitter, sweet, and umami. Umami is the savory taste you experience when you eat steak or other protein-rich substances.

Sensory transduction in the gustatory system begins when a food molecule, or tastant, stimulates a receptor in a taste cell and causes the receptor to depolarize (Figure 5.12). Each of the basic taste sensations has a different form of chemical signal transduction. For example, the experience of a salty taste begins when the salt molecule (NaCl) breaks down into Na+ and Cl−, and the Na+ ion is absorbed by a taste receptor, leading the cell to depolarize. Other taste transduction pathways, such as sweet carbohydrate tastants, are more complex, involving receptor binding that does not lead directly to depolarization. Rather, the presence of certain tastants will initiate a cascade of chemical “messengers” that eventually leads to cellular depolarization. Synapsing with the taste cells in the taste buds are bipolar neurons. Their axons form the chorda tympani nerve.

The chorda tympani nerve joins other fibers to form the facial nerve (the 7th cranial nerve). This nerve projects to the gustatory nucleus, located in the rostral region of the nucleus of the solitary tract in the brainstem. Meanwhile, the caudal region of the solitary nucleus receives sensory neurons from the gastrointestinal tract. The integration of information at this level can provide a rapid reaction. For example, you might gag if you taste something that is “off,” a strong signal that the food should be avoided.

The next synapse in the gustatory system is on the ventral posterior medial nucleus (VPM) of the thalamus. Axons from the VPM synapse in the primary gustatory cortex. This is a region in the insula and operculum, structures at the intersection of the temporal and frontal lobes (Figure 5.12). Primary gustatory cortex is connected to secondary processing areas of the orbitofrontal cortex, providing an anatomical basis for the integration of tastes and smells. While there are only five types of taste cells, we are capable of experiencing a complex range of tastes. This ability must result from the integration of information conveyed from the taste cells and processed in areas like the orbitofrontal cortex.

The tongue does more than just taste. Some papillae contain nociceptive receptors, a type of pain receptor. These are activated by irritants such as capsaicin (contained in chili peppers), carbon dioxide (carbonated drinks), and acetic acid (vinegar). The output from these receptors follows a different path, forming the trigeminal nerve (cranial nerve V). This nerve not only carries pain information but also signals position and temperature information. You are well aware of the reflex response to activation by these irritants if you have ever eaten a hot chili: salivation, tearing, vasodilation (the red face), nasal secretion, bronchospasm (coughing), and decreased respiration. All these are meant to dilute that irritant and get it out of your system as quickly as possible.

FIGURE 5.12 The gustatory transduction pathway.
(a) Three different types of taste papillae span the surface of the tongue. Each cell is sensitive to one of five basic tastes: salty, sweet, sour, bitter, and umami. The bar graph shows how sensitivity for four taste sensations varies between the three papillae. (b) The papillae contain the taste buds. (c) Taste pores on the surface of the tongue open into the taste bud, which contains taste cells. (d) The chorda tympani nerve, formed by the axons from the taste cells, joins with the facial nerve to synapse in the nucleus of the solitary tract in the brain stem, as do the sensory nerves from the GI tract via the vagus nerve. The taste pathway projects to the ventral posterior medial nucleus of the thalamus and information is then relayed to the gustatory cortex in the insula.

Taste perception varies from person to person because the number and types of papillae and taste buds vary considerably between individuals. In humans, the number of taste buds varies from 120 to 668 per cm2. Interestingly, women generally have more taste buds than men (Bartoshuk et al., 1994). People with large numbers of taste buds are known as supertasters. They taste things more intensely, especially bitterness, and feel more pain from tongue irritants. You can spot the two ends of the tasting spectrum at the table. One is pouring on the salsa or drinking grapefruit juice while the other is cringing.

The basic tastes give the brain information about the types of food that have been consumed. The sensation of umami tells the body that protein-rich food is being ingested, sweet tastes indicate carbohydrate intake, and salty tastes give us information that is important for the balance between minerals or electrolytes and water. The tastes of bitter and sour likely developed as warning signals. Many toxic plants taste bitter, and a strong bitter taste can induce vomiting. Other evidence suggesting that bitterness is a warning signal is the fact that we can detect bitter substances 1,000 times better than, say, salty substances. Therefore, a significantly smaller amount of bitter tastant will yield a taste response, allowing toxic bitter substances to be avoided quickly. No wonder supertasters are especially sensitive to bitter tastes. Similarly, but to a lesser extent, sour indicates spoiled food (e.g., “sour milk”) or unripe fruits.

FIGURE 5.13 The neural correlates of satiation.
(a) Participants use a 10-point scale to rate the motivation and pleasantness of chocolate when offered a morsel seven times during the PET session. Desire and enjoyment declined over time. (b) Activation as measured during PET scanning during repeated presentations of chocolate (red). Water was presented during the first and last scans (blue). Across presentations, activity dropped in primary gustatory cortex (left) and increased in orbitofrontal cortex (right). The former could indicate an attenuated response to the chocolate sensation as the person habituates to the taste. The latter might correspond to a change in the participants’ desire (or aversion) to chocolate.

Humans can readily learn to discriminate similar tastes. Richard Frackowiak and his colleagues at University College London (Castriota-Scanderberg et al., 2005) studied wine connoisseurs (sommeliers), asking how their brain response compared to that of nonexperts when tasting wines that varied in quite subtle ways. In primary gustatory areas, the two groups showed a very similar response. The sommeliers, however, exhibited increased activation in the insula cortex and parts of the orbitofrontal cortex in the left hemisphere, as well as greater activity bilaterally in dorsolateral prefrontal cortex. This region is thought to be important for high-level cognitive processes such as decision making and response selection (see Chapter 12).

The orbitofrontal cortex also appears to play an important role in processing the pleasantness and reward value of eating food. Dana Small and her colleagues (2001) at Northwestern University used positron emission tomography (PET) to scan the brains of people as they ate chocolate (Figure 5.13). During testing, the participants rated the pleasantness of the chocolate and their desire to eat more chocolate. Initially, the chocolate was rated as very pleasant and the participants expressed a desire to eat more. But as the participants became satiated, their desire for more chocolate dropped. Moreover, although the chocolate was still perceived as pleasant, the intensity of their pleasure ratings decreased.

By comparing the neural activation in the beginning trials with the trials at the end of the study, the researchers were able to determine which areas of the brain participated in processing the reward value of the chocolate (the pleasantness) and the motivation to eat (the desire to have more chocolate). The posteromedial portion of the orbitofrontal cortex was activated when the chocolate was highly rewarding and the motivation to eat more was strong. In contrast, the posterolateral portion of the orbitofrontal cortex was activated during the satiated state, when the chocolate was unrewarding and the motivation to eat more was low. Thus, the orbitofrontal cortex appears to be a highly specialized taste-processing region containing distinct areas able to process opposite ends of the reward value spectrum associated with eating.

Somatosensory perception is the perception of all mechanical stimuli that affect the body. This includes interpretation of signals that indicate the position of our limbs and the position of our head, as well as our sense of temperature, pressure, and pain. Perhaps to a greater degree than with our other sensory systems, the somatosensory system includes an intricate array of specialized receptors and vast projections to many regions of the central nervous system.

Somatosensory receptors lie under the skin (Figure 5.14) and at the musculoskeletal junctions. Touch is signaled by specialized receptors in the skin, including Meissner’s corpuscles, Merkel’s cells, Pacinian corpuscles, and Ruffini corpuscles. These receptors differ in how quickly they adapt and in their sensitivity to various types of touch, such as deep pressure or vibration. Pain is signaled by nociceptors, the least differentiated of the skin’s sensory receptors. Nociceptors come in three flavors: thermal receptors that respond to heat or cold, mechanical receptors that respond to heavy mechanical stimulation, and polymodal receptors that respond to a wide range of noxious stimuli including heat, mechanical insults, and chemicals. The experience of pain is often the result of chemicals, such as histamine, that the body releases in response to injury. Nociceptors are located on the skin, below the skin, and in muscles and joints. Afferent pain neurons may be either myelinated or unmyelinated. The myelinated fibers quickly conduct information about pain. Activation of these cells usually produces immediate action. For example, when you touch a hot stove, the myelinated nociceptors can trigger a response that will cause you to quickly lift your hand, possibly even before you are aware of the temperature. The unmyelinated fibers are responsible for the duller, longer-lasting pain that follows the initial burn and reminds you to care for the damaged skin.

Specialized nerve cells provide information about the body’s position, or what is called proprioception (proprius: Latin for “own,” –ception: “receptor”; thus, a receptor for the self). Proprioception allows the sensory and motor systems to represent information about the state of the muscles and limbs. Proprioceptive cues, for example, signal when a muscle is stretched and can be used to monitor if that movement is due to an external force or from our own actions (see Chapter 8).

Somatosensory receptors have their cell bodies in the dorsal-root ganglia (or equivalent cranial nerve ganglia). The somatosensory receptors enter the spinal cord via the dorsal root (Figure 5.15). Some synapse on motor neurons in the spinal cord to form reflex arcs. Other axons synapse on neurons that send axons up the dorsal column of the spinal cord to the medulla. From here, information crosses over to the ventral posterior nucleus of the thalamus and then on to the cerebral cortex. As in vision (which is covered later in the chapter) and audition, the primary peripheral projections to the brain are crosswired; that is, information from one side of the body is represented primarily in the opposite, or contralateral, hemisphere. In addition to the cortical projections, proprioceptive and somatosensory information is projected to many subcortical structures, such as the cerebellum.

FIGURE 5.15 The major somatosensory pathway (representative).
From skin to cortex, the primary pathway of the somatosensory system.

The initial cortical receiving area is called primary somatosensory cortex or S1 (Figure 5.16a), which includes Brodmann areas 1, 2, and 3. S1 contains a somatotopic representation of the body, called the sensory homunculus (Figure 5.16b). Recall from Chapter 2 that the relative amount of cortical representation in the sensory homunculus corresponds to the relative importance of somatosensory information for that part of the body. For example, the hands cover a much larger portion of the cortex than the trunk does. The larger representation of the hands is essential given the great precision we need in using our fingers to manipulate objects and explore surfaces. When blindfolded, we can readily identify an object placed in our hand, but we would have great difficulty in identifying an object rolled across our back.

Somatotopic maps show considerable variation across species. In each species, the body parts that are the most important for sensing the outside world through touch are the ones that have the largest cortical representation. A great deal of the spider monkey’s cortex is devoted to its tail, which it uses to explore objects that might be edible foods or for grabbing onto tree limbs. The rat, on the other hand, uses its whiskers to explore the world; so a vast portion of the rat somatosensory cortex is devoted to representing information obtained from the whiskers (Figure 5.17).

Secondary somatosensory cortex (S2) builds more complex representations. From touch, for example, S2 neurons may code information about object texture and size. Interestingly, because of projections across the corpus callosum, S2 in each hemisphere receives information from both the left and the right sides of the body. Thus, when we manipulate an object with both hands, an integrated representation of the somatosensory information can be built up in S2.

Looking at the somatotopic maps may make you wonder just how much of that map is set in stone. What if you worked at the post office for many years sorting mail. Would you see changes in parts of the visual cortex that discriminate numbers? Or if you were a professional violinist, would your motor cortex be any bigger than that of the person who has never picked up a bow? Would anything happen to the part of your brain that represents your finger if you lost it in an accident? Would that part atrophy, or does the neighboring finger expand its representation and become more sensitive?

In 1949, Donald Hebb bucked the assumption that the brain was set in stone after the early formative years. He suggested a theoretical framework for how functional reorganization, or what neuroscientists refer to as cortical plasticity, might occur in the brain through the remodeling of neuronal connections. Since then, more people have been looking for and observing brain plasticity in action. Michael Merzenich (Merzenich & Jenkins, 1995; Merzenich et al., 1988) at the University of California, San Francisco, and Jon Kaas (1995) at Vanderbilt University discovered that in adult monkeys, the size and shape of the cortical sensory and motor maps can be altered by experience. For example, when the nerve fibers from a finger to the spinal cord are severed (deafferented), the relevant part of the cortex no longer responds to the touch of that finger (Figure 5.18). Although this is no big surprise, the strange part is that the area of the cortex that formerly represented the denervated finger soon becomes active again. It begins to respond to stimulation from the finger adjacent to the amputated finger. The surrounding cortical area fills in and takes over the silent area. Similar changes are found when a particular finger is given extended sensory stimulation: It gains a little more acreage on the cortical map. This functional plasticity suggests that the adult cortex is a dynamic place where changes can still happen, and it demonstrates a remarkable plasticity.

Extending these findings to humans, Vilayanur Ramachandran at the University of California, San Diego, studied the cortical mapping of human amputees. Look again at the human cortical somatosensory map in Figure 5.16b. What body part is represented next to the fingers and hand? Ramachandran reasoned that a cortical rearrangement ought to take place if an arm is amputated, just as had been found for the amputation of a digit in monkeys. Such a rearrangement might be expected to create bizarre patterns of perception, since the face area is next to the hand and arm area. Indeed, in one case study, Ramachandran examined a young man whose arm had been amputated just above the elbow a month earlier (1993). When a cotton swab was brushed lightly against his face, he reported feeling his amputated hand being touched! Feelings of sensation in missing limbs are the well-known phenomenon of phantom limb sensation. The sensation in the missing limb is produced by touching a body part that has appropriated the missing limb’s old acreage in the cortex. In this case, the sensation was introduced by stimulating the face. Indeed, with careful examination, a map of the young man’s hand could be demonstrated on his face (Figure 5.19).

These examples of plasticity led researchers to wonder if changes in experience within the normal range—say, due to training and practice—also result in changes in the organization of the adult human brain. Thomas Elbert and his colleagues at the University of Konstanz used magnetoencephalography (MEG) to investigate the somatosensory representations of the hand area in violin players (Elbert et al., 1995). They found that the responses in the musicians’ right hemisphere, which controls the left-hand fingers that manipulate the violin strings, were stronger than those observed in nonmusicians (Figure 5.20). What’s more, they observed that the size of the effect (the enhancement in the response) correlated with the age at which the players began their musical training. These findings suggest that a larger cortical area was dedicated to representing the sensations from the fingers of the musicians, owing to their altered but otherwise normal sensory experience. Another study used a complex visual motor task: juggling. After 3 months of training, the new jugglers had increased gray matter in the extrastriate motion-specific area in their visual cortex and in the left parietal sulcus, an area that is important in spatial judgments. (Draganski et al., 2004). Indeed, there is evidence that cortical reorganization can occur after just 15 to 30 minutes of practice (Classen et al., 1998).

The kicker is, however, that when the jugglers stopped practicing, these areas of their brain returned to their pretraining size, demonstrating something that we all know from experience: Use it or lose it. The realization that plasticity is alive and well in the brain has fueled hopes that stroke victims who have damaged cortex with resultant loss of limb function may be able to structurally reorganize their cortex and regain function. How this process might be encouraged is actively being pursued. One approach is to better understand the mechanisms involved.

Most of the evidence for the mechanisms of cortical plasticity comes from animal studies. The results suggest a cascade of effects, operating across different timescales. Rapid changes probably reflect the unveiling of weak connections that already exist in the cortex. Longer-term plasticity may result from the growth of new synapses and/or axons.

Immediate effects are likely to be due to a sudden reduction in inhibition that normally suppresses inputs from neighboring regions. Reorganization in the motor cortex has been found to depend on the level of gamma-aminobutyric acid (GABA), the principal inhibitory neurotransmitter (Ziemann et al., 2001). When GABA levels are high, activity in individual cortical neurons is relatively stable. If GABA levels are lower, however, then the neurons may respond to a wider range of stimuli. For example, a neuron that responds to the touch of one finger will respond to the touch of other fingers if GABA is blocked. Interestingly, temporary deafferentation of the hand (by blocking blood flow to the hand) leads to a lowering of GABA levels in the brain. These data suggest that short-term plasticity may be controlled by a release of tonic inhibition on synaptic input (thalamic or intracortical) from remote sources.

Changes in cortical mapping over a period of days probably involve changes in the efficacy of existing circuitry. After loss of normal sensory input (e.g., through amputation or peripheral nerve section), cortical neurons that previously responded to that input might undergo “denervation hypersensitivity.” That is, the strength of the responses to any remaining weak excitatory input is upregulated: Remapping might well depend on such modulations of synaptic efficacy. Strengthening of synapses is enhanced in the motor cortex by the neurotransmitters norepinephrine, dopamine, and acetylcholine; it is decreased in the presence of drugs that block the receptors for these transmitters (Meintzschel & Ziemann, 2005). These changes are similar to the forms of long-term potentiation and depression in the hippocampus that are thought to underlie the formation of spatial and episodic memories that we will discuss in Chapter 9.

Finally, some evidence in animals suggests that the growth of intracortical axonal connections and even sprouting of new axons might contribute to very slow changes in cortical plasticity.

Now let’s turn to a more detailed analysis of the most widely studied sense: vision. Like most other diurnal creatures, humans depend on the sense of vision. Although other senses, such as hearing and touch, are also important, visual information dominates our perceptions and appears even to frame the way we think. Much of our language, even when used to describe abstract concepts with metaphors, makes reference to vision. For example, we say “I see” to indicate that something is understood, or “Your hypothesis is murky” to indicate confused thoughts.

One reason vision is so important is that it enables us to perceive information at a distance, to engage in what is called remote sensing or exteroceptive perception. We need not be in immediate contact with a stimulus to process it. Contrast this ability with the sense of touch. For touch, we must be in direct contact with the stimulus. The advantages of remote sensing are obvious. An organism surely can avoid a predator better when it can detect the predator at a distance. It is probably too late to flee once a shark has sunk its teeth into you, no matter how fast your neural response is to the pain.

The Receptors Visual information is contained in the light reflected from objects. To perceive objects, we need sensory detectors that respond to the reflected light. As light passes through the lens of the eye, the image is inverted and focused to project on the back surface of the eye (Figure 5.21), the retina. The retina is only about 0.5 mm thick, but it is made up of 10 densely packed layers of neurons. The deepest layers are composed of millions of photoreceptors, the rods and cones. These contain photopigments, protein molecules that are sensitive to light. When exposed to light, the photopigments become unstable and split apart. Unlike most neurons, rods and cones do not fire action potentials. The decomposition of the photopigments alters the membrane potential of the photoreceptors and triggers action potentials in downstream neurons. Thus, photoreceptors provide for translation of the external stimulus of light into an internal neural signal that the brain can interpret.

The rods contain the pigment rhodopsin, which is destabilized by low levels of light. Rods are most useful at night when light energy is reduced. Rods also respond to bright light, but the pigment quickly becomes depleted and the rods cease to function until it is replenished. Because this takes several minutes, they are of little use during the day. Cones contain a different type of photopigment, called a photopsin. Cones require more intense levels of light but can replenish their photopigments rapidly. Thus, cones are most active during daytime vision. There are three types of cones, defined by their sensitivity to different regions of the visible spectrum: (a) a cone that responds to short wavelengths, the blue part of the spectrum; (b) one that responds to medium wavelengths, the greenish region; and (c) one that responds to the long “reddish” wavelengths (Figure 5.22). The activity of these three different receptors ultimately leads to our ability to see color.

FIGURE 5.21 Anatomy of the eye and retina.
Light enters through the cornea and activates the receptor cells of the retina located along the rear surface. There are two types of receptor cells: rods and cones. The output of the receptor cells is processed in the middle layer of the retina and then relayed to the central nervous system via the optic nerve, the axons of the ganglion cells.

Rods and cones are not distributed equally across the retina. Cones are densely packed near the center of the retina, in a region called the fovea. Few cones are in the more peripheral regions of the retina. In contrast, rods are distributed throughout the retina. You can easily demonstrate the differential distribution of rods and cones by having a friend slowly bring a colored marker into your view from one side of your head. Notice that you see the marker and its shape well before you identify its color, because of the sparse distribution of cones in the retina’s peripheral regions.

The Retina to the Central Nervous System The rods and cones are connected to bipolar neurons that then synapse with the ganglion cells, the output layer of the retina. The axons of these cells form a bundle, the optic nerve, that transmits information to the central nervous system. Before any information is shipped down the optic nerve, however, extensive processing occurs within the retina, an elaborate convergence of information. Indeed, though humans have an estimated 260 million photoreceptors, we have only 2 million ganglion cells to telegraph information from the retina. Many rods feed into a single ganglion cell. By summing their outputs, the rods can activate a ganglion cell even in low light situations. For cones, however, the story is different: Each ganglion cell is innervated by only a few cones. Thus, they carry much more specific information from only a few receptors, ultimately providing a sharper image. The compression of information, as with the auditory system, suggests that higher-level visual centers should be efficient processors to unravel this information and recover the details of the visual world.

Figure 5.23 diagrams how visual information is conveyed from the eyes to the central nervous system. As we discussed in the last chapter, before entering the brain, each optic nerve splits into two parts. The temporal (lateral) branch continues to traverse along the ipsilateral side. The nasal (medial) branch crosses over to project to the contralateral side; this crossover place is called the optic chiasm. Given the eye’s optics, the crossover of nasal fibers ensures that visual information from each side of external space will be projected to contralateral brain structures. Because of the retina’s curvature, the temporal half of the right retina is stimulated by objects in the left visual field. In the same fashion, the nasal hemiretina of the left eye is stimulated by this same region of external space. Because fibers from each nasal hemiretina cross, all information from the left visual field is projected to the right hemisphere, and information from the right visual field is projected to the left hemisphere.

Each optic nerve divides into several pathways that differ with respect to where they terminate in the subcortex. Figure 5.23 focuses on the pathway that contains more than 90 % of the axons in the optic nerve, the retinogeniculate pathway, the projection from the retina to the lateral geniculate nucleus (LGN) of the thalamus. The LGN is made up of six layers. One type of ganglion cell, the M cell, sends output to the bottom two layers. Another type of ganglion cell the P cell, projects to the top four layers. The remaining 10 % of the optic nerve fibers innervate other subcortical structures, including the pulvinar nucleus of the thalamus and the superior colliculus of the midbrain. Even though these other receiving nuclei are innervated by only 10 % of the fibers, these pathways are still important. The human optic nerve is so large that 10 % of it constitutes more fibers than are found in the entire auditory pathway. The superior colliculus and pulvinar nucleus play a large role in visual attention.

The final projection to the visual cortex is via the geniculocortical pathway. This bundle of axons exits the LGN and ascends to the cortex, and almost all of the fibers terminate in the primary visual cortex (V1) of the occipital lobe. Thus visual information reaching the cortex has been processed by at least four distinct neurons: photoreceptors, bipolar cells, ganglion cells, and LGN cells. Visual information continues to be processed as it passes through higher order visual areas in the cortex.

There are diseases and accidents that damage the eyes’ photoreceptors, but otherwise leave the visual pathway intact. Until recently, people in this situation would go blind. But things are looking brighter for these patients thanks to microelectronics (see “How the Brain Works: When the Receptors No Longer Function”).

Keeping the Picture Straight: Retinotopic Maps Due to the optics of the eye, light reflecting off of objects in the environment strikes the eye in an orderly manner. Light reflected off of an object located to the right of someone’s gaze will activate photoreceptors on the medial, or nasal, side of the right retina and lateral or temporal side of the left retina. As this information is projected upstream via the optic nerve, however, the direct link between neural activity and space is lost. Nonetheless, neurons in the visual system represent space. This is shown by the fact that most visual neurons only respond when a stimulus is presented in a specific region of space, or what is defined as the receptive field of the neuron. For example, a cell in the right visual cortex may respond to a bar of light, but only if that bar is presented in a specific region of space (e.g., the upper left visual field; see Figure 3.19). Moreover, there is an orderly relationship between the receptive fields of neighboring cells. Thus, external space is represented continuously within neural regions such as the LGN or V1. As with the somatosensory and auditory systems, the receptive fields of visual cells form an orderly mapping between an external dimension (in this case, spatial location) and the neural representation of that dimension. In vision, these topographic representations are referred to as retinotopic maps. A full retinotopic map contains a representation of the entire contralateral hemifield (e.g., left hemisphere V1 will have a full representation of the right side of space).

Receptive fields range in size, becoming larger across the visual system (Figure 5.24). LGN cells have receptive fields responding only if the stimulus falls within a very limited region of space, about one degree of visual angle. Cells in V1 have slightly larger receptive fields, and this magnification process continues through the visual system: Cells in the temporal lobe have receptive fields that may encompass an entire hemifield.

A primary physiological method for establishing visual areas is to measure how spatial information is represented across a region of cortex. Figure 5.24 shows a map of the visual areas of the cortex as defined by their physiology. Each box in the figure stands for a distinct region of visual processing, defined because the region contains its own retinotopic map. Thus, the boundaries between anatomically adjacent visual areas are marked by topographic discontinuities (Figure 5.25). As one area projects to another, topography and precise spatial information is preserved by these multiple retinotopic maps, at least in early visual areas. Over 30 distinct cortical visual areas have been identified in the monkey, and the evidence indicates that humans have even more.

After being blind for 5 years, a patient sits at a table and is able to identify not only where a mug and various cutlery are placed, but can also tell that his name, spelled out in large letters, has been spelled incorrectly. He is one of three patients who have had an electronic chip implanted behind the retina (Zrenner et al., 2011). This chip is designed for patients who are suffering from blindness caused by degenerative diseases that affect photoreceptors and result in progressive vision loss. In the first few years of blindness the other cells of the retina remain intact—a situation this particular retinal implant uses to its advantage.

The tiny implant chip, measuring 3 mm by 3.1 mm, contains 1,500 light-sensitive microphotodiodes (Figure 1). Light enters the eye through the lens, passes through the transparent retina, and hits the chip. The image is simultaneously captured several times per minute by all of the photodiodes, each of which controls a tiny amplifier connected to an electrode, together known as an element (pixel). Each element generates a voltage at its electrode, the strength of which depends on the intensity of light hitting the photodiode. The voltage then passes to the adjacent bipolar neurons in the retina, and the signal proceeds through the rest of the visual pathway. One question facing those designing retinal implants is, how many photodiodes are needed to gain an acceptable image? When you consider that the eye contains millions of photoreceptors, 1,500 seems like a drop in the bucket. Indeed, this number produces only crude images. This system is in its infancy, but it allows a blind person to navigate and make simple discriminations. The chip is powered by an implanted cable that runs from the eye under the temporalis muscle and out from behind the ear, where it is attached to a wirelessly operated power control unit that the patient wears around his neck. This implant was placed temporarily, for just a few weeks, to test the concept. The next-generation system, currently being tested, is not cable bound. Instead, an encapsulated coil is implanted behind the ear and connected to a transmitter that magnetically attaches to an outside power coil.

FIGURE 1 Retinal implant device.

Note that the names for the areas shown in Figure 5.24 primarily draw on the nomenclature developed by physiologists (see Chapter 2). Striate cortex, or V1, is the initial projection region of geniculate axons. Although other areas have names such as V2, V3, and V4, this numbering scheme should not be taken to mean that the synapses proceed sequentially from one area to the next. The lines connecting these extrastriate visual areas demonstrate extensive convergence and divergence across visual areas. In addition, connections between many areas are reciprocal; areas frequently receive input from an area to which they project.

Cellular Properties Vary Across Cortical Visual Areas Why would it be useful for the primate brain to have evolved so many visual areas? One possibility is that visual processing is hierarchical. Each area, representing the stimulus in a unique way, successively elaborates on the representation derived by processing in earlier areas. The simple cells of the primary visual cortex calculate edges. Complex cells in secondary visual areas use the information from many simple cells to represent corners and edge terminations. In turn, higher order visual neurons integrate information from complex cells to represent shapes. Successive elaboration culminates in formatting the representation of the stimulus so that it matches (or doesn’t match) information in memory. An interesting idea, but there is a problem. As Figure 5.24 shows, there is no simple hierarchy; rather, extensive patterns of convergence and divergence result in multiple pathways.

An alternative hypothesis is based on the idea that visual perception is an analytic process. Although each visual area provides a map of external space, the maps represent different types of information. For instance, neurons in some areas are highly sensitive to color variation. In other areas, they are sensitive to movement but not to color.

Based on this hypothesis, neurons within an area not only code where an object is located in visual space but also provide information about the object’s attributes. By this perspective, visual perception can be considered to entail a divide-and-conquer strategy. Rather than all attributes of an object being represented by all visual areas, each visual area provides its own limited analysis. Processing is distributed and specialized. As signals advance through the visual system, different areas elaborate on the initial information in V1 and begin to integrate this information across dimensions to form recognizable percepts. Early work on these ideas is presented in “Milestones in Cognitive Neuroscience: Pioneers in the Visual Cortex.”

Specialized Function of Visual Areas in Monkeys Extensive physiological evidence supports the specialization hypothesis. Consider cells in area MT (sometimes referred to as V5), so named because it lies in the middle temporal lobe region of the macaque monkey, a species used in many physiology studies. Single-cell recordings reveal that neurons in this region do not show specificity regarding the color of the stimulus. These cells will respond similarly to either a green or a red circle on a white background. Even more striking, these neurons respond weakly when presented with an alternating pattern of red and green stripes whose colors are equally bright.

In contrast, MT neurons are quite sensitive to movement and direction, as Figure 5.26 shows (Maunsell & Van Essen, 1983). The stimulus, a rectangular bar, was passed through the receptive field of a specific MT cell in varying directions. The cell’s response was greatest when the stimulus was moved downward and left. In contrast, this cell was essentially silent when the stimulus was moved upward or to the right. Thus the cell’s activity correlates with two attributes of the stimulus. First, the cell is active only when the stimulus falls within its receptive field. Second, the response is greatest when the stimulus moves in a certain direction. Activity in MT cells also correlates with the speed of motion. The cell in Figure 5.26 responded maximally when the bar was moved rapidly. At lower speeds, the bar’s movement in the same direction failed to raise the response rate above baseline.

Specialized Function of Human Visual Areas Single-cell recording studies have provided physiologists with a powerful tool for mapping the visual areas in the monkey brain and characterizing the functional properties of the neurons within these areas. This work has yielded strong evidence that different visual areas are specialized to represent distinct attributes of the visual scene. Inspired by these results, researchers have employed neuroimaging techniques to describe the functional architecture of the human brain.

In a pioneering study, Semir Zeki (1993) of University College London and his colleagues at London’s Hammersmith Hospital used positron emission tomography (PET) to explore similar principles in the human visual system, starting with the goal of identifying areas that were involved in processing color or motion information. They used subtractive logic by factoring out the activation in a control condition from the activation in an experimental condition. Let’s check out the color experiment to see how this works. For the control condition, participants passively viewed a collage of achromatic rectangles. Various shades of gray, spanning a wide range of luminances, were chosen. The control stimulus was expected to activate neural regions with cells that are contrast sensitive (e.g., sensitive to differences in luminance).

For the experimental condition, the gray patches were replaced by a variety of colors (Figure 5.27a). Each color patch was matched in luminance to its corresponding gray patch. With this setup, neurons sensitive to luminance information should be equally activated in control and experimental conditions. The colored stimulus, however, should produce more activity in neural regions sensitive to chromatic information. These regions should be detected if the metabolic activity recorded when participants viewed the gray stimulus is subtracted from the activity recorded when participants viewed the color stimulus.

Like the voyages of 15th-century European explorers, initial investigations into the neurophysiology of the cerebral cortex required a willingness to sail in uncharted waters. The two admirals in this enterprise were David Hubel and Torsten Wiesel. Hubel and Wiesel arrived at Johns Hopkins University in the late 1950s, hoping to extend the pioneering work of Steve Kuffler (1953). Kuffler’s research had elegantly described the receptive-field organization of ganglion cells in the cat retina, laying out the mechanisms that allowed cells to detect the edges that define objects in the visual world. Rather than focusing on the lateral geniculate nucleus (LGN), the next relay in the system, Hubel and Wiesel (1977) set their sights on the primary visual cortex. Vernon Mountcastle, another Hopkins researcher, was just completing his seminal work, in which he laid out the complex topographic organization of the somatosensory cortex (Mountcastle, 1976). Hubel and Wiesel were inspired to look for similar principles in vision.

During the first few weeks of their recordings, Hubel and Wiesel were puzzled by what they observed. Although they had little difficulty identifying individual cortical cells, the cells failed to respond to the kinds of stimuli that had proved so effective in Kuffler’s studies: small spots of light positioned within a cell’s receptive fields. Indeed, the lack of consistent responses made it difficult to determine where the receptive field was situated. Hubel and Wiesel had a breakthrough, though, when they switched to dark spots, which they created by placing an opaque disk on a glass slide. Although the cell did not respond to the dark spot, Hubel and Wiesel noticed a burst in activity as the edge of the glass moved across part of the retina. After hours of play with this stimulus, the first organizational principle of primary visual cortex neurons became clear: Unlike the circular receptive fields of ganglion cells, cortical neurons were responsive to edges.

Subsequent work revealed that LGN cells and ganglion cells behave similarly: Both are maximally excited by small spots of light. Such cells are best characterized as exhibiting a concentric center–surround organization. Figure 1 shows the receptive field of an LGN cell. When a spot of light falls within the excitatory center region, the cell is activated. If the same spot is moved into the surrounding region, the activity is inhibited. A stimulus that encompasses both the center and the surrounding region will fail to activate the cell, because the activity from the excitatory and inhibitory regions will cancel each other out. This observation clarifies a fundamental principle of perception: The nervous system is most interested in change. We recognize an elephant not by the homogeneous gray surface of its body, but by the contrast of the gray edge of its shape against the background.

FIGURE 1 Characteristic response of a lateral geniculate nucleus (LGN) cell.
Cells in the LGN have concentric receptive fields with either an on-center, off-surround organization or an off-center, on-surround organization. The on-center, off-surround cell shown here fires rapidly when the light encompasses the center region (a) and is inhibited when the light is positioned over the surround (b). A stimulus that spans both the center and the surround produces little change in activity (c). Thus, LGN cells are ideal for signaling changes in illumination, such as those that arise from stimulus edges.

FIGURE 2 Simple cells in the primary visual cortex can be formed by the linking of outputs from concentric lateral geniculate nucleus (LGN) cells with adjacent receptive fields. In addition to signaling the presence of an edge, simple cells are selective for orientation. The simple cell illustrated here is either excited or inhibited by an edge that follows its preferred orientation. It shows no change in activity if the edge is at a perpendicular orientation.

In Figure 2, outputs from three LGN cells with receptive fields centered at slightly different positions are linked to a single cortical neuron. This cortical neuron would continue to have a center–surround organization, but for this cell the optimal stimulus would have to be an edge. In addition, the cell would be selective for edges in a certain orientation. As the same stimulus was rotated within the receptive field, the cell would cease to respond, because the edge would now span excitatory and inhibitory regions of the cell. Hubel and Wiesel called these cells simple cells, because their simple organization would extract a fundamental feature for shape perception: the border of an object. The same linking principle can yield more complex cells—cells with a receptive-field organization that makes them sensitive to other features, such as corners or edge terminations.

Orientation selectivity has proved to be a hallmark of neurons in the primary visual cortex. Across a chunk of cortex measuring 2 mm by 2 mm, the receptive fields of neurons are centered on a similar region of space (Figure 3). Within the chunk, the cells vary in terms of their preferred orientation, and they alternate between columns that are responsive to inputs from the right and left eyes. A series of such chunks allows for the full representation of external space, providing the visual system with a means of extracting the visible edges in a scene.

Hubel and Wiesel’s studies established how a few organizational principles can serve as building blocks of perception derived from simple sensory neurons. The importance of their pioneering studies was acknowledged in 1981, when they shared the Nobel Prize in Physiology or Medicine.

FIGURE 3 Feature representation within the primary visual cortex.
(a) As the recording electrode is moved along the cortex, the preferred orientation of the cells continuously varies. The preferred orientation is plotted as a function of the location of the electrode. (b) The orientation columns are crossed with ocular dominance columns to form a cortical module. Within a module, the cells have similar receptive fields (location sensitivity), but they vary based on input source (left or right eye) and sensitivity to orientation, color, and size. For example, the so-called blobs contain cells that are sensitive to color and finer details in the visual input. This organization is repeated for each module.

The same logic was used to design the motion experiment. For this study, the control stimulus consisted of a complex black-and-white collage of squares (Figure 5.27b). The same stimulus was used in the experimental condition, except that the squares were set in motion. They would move in one direction for 5 seconds and then in the reverse direction for the next 5 seconds.

The results of the two studies provided clear evidence that the two tasks activated distinct brain regions (Figure 5.28). After subtracting activation during viewing of the achromatic collage, investigators found numerous residual foci of activation when participants viewed the colored collage. These foci were bilateral and located in the most anterior and inferior regions of the occipital lobe (Figure 5.28a). Although the spatial resolution of PET is coarse, these areas were determined to be in front of the striate (V1) and prestriate (V2) cortex. In contrast, after the appropriate subtraction in the motion experiment, the residual foci were bilateral but near the junction of the temporal, parietal, and occipital cortices (Figure 5.28b). These foci were more superior and much more lateral than the color foci.

Zeki and his colleagues were so taken with this dissociation that they proposed applying the nomenclature developed by primate researchers here. They labeled the area activated in the color foci as area V4 and the area activated in the motion task as V5. Note that researchers frequently refer to area V5 as human area MT, even though the area is not in the temporal lobe in the human brain. Of course, with PET data we cannot be sure that the foci of activation really consist of just one visual area.

A comparison of Figures 5.25 and 5.28 reveals striking between-species differences in the relative position of the color and motion areas. For example, human MT is on the lateral surface of the brain, whereas the monkey MT is more medial. Such differences probably exist because the surface area of the human brain is substantially larger, and this expansion required additional folding of the continuous cortical sheet.

The activation maps in Zeki’s PET study are rather crude. Vision scientists now employ sophisticated fMRI techniques to study the organization of human visual cortex. In these studies, a stimulus is systematically moved across the visual field (Figure 5.29). For example, a semicircular checkerboard pattern is slowly rotated about the center of view. In this way, the blood oxygen level– dependent (BOLD) response for areas representing the superior quadrant will be activated at a different time than areas representing the inferior quadrant—and in fact, the representation of the entire visual field can be continuously tracked. To compare areas that respond to foveal stimulation and those that respond to peripheral stimulation, researchers use a dilating and contracting ring stimulus. By combining these different stimuli, they can measure the cortical representation of external space.

Due to the convoluted nature of the human visual cortex, the results from such an experiment would be difficult to decipher if we were to plot the data on the anatomical maps found in a brain atlas. To avoid this problem, vision scientists prefer to work with flat maps of the brain. High-resolution anatomical MRI scans are obtained, and computer algorithms transform the folded, cortical surface into a two-dimensional map by tracing the gray matter. The activation signals from the fMRI study are then plotted on the flattened surface, and areas that were activated at similar times are color-coded.

Researchers have used this procedure to reveal the organization of the human visual system in exquisite detail. Activation maps, plotted on both a normal brain and as flattened maps, are shown in Figure 5.30.8/Apr/2018 19:47sual cortex (V1) lies along the calcarine sulcus. As in all physiological studies, the physical world is inverted. Except for the most anterior aspects of visual cortex, areas above the sulcus are active when the rotating stimulus is in the lower quadrant; the reverse is true when the stimulus is in the upper quadrant. Moreover, the activation patterns show a series of repetitions across the visual cortex indicating distinct topographic maps. Following the conventions adopted in the single-cell studies in monkeys, the visual areas are numbered in increasing order, where primary visual cortex (V1) is most posterior and secondary visual areas (V2, V3/VP, V4) more anterior.

Functional MRI mapping procedures can reveal multiple visual areas and can be used for comparison with the data obtained in work with monkeys. Within lateral occipital cortex (LOC), two subareas, LO1 and LO2, are evident. These regions had not been identified in previous studies of the monkey, and they provide further evidence of the expansion of visual cortex in humans (Figure 5.30). Interestingly, although activity in these areas is not modulated by motion per se, the regions do show an increase in the BOLD response when motion signals define object boundaries (e.g., a moving stimulus occludes the background) as well as when viewing displays of objects compared to scrambled images.

Figure 5.30b also shows how eccentricity, the distance away from the fixation point, is also represented in these visual areas. Eccentricity refers to the radial distance from the center of vision (the foveal region) to the periphery. The cortical representation of the fovea, the regions shown in purple, pink, and red, is quite large. Visual acuity is much greater at the fovea due to the disproportionate amount of cortex that encodes information from this part of space.

FIGURE 5.30 Two retinotopic areas in human lateral occipital cortex (LOC).
(a) The circular displays at the bottom represent the display on which a stimulus was projected, with the person instructed to fixate at the center of the crosshair. Across the scanning run, the position of the stimulus spans visual space. Left side shows color coding of activation patterns on flat map of visual cortex when the angular position of a stimulus was varied. For example, areas responding when the stimulus was presented below fixation are coded as red. Multiple retinotopic maps are evident in dorsal and ventral regions. Right side shows color coding of activation patterns when the eccentricity of the stimulus was varied (e.g., dark purple indicates activation areas when stimulus was at center of fixation). (b) Position of visual areas shown in (a) on an inflated brain. The size and location can only be approximated in a lateral view of the 3-d image.

As we discussed in Chapter 3, technology marches on, and even more powerful tools are constantly being developed to provide better images of brain function. In the MRI world, stronger magnets improve the resolution of the fMRI signal. A 7-tesla (T) fMRI system is capable of providing detailed pictures of organizational principles within a visual area (Yacoub, 2008). Within V1, a 7-T magnet can reveal ocular dominance columns whose areas have similar retinotopic tuning, thus showing a preference for input from either the right or left eye. A shift across voxels in terms of orientation tuning is also visible. Such specificity is striking when we keep in mind that the activation within a voxel reflects the contribution of millions of neurons. Orientation tuning does not mean that all of these neurons have similar orientation preferences. Rather, it means that the relative contribution of orientation-selective neurons varies across voxels. Some voxels have a stronger contribution from vertically oriented cells; others, a stronger contribution from horizontally oriented cells (Figure 5.31).

In Chapter 6, we will explore the question of how our sensory experiences are turned into percepts—how we take the information from our sensory systems and use it to recognize objects and scenes. Here we briefly discuss the relationship between sensation and perception, describing experiments that ask how activation in early sensory areas relates to our perceptual experience. For example, is activation in early visual cortex sufficient to support perception? Or does that information have to be relayed to higher visual areas in order for us to recognize the presence of a stimulus? We have seen in the previous section that certain elementary features are represented in early sensory areas, usually with some form of topographic organization. Cells in auditory cortex are tuned to specific frequency bands; cells in visual cortex represent properties such as orientation, color, and motion. The information represented in primary sensory areas is refined and integrated as we move into secondary sensory areas. An important question is: At what stage of processing does this sensory stimulation become a percept, something we experience phenomenally?

One way to study this question is to “trick” our sensory processing systems with stimuli that cause us to form percepts that do not correspond to the true stimuli in the environment. In other words, what we perceive is an illusion. By following the processing of such stimuli using fMRI, we can attempt to determine where in the processing stream the signals become derailed. For instance, if we look at a colored disc that changes color every second from red to green, we have no problem seeing the two colors in succession. If the same display flips between the two colors 25 times per second (or 25 Hz), however, then the percept is of a fused color—a constant, yellowish white disc (the additive effects of red and green light). This phenomenon is known as flicker fusion. At what stage in the visual system does the system break down, failing to keep up with the flickering stimulus? Does it occur early in processing within the subcortical structures, or is it later, in one of the cortical visual areas?

Using a flickering stimulus, Sheng He and colleagues tested participants while observing the changes in visual cortex (Jiang et al., 2007). In Figure 5.32, compare the fMRI BOLD responses for visual areas V1, V4, and VO during a 5-Hz full-contrast flicker condition (perceptually two colors), a 30-Hz full-contrast flicker condition (perceptually one fused color), and a control condition, which was a 5-Hz subthreshold contrast condition (perceptually indistinguishable from the 30-Hz flicker). Subcortical processing and several of the lower cortical processing areas, V1 and V4, were able to distinguish between the 5-Hz flicker, the 30-Hz flicker, and the 5-Hz nonflickering control. In contrast, the BOLD response within a visual area just adjacent to V4, VO, did not differentiate between the high-flicker stimulus and the static control stimulus (Figure 5.32). We can conclude that the illusion—a yellowish object that is not flickering—is formed in this higher visual area (known variously as either VO or V8), indicating that although the information is sensed accurately at earlier stages within the visual stream, conscious perception, at least of color, is more closely linked to higher-area activity.

In a related study, John-Dylan Haynes and Geraint Rees at the University College London asked if fMRI could be used to detect the neural fingerprints of unconscious “perception” (Haynes & Rees, 2005). Participants were asked to decide which of two ways a stimulus was oriented (Figure 5.33). When shown the stimulus for just a 20th of a second, people can identify its orientation with a high degree of accuracy. If, however, the stimulus is presented even faster—say, for just a 30th of a second—and it is preceded and followed by a mask of crosshatched lines, performance drops to chance levels. Nonetheless, by using a sophisticated pattern recognition algorithm on the fMRI data, the researchers were able to show that activity in V1 could distinguish which stimulus had been presented—an effect that was lost in V2 and V3.

As the preceding examples indicate, our primary sensory regions provide a representation that is closely linked to the physical stimulus, and our perceptual experience is more dependent on activity in secondary and association sensory regions. Note, though, that the examples base this argument on the fact that the absence of a perceptual experience was matched by the absence of detectable activity in secondary regions. We can also consider the flip side of the coin, by asking what brain regions show activation patterns that are correlated with illusory percepts. Stare at the Enigma pattern in Figure 5.34. After a few seconds, you should begin to see scintillating motion within the blue circles—an illusion created by their opposed orientation to the radial black and white lines. What are the neural correlates of this illusion? We know that moving patterns produce a strong hemodynamic response in V5. Is this same area also activated during illusory motion? Both PET and fMRI have been used to show that viewing displays like the Enigma pattern does indeed lead to pronounced activity in V5. This activation is selective: Activity in V1 does not increase during illusory motion.

An even stronger case for the hypothesis that perception is more closely linked to secondary sensory areas would require evidence showing that activity in these areas can be sufficient, and even predictive of perception. This idea was tested in a remarkable study performed by Michael Shadlen and his colleagues at the University of Washington (Ditterich et al., 2003). They used a reverse engineering strategy to manipulate activation patterns in sensory cortex. As we noted earlier, physiologists usually eavesdrop on neurons in sensory cortex using electrodes that probe how cells respond to information in the environment. The same electrodes can also be used to activate cells. When a current passes through the electrode, neurons near the tip of the electrode are activated. In the Shadlen study, researchers used this method to measure motion perception. Monkeys were trained to make an eye movement, indicating the perceived direction of a patch of moving dots (Figure 5.35). To make the task challenging, only a small percentage of the dots moved in a common direction; the rest moved in random directions. The researchers then recorded from a cell in area MT. After determining the directional tuning of that cell, they passed a current through the electrode while the stimulus was present. This manipulation increased the probability that the monkey would report that the stimulus was moving in the cell’s preferred direction (Figure 5.35). Note that the electrical current, at least with this method, will likely induce activity in many neurons. Nonetheless, the finding that the animal’s percept was altered suggests that neighboring cells have similar direction-tuning properties, consistent with a topographic representation of motion direction in MT.

Of course, with the monkeys, we can only infer their perception from behavior; it is problematic to infer that these percepts correspond to conscious experience. Similar stimulation methods have been used on rare occasions in humans during intraoperative surgical procedures. In one such procedure, electrodes were positioned along the ventral regions of visual cortex (Murphey et al., 2008). This region includes at least two areas that are known to be involved with color processing: the posterior center in the lingual gyrus of the occipital lobe (V4) and the anterior center in the medial fusiform gyrus of the temporal lobe, which has been labeled V4a. When used as recording devices, electrodes in either area responded in a selective manner to chromatic stimuli. For example, the activity at one location was stronger to one color as compared to another. Even more interesting was what happened when the electrodes were used as stimulating devices. In the anterior color region, stimulation led to the patient reporting seeing a colored, amorphous shape. Moreover, the color of the illusion was similar to the preferred color for that site. Thus, in this higher visual area, there was a close correspondence between the perception of a color when it was elicited by a visual stimulus and when the cortex was electrically stimulated.

Occasionally, when viewing illusions with a friend, you will find that the two of you don’t have the same reaction. You might be saying, “This is crazy!” Meanwhile, your friend is shrugging her shoulders, wondering what you are seeing. Although we commonly accept the idea that people have different emotional reactions to similar situations, we tend to assume that everyone perceives the same things. In this example, we might assume your friend just doesn’t know how to “look” at the display in the right way. To test this assumption, researchers sought to identify neural biomarkers that might account for individual differences in perception (Schwarzkopf et al., 2011).

Figure 5.36 shows one of the classic illusions in visual perception: the Ebbinghaus illusion, devised by Hermann Ebbinghaus (1850–1909), a German pioneer in experimental psychology. Compare the size of the two circles in the middle of the displays on the left and right. Does one look larger than the other? By how much? Everyone sees the middle circle on the right as larger than the one on the left, but people vary considerably regarding how much larger they think the circle is. Some individuals see the right inner circle as larger by only about 10 %. For others, the illusion is close to 50 %. These differences are quite reliable and can be observed across a range of size illusions, leading the research team to wonder about their underlying cause. They used fMRI to identify retinotopic areas and then measured the size of the functionally defined area. Remarkably, they observed a negative correlation between the size of the illusion and the size of V1. The smaller the area of V1, the larger the perceived illusion. This correlation was not found with V2 or V3.

Why might people with a larger V1 show a smaller illusion? One hypothesis is that with a large visual cortex, each region of space has a better representation. A corollary of this is that each region of space is less likely to be influenced from neighboring regions of space. Hence, in the Ebbinghaus illusion, the neighboring circles have less influence on the central circle when a larger V1 provides a higher-resolution representation of space. Perception, then, is in the brain anatomy of the beholder.

Before the advent of neuroimaging, much of what we learned about visual processing in the human brain came from lesion studies. In 1888, Louis Verrey (cited in Zeki, 1993), described a patient who, after suffering a stroke, had lost the ability to perceive colors in her right visual field. Verrey reported that while the patient had problems with acuity within restricted portions of this right visual field, the color deficit was uniform and complete. After his patient’s death, Verrey performed an autopsy. What he found led him to conclude there was a “centre for the chromatic sense” (Zeki, 1993) in the human brain, which he located in the lingual and fusiform gyri. We can guess that this patient’s world looked similar to the drawing in Figure 5.37: On one side of space, the world was multicolored; on the other, it was a montage of grays.

When we speak of someone who is color-blind, we are usually describing a person who has inherited a gene that produces an abnormality in the photoreceptor system. Dichromats, people with only two photopigments, can be classified as red–green color-blind if they are missing the photopigment sensitive to either medium or long wavelengths, or blue–yellow color-blind if they are missing the short-wavelength photopigment. Anomalous trichromats, in contrast, have all three photopigments, but one of the pigments exhibits abnormal sensitivity. The incidence of these genetic disorders is high in males: about 8 % of the population. The incidence in females is less than 1 %.

Much rarer are disorders of color perception that arise from disturbances of the central nervous system. These disorders are called achromatopsia (from the prefix a−, “without,” and the stem chroma, “hue”). J. C. Meadows (1974) of the National Hospital for Neurology and Neurosurgery in London described one such patient as follows:

Patients with achromatopsia often report that colors have become a bland palette of “dirty shades of gray.” The shading reflects variations in brightness rather than hue. Other aspects of vision, such as depth and texture perception, remain intact, enabling someone with achromatopsia to see and recognize objects in the world. Indeed, color is not a necessary cue for shape perception. The subtlety of color perception is underscored when we consider that people often do not notice the change from black and white to color when Dorothy lands in Oz in the movie The Wizard of Oz. Nonetheless, when lost forever, this subtlety is sorely missed.

Achromatopsia has consistently been associated with lesions that encompass V4 and the region anterior to V4. The lesions, however, typically extend to neighboring regions of the visual cortex. Color-sensitive neurons are also orientation selective; as such, many achromatic patients have difficulty with form perception.

The hypothesis linking achromatopsia with deficits in form perception was carefully explored in the case study of a patient who suffered a stroke resulting in a small lesion near the temporo-occipital border in the right hemisphere. The damage was centered in area V4 and anterior parts of the visual cortex (Figure 5.38a). To assess the patient’s achromatopsia, a hue-matching experiment was performed in which a sample color was presented at the fovea, followed by a test color in one of the four quadrants of space. The patient’s task was to judge if the two colors were the same. The difference between the sample and test color was adjusted until the patient was performing correctly on 80 % of the trials, and this difference was measured separately for each quadrant. Regardless of the sample hue, the patient was severely impaired on the hue-matching task when the test color was presented in the upper left visual field (Figure 5.38b). The fact that the deficit was found only in the upper contralesional visual field is consistent with previous reports of achromatopsia.

The next order of business was to examine shape perception. Would the patient show similar deficits in shape perception in this quadrant? If so, what types of shape perception tasks would reveal the impairment? To answer these questions, a variety of tasks were administered. The stimuli are shown in Figure 5.39. On the basic visual discriminations of contrast, orientation, and motion, the patient’s performance was similar for all four quadrants and comparable to the performance of control participants. He showed impairment on tests of higher order shape perception, however; and again, this impairment was restricted to the upper left quadrant. For these tasks, shape information requires combining information from neurons that might detect simple properties such as line orientation. For example, the orientation of the line separating the two semicircles (Figure 5.39d) is defined only by the combination of the lengths of the individual stripes and their offset.

Characterizing area V4 as a “color” area is too simplistic. This area is part of secondary visual areas devoted to shape perception. Color can provide an important cue about an object’s shape. V4 may be invaluable for using color information as one cue to define the boundaries that separate the objects that form our visual environment.

Revisiting patient P.T. Let’s return to patient P.T., who we met at the beginning of the chapter. Recall that he had difficulty recognizing familiar places and objects following a stroke to his right hemisphere. Further examination revealed some puzzling features of his perceptual deficits.

P.T. was shown two paintings: one by Monet, depicting a subdued 19th-century countryman dressed in his Sunday suit; the other by Picasso, of a woman with a terrified expression (Figure 5.40). P.T. was asked to describe what he saw in each painting. When shown the Monet, he looked puzzled. He saw no definable forms, just an abstract blend of colors and shapes. His problem in interpreting the painting was consonant with the deficits he experienced at home. Yet he readily identified the figure in Picasso’s painting and pointed out that it was a woman, or perhaps a young girl. This dissociation is compelling, for most would readily agree that the Monet is more realistic. Picasso painted the parts of his work as separate units. He used sharp contrasts in brightness and vivid colors to highlight facial regions. Monet opted for a softer approach, in which parts are best seen in a continuous whole, with gradual changes in contrast and color. Can any of these factors account for P.T.’s performance in identifying the figures in Picasso and Monet?

The neurologist evaluating P.T. emphasized that the primary problem stemmed from a deficit in color perception. This hypothesis is in accord with one of the primary differences between the Monet and the Picasso. In the Monet painting, the boundaries between the face and the background are blended: Gradual variations in color demarcate the facial regions and separate them from the background landscape. A deficit in color perception provided a parsimonious account of the patient’s problems in recognizing faces and landscapes. The rolling green hills of an Oregon farm can blur into a homogeneous mass if a person cannot discern fine variations in color. In a similar way, each face has its characteristic coloration.

It seems equally plausible, however, that the problem stemmed from a deficit in contrast or contour perception. These features are salient in the Picasso and absent in the Monet. Indeed, we know from our recent discussion of V4 that color and shape perception are often conflated. It is clear that the patient’s stroke affected primarily the cortical projections of the pathways essential for color and form perception. In contrast, the cortical projections of the pathway involved in motion were intact. The patient had no trouble recognizing his wife as she moved from the stove to the kitchen table; indeed, P.T. commented that her idiosyncratic movement enabled him to recognize her.

In 1983, researchers at the Max Planck Institute in Munich reported a striking case of a woman who had incurred a selective loss of motion perception, or akinetopsia (Zihl et al., 1983). For this woman, whom we call M.P., perception was akin to viewing the world as a series of snapshots. Rather than seeing things move continuously in space, she saw moving objects appear in one position and then another. When pouring a cup of tea, M.P. would see the liquid frozen in air, like a glacier. She would fail to notice the tea rising in her cup and would be surprised when the cup overflowed. The loss of motion perception also made M.P. hesitant about crossing the street. As she noted, “When I’m looking at the car first, it seems far away. But then, when I want to cross the road, suddenly the car is very near” (Zihl et al., 1983, p. 315).

Examination revealed M.P.’s color and form perception to be intact. Her ability to perceive briefly presented objects and letters, for example, was within the normal range. Nonetheless, her ability to judge the direction and speed of moving objects was severely impaired. This deficit was most apparent with stimuli moving at high speeds. At speeds faster than 20°/s, M.P. never reported detecting the motion. She could see that a dot’s position had changed and hence could infer motion. But her percept was of two static images; there was no continuity from one image to the other. Even when presented with stimuli moving more slowly, M.P. was hesitant to report a clear impression of motion.

CT scans of M.P. revealed large, bilateral lesions involving the temporoparietal cortices. On each side, the lesions included posterior and lateral portions of the middle temporal gyrus. These lesions roughly corresponded to areas that participate in motion perception. Furthermore, the lesions were lateral and superior to human V4, including the area identified as V5, the human equivalent of area MT (Figure 5.41).

Although the case of M.P. has been cited widely for many years, the fact that similar patients have not been identified suggests that severe forms of akinetopsia result only from bilateral lesions. With unilateral lesions, the motion perception deficits are much more subtle (Plant et al., 1993). Perhaps people can perceive motion as long as human V5 is intact in at least one hemisphere. Motion, by definition, is a dynamic percept, one that typically unfolds over an extended period of time. With longer viewing times, signals from early visual areas in the impaired hemisphere have an opportunity to reach secondary visual areas in the unimpaired hemisphere. The receptive fields in primate area V5 are huge and have cells that can be activated by stimuli presented in either visual field.

Still, the application of transcranial magnetic stimulation (TMS; see Chapter 3) over human V5 can produce transient deficits in motion perception. In one such experiment, participants were asked to judge whether a stimulus moved up or down (Stevens et al., 2009). To make the task demanding, the displays consisted of a patch of dots, only some of which moved in the target direction; the rest moved in random directions. Moreover, the target was preceded and followed by “masking” stimuli in which all of the dots moved in random directions. Thus, the stimulus direction was visible during only a brief 100-ms window (Figure 5.42). TMS was applied over either V5 or a control region, the motor cortex. Performance of the motion task was disrupted by stimulation over V5, creating a transient form of akinetopsia.

FIGURE 5.41 Reconstruction of a lesion producing severe akinetopsia.
Three horizontal sections showing the patient’s bilateral lesions in the left and right hemispheres. Note that the lesions encompass area MTG in both hemispheres.

FIGURE 5.42 TMS over human V5 disrupts motion perception. (a) The stimulus was an 80 ms display of moving dots in which a small percentage of the dots moved in the same direction. This display was preceded and followed by displays in which the direction of motion for all of the dots was random. (b) Performance was impaired when the TMS was applied over V5, relative to two control conditions (TMS over motor cortex or no TMS). (c) When the timing of the TMS pulse was varied to either come before the stimulus (negative values) or after the stimulus (positive values), two epochs of disruption were identified.

One feature of TMS that makes it such an excellent research tool is that investigators can vary the timing of the magnetic pulses to determine the time of maximum disruption. Knowing when a disruption occurs can help locate where it is occurring. To the researchers’ surprise, TMS disrupted performance at two distinct intervals. One was when the pulse was applied about 100 ms before the onset of the target stimulus. The second was approximately 150 ms after the onset of the target stimulus. This latter timing isn’t so surprising. It coincides with estimates of when activity within V5 would be important for integrating motion information to determine the direction of a moving stimulus. Thus, the researchers assumed that the pulses applied at this point in time added noise to the representations in V5.

What was that first disruption, when the TMS pulse was delivered before the onset of the target stimulus? The phenomenon was puzzling. The deficit here is unlikely to be the direct result of a perturbation of V5 neurons, because if that were true, we should not see performance improve before falling off again. Two other hypotheses should be considered. First, TMS at this point might disrupt the observer’s attentional focus, making it hard to orient to the target stimulus. Second, TMS over V5 may not only cause neurons in V5 to fire but also trigger neural firing in V1 after a short delay. This second hypothesis is based on the understanding that cortical connectivity and processing along sensory pathways, and indeed, across the cortex, are almost always bidirectional. Although models of visual perception tend to emphasize that processing proceeds from a primary region such as V1 to a secondary visual area such as V5, prominent pathways also are going in the reverse direction. Based on the second hypothesis, the first dip in performance is due to the indirect effect of the TMS pulse on V1 activity, and the second dip in performance is due to the direct effect of the TMS pulse on V5 activity. This observation is roughly consistent with the temporal pattern of activity observed in single-cell recordings in these two areas in response to moving stimuli.

Almost all of the ascending axons from the LGN terminate in the primary visual cortex. An individual with damaged primary visual cortex is expected to be blind; and indeed, this is what is observed. The blindness may be incomplete, however. If the lesion is restricted to one half of the visual field, the loss of perception will be restricted to the contralateral side of space; such a deficit is referred to as hemianopia. Smaller lesions may produce more discrete regions of blindness, or scotomas. Patients with primary visual cortex lesions are unable to report seeing anything presented within a scotoma. As anatomists have shown, however, the cortex includes not only multiple visual pathways but also prominent subcortical visual pathways. These observations have led to some surprising findings showing that visual capabilities may persist even in the absence of the primary visual cortex.

Cortical and Subcortical Perception in the Hamster As mentioned previously, in humans about 90% of the optic nerve fibers project to the LGN. The other 10% project to other subcortical nuclei, and the most prominent projection is to the superior colliculus (SC). What’s more, the proportion of retinocollicular fibers is even larger in most other species.

The SC plays a critical role in producing eye movements. If this midbrain structure becomes atrophied, as in a degenerative disorder such as supranuclear palsy, eye movements become paralyzed. Stimulation of neurons in the SC can also trigger eye movements; the direction of movement depends on the stimulation site. Observations like this emphasize an important motor role for the SC, but it is also interesting to ask about the representation of the visual world in the SC. What kinds of visual behaviors are possible from this system?

Gerald Schneider (1969), at the Massachusetts Institute of Technology, provided an important insight into this question. Hamsters were trained to perform the two tasks illustrated in Figure 5.43. In one task, the hamsters were trained to turn their heads in the direction of a sunflower seed held in an experimenter’s hand (Figure 5.43a). The task was easy for hamsters because they have a strong propensity to find sunflower seeds and put them in their cheeks.

The second task presented more of a challenge. Here the animals were trained to run down a two-armed maze and enter the door behind which a sunflower seed was hidden (Figure 5.43b). The task required the animals to make simple visual discriminations, such as distinguishing between black and white doors or between doors with vertical or horizontal stripes. For normal hamsters, the discriminations are not taxing. Within a few trials, they became proficient in selecting the correct door in almost all trials.

After training, Schneider divided the hamsters into two experimental groups. One group received bilateral lesions of the visual cortex, including all of areas 17 and 18 (Figure 5.43c). For the second group, the superior colliculus was rendered nonfunctional by the ablation of its input fibers (Figure 5.43d). This strategy was necessary because direct lesions to the colliculus, which borders many brainstem nuclei that are essential for life, are likely to kill the animals.

The two lesions yielded a double dissociation. Cortical lesions severely impaired the animals’ performance on the visual identification tasks. The animals could run down the maze and had sufficient motor capabilities to enter one of the doors, but they could not discriminate black from white or horizontal from vertical stripes. In contrast, the animals with collicular lesions demonstrated no impairment on this task.

On the sunflower seed localization task, the deficits were reversed. Animals with cortical lesions were perfect at this task once they had recovered from the surgery. Yet animals with collicular lesions acted as though they were blind. They made no attempt to orient toward the seeds—and not because they were unmotivated or had a motor problem. If the seed brushed against a whisker, the animal rapidly turned toward it and gobbled it up.

These data provide compelling evidence for dissociable functions of the hamsters’ superior colliculus and visual cortex. The collicular lesions impaired their ability to orient toward the position of a stimulus, and the cortical lesions disrupted visual acuity. For the hamster, this double dissociation might be thought of as reflecting two systems: one devoted to spatial orientation, the other devoted to object identification. As we will see in the next chapter, the idea that the representation of the properties of a stimulus and its location may entail different neural pathways is also an important idea for understanding visual processing within the cortex. We will return to the issue of residual perception following damage to the primary visual cortex in Chapter 14 when we turn to the question of consciousness.

Each of our senses gives us unique information about the world we live in. Color is a visual experience; pitch is uniquely auditory. Even though the information provided by each sense is distinct, the resulting representation of the surrounding world is not one of disjointed sensations, but of a unified multi-sensory experience. A meal in a restaurant is more than just the taste of the food. Restaurant owners know that the visual presentation of the food and the surroundings, the background noise or the lack of it, the comfort of the chairs, the fragrances from the kitchen, the interaction with the server, all contribute to how you will rate the restaurant’s cooking—that is, the combined experience of all the senses affects the taste of the food. How much of that experience happens because it is expected? If all else is perfect, you may rate the food better than it actually is because you expect it to be in line with your other sensations. Or, in contrast, even if you are served the most delicious fettuccine in the world, if the restaurant has the fragrance of cabbage, a 4-year-old is screaming and kicking in the booth behind you, and a rude server delivers your meal on a greasy plate, you most likely will not judge the pasta to be so great. Much of what we experience is what we expect to experience. At a Washington, D.C., metro station, most people don’t expect to hear a virtuoso. When the virtuoso Joshua Bell, clad in jeans and a T-shirt, propped open his violin case for change and played six classical masterpieces on one of the finest-sounding violins ever made—a 1713 creation by Antonio Stradivari—only a handful of the hundreds of commuters passing by stopped to listen. A few nights earlier, they would have had to pay over $100 to hear Mr. Bell perform at a nearby concert hall. With our eyes closed and nose pinched, if we are asked to bite into an “apple” and guess whether it is a Fuji or a Golden Delicious, most of us will say one or the other. We wouldn’t be able to tell, at least in the first bite, that we have been tricked into biting an onion.

When you sit enthralled in a movie theater, staring up at the screen, you have the perception that the voices are coming from the actors. Nevertheless, the sounds are actually coming from the speakers located at the sides of the screen. How about the puppet sitting on the lap of the ventriloquist? We know that the ventriloquist is doing the talking, but we see the puppet moving his lips: We have the perception that it is the puppet who is talking. In both cases, the location of the auditory cue is “captured” by the location of the visual cue. We can study our sensory systems in isolation, but perception is really a synthetic process, one in which the organism uses all available information to converge on a coherent representation of the world.

A particularly powerful demonstration of the multimodal nature of perception comes from the world of speech perception. Most people think of speech as an inherently auditory process—we decipher the sounds of language to identify phonemes, combining these into words, sentences, and phrases (see Chapter 11). Speech perception can certainly occur if the input is limited to audition: We can readily understand a friend over the phone, and people who are congenitally blind learn to speak with minimal difficulty. If you are learning a new language, however, then that phone conversation is notoriously more difficult than if the conversation is face-to-face: The sounds we hear can be influenced by visual cues. This principle has been shown in what has come to be called the McGurk effect, in which the perception of speech—what you believe that you “hear”—is influenced by the lip movements that your eyes see. Examples of this compelling visual-auditory illusion can be found at www.youtube.com/watch?v=G-lN8vWm3m0.

Cross-modal capture effects aren’t limited to interactions between vision and audition. We can even be fooled into misidentifying an inanimate object as part of our body. In the rubber hand illusion, a rubber left hand is placed in a biologically plausible position on a table in full view of the subject, while her real left arm and hand are blocked from her view by a screen (see http://www.youtube.com/watch?v=TCQbygjG0RU). The researcher then runs a brush over the person’s hand (still blocked from her view) while performing the same action with a different brush in the corresponding direction over the rubber hand that the subject sees. After a couple of minutes, she will “feel” that the rubber hand is her own. If blindfolded and asked to point to her hand, she will point to the rubber hand rather than her own. Even more dramatic, if the experimenter suddenly reaches out and hits the rubber hand with a hammer, she is likely to scream.

These illusions work because they take advantage of correlations that are generally present between the senses in day-to-day life. The gestures of a speaker’s lips normally conform to the sounds we hear; when we see something close to our hand and feel something touching our hand, we correctly assume they are one and the same. It is only through the illusion that the processing can be teased apart and we realize that information from different sensory systems have been integrated in our brain.

How Does It Happen? How does the brain integrate information from the different senses to form a coherent percept? An older view was that some senses dominated others. In particular, vision was thought to dominate over all of the other senses, as in the examples given earlier. A more recent alternative is that the brain combines the input from multiple sensory systems about a particular external property (e.g., the location of a sound or touch), weighs the reliability of each sense, and makes an estimate, a decision, from this information about the external property in question. In this view, visual capture occurs because the brain judges visual information in most circumstances to be the most reliable and thus, gives it the most weight. The system is flexible, however, and the context can lead to a change in how information is weighed. When walking in the woods at dusk, we give more emphasis to somatosensory information as we step gingerly to avoid roots or listen carefully for breaking twigs that might signal that we’ve wandered off the path. It appears that other considerations are factored in and tip the weighting of information scales; in this case, the ambient light, or lack of it, favors the other senses.

So, sometimes the visual system can be overruled. A compelling demonstration of this is shown by the finding that when a flash of light is paired with two beeps, participants perceive the light as having flashed twice (Shams, 2000). This illusion, known as auditory driving, differs some from our previous examples. Instead of all of the modalities passing on information about one external property (the puppet or the rubber hand), here the stimulation of one sense (the ear) appears to affect the judgment about a property typically associated with a different sense. Specifically, the auditory beeps create a context of two events, a feature that the brain then applies to the light, creating a coherent percept.

How sensory processing is integrated between modalities is currently a hot topic. It includes the usual cast of questions: Where is information from different sensory systems integrated in the brain? Is it early or late in processing? What are the pathways that are involved?

Where Does It Happen? Brain regions containing neurons that respond to more than one sense are described as multisensory. Multisensory integration (Holmes & Spence, 2005) occurs at many different regions in the brain, both subcortically and cortically. Let’s look at some of the studies that have been exploring this question.

In animal studies, neurophysiological methods have been especially useful: Once an electrode has been placed in a targeted brain region, the animal can be presented with a range of stimuli to see if, and by what, the region is activated. For instance, when exploring visual responses, the researcher might vary the position of the stimulus, or its color or movement. To evaluate multisensory processing, the researcher can present stimuli along different sensory channels, asking not only if the cell responds to more than one sense but also about the relationship between the responses to stimuli from different senses.

Subcortical: Superior Colliculus. One well-studied multimodal site is the superior colliculus, the subcortical midbrain region that we discussed earlier in regard to eye movements. The superior colliculus contains orderly topographic maps of the environment in visual, auditory, and even tactile domains (Figure 5.45). Many cells in the superior colliculus show multisensory properties, being activated by inputs from more than one sensory modality. These neurons combine information from different sensory channels and integrate that information. In fact, the response of the cell is stronger when there are inputs from multiple senses compared to when the input is from a single modality (Stein, 2004). Such enhanced responses are most effective when a unimodal stimulus fails to produce a response on its own. In this way the combination of weak, even subthreshold, unimodal signals can be detected and cause participants to orient toward the stimulus. Multisensory signals are also treated by the brain as more reliable than signals from a single sensory channel. A rustling sound in the grass could indicate the presence of a snake, or just the rising evening breeze. But if that sound is combined with a glimmer of something slithering along, you can bet the brain will generate a fast-response eye movement to verify the presence of a snake.

Integration effects require that the different stimuli be coincident in both space and time. For example, if a visual event is spatially and temporally synchronous with a loud noise, as in the auditory driving example described earlier, the resulting multisensory response will be enhanced. If, however, the sound originates from a different location than the light, or is not temporally synchronized with the light, the response of the collicular cell will be lower than if either stimulus were presented alone. Such effects again demonstrate how the brain weights information in terms of its reliability. In the natural world, we have learned that visual and auditory cues are usually closely synchronized; we can learn that a distant visual event such as a flash of lightning will be followed by a crack of thunder. Because they are not coincident in time and space, however, our orienting system here will be driven by just one or the other, especially since these signals can be quite intense.

Cortical Processing. Multisensory activity is also observed in many cortical regions. The superior temporal sulcus (STS) is known to have connections both coming from and going to the various sensory cortices. Neurophysiologists have identified cells in the STS of monkeys that respond to visual, auditory, and somatosensory stimuli (Hikosaka et al., 1988).

Functional MRI has also been used to identify areas exhibiting multisensory areas of the cortex. The crude resolution of this technique makes it impossible to determine if the BOLD response reflects the activity of multisensory neurons or neighboring clusters of neurons that respond to a single modality. Researchers can build on the ideas of multisensory integration, however, to ask if the activation reflects the combination of different sensory cues. For example, the STS in the left hemisphere is active when people are actively engaged in lip-reading (something that we unconsciously use during normal speech comprehension), but not when the sounds are mismatched to the lip movements (Calvert et al., 1997). Other brain regions showing similar sensory integration effects include various regions of the parietal and frontal lobes, as well as the hippocampus (Figure 5.46).

With careful study, we can actually see multisensory effects even in areas that are traditionally thought to be sensory specific. For instance, in one fMRI study, activation in auditory cortex was greater when the sounds were accompanied by simultaneous visual stimulation (Kayser et al., 2007). Given the slow rise time of the BOLD response, this increase may have been more of a preparatory response that treated the visual signals as a cue for sounds. Event-related potential (ERP) studies have found, however, that the very early visual component of the ERP wave is enhanced when the visual stimulus is presented close in space to a corresponding tactile stimulus (Kennett et al., 2001).

Vincenzo Romei (2007) and his colleagues at the University of Geneva have sought to understand how early sensory areas might interact to support multisensory integration. Participants in one of their studies were required to press a button as soon as they detected a stimulus. The stimulus could be a light, a sound, or both. To disrupt visual processing, the researchers applied a TMS pulse over the visual cortex just after the stimulus onset. As expected, the response time (RT) to the visual stimulus was slower on trials in which the TMS pulse was applied compared to trials without TMS. But surprisingly, the RT to the auditory stimulus was faster after TMS over the visual cortex.

Why might disruption of the visual cortex improve a person’s ability to detect a sound? One possibility is that the two sensory systems are in competition with one another. Thus, TMS of the visual cortex handicaps a competitor of auditory cortex. Alternatively, neurons in visual cortex that are activated by the TMS pulse might produce signals that are sent to auditory cortex (as part of a multisensory processing pathway), and in this way enhance auditory cortex activity and produce faster RTs to the sounds (Figure 5.47).

Romei came up with a clever way to evaluate these two hypotheses by looking at the reverse situation, asking if an auditory stimulus could enhance visual perception. When TMS is applied over visual cortex, people report seeing phosphenes—an illusory flash of light. Phosphenes can also be produced mechanically by rubbing the eye. (The next time you go to the Louvre in Paris and stand in front of the huge epic painting of the Raft of the Medusa by Géricault, you can wow your neuroscientist friends with this bit of trivia: The word phosphene was coined by J. B. H. Savigny, the ship surgeon of the Méduse.) Romei first determined the intensity level of TMS required to produce phosphenes for each person. He then randomly stimulated the participants at a level that was a bit below the threshold in one of two conditions: alone or concurrently with an auditory stimulus. At this subthreshold level, the participants perceived phosphenes when the auditory stimulus was present, but not when the TMS pulse was presented alone. This finding supports the hypothesis that auditory and visual stimuli can enhance perception in the other sensory modality.

What are the Pathways? All the hubbub about multisensory processing has spawned several hypotheses about the pathways and connections between the processing areas and the resulting way that the processing occurs. The most radical suggestion is that the entire neocortex is in some sense multisensory, and the initial integration has occurred subcortically (Figure 5.48a). We do know from neuroanatomy that there is multisensory input to the cortex from the thalamus, but it would be an exaggeration to think that the entire cortex is multisensory. A lesion of primary visual cortex produces a profound and permanent blindness with no real effect on the other senses (or, if anything, some enhanced sensitivity in the other senses). The primary sensory cortical regions, and even secondary sensory regions, are clearly dedicated to a single modality. A less radical version is that the cortex has specific sensory areas, but they contain some multisensory interneurons (Figure 5.48b).

Alternatively, multisensory integration may involve projections originating in modality-specific cortical areas. These projections could go from one sensory region to another, allowing for fast modulation within primary and secondary sensory regions (Figure 5.48c). Or, the projections could be to multisensory convergence zones in the cortex, which in more traditional models of sensory function were referred to as association sensory areas. In these models, cross-modal influences on early sensory signals occur via feedback connections from the convergence zones to sensory-specific areas of the cortex (Figure 5.48d). All of these ideas likely contain some degree of truth.

As we have pointed out repeatedly, the sensory systems of the brain have evolved to reconstruct the external environment. This process is surely facilitated by exploiting all of the available information.

J.W. experiences the world differently from most people. He tastes words. The word exactly, for example, tastes like yogurt, and the word accept tastes like eggs. Most conversations are pleasant tasting; but when J.W. is tending bar, he cringes whenever Derek, a frequent customer, shows up. For J.W., the word Derek tastes of earwax!

This phenomenon, in which the senses are mixed, is known as synesthesia, from the Greek syn– (“union” or “together”) and aesthesis (“sensation”). Synesthesia is characterized by an idiosyncratic union between (or within) sensory modalities. Tasting words is an extremely rare form of synesthesia. More common are synesthesias in which people hear words or music as colors, or see achromatic lettering (as in books or newspapers) as colored.

The frequency of synesthesia is hard to know, given that many individuals are unaware that their multisensory percepts are odd: Estimates range from as rare as one in 2,000 to as high as one in 200. Synesthesia tends to recur in families, indicating that at least some forms have a genetic basis (Baron-Cohen et al., 1996; Smilek et al., 2005). If you think that you may experience some form of synesthesia, you can find out by taking the tests at this website: http://synesthete.org/.

Colored-grapheme synesthesia, in which black or white letters or digits are perceived in assorted colors, is the best-studied form of synesthesia. A synesthete might report “seeing” the letter A as red, the letter B as yellow, and so forth for the entire set of characters, as in the example shown in Figure 5.49. The appearance of color is a feature of many forms of synesthesia. In colored hearing, colors are experienced for spoken words or for sounds like musical notes. Colored touch and colored smell have also been reported. Much less common are synesthetic experiences that involve other senses. J.W. experiences taste with words; other rare cases have been reported in which touching an object induces specific tastes.

The associations are idiosyncratic for each synesthete. One person might see the letter B as red, another as green. Although the synesthetic associations are not consistent across individuals, they are consistent over time for an individual. A synesthete who reports the letter B as red when tested the first time in the lab will have the same percept if retested a few months later.

Given that synesthesia is such a personal experience, researchers have had to come up with clever methods to verify and explore this unique phenomenon. One approach with colored-grapheme synesthesia is to create modified versions of the Stroop task. As described in Chapter 3 (page 78), the Stroop task requires a person to name the color of written words. For instance, if the word green is written in red ink, the subject is supposed to say “red.” In the synesthetic variant of the Stroop task with a colored-grapheme synesthete, the stimuli are letters, and the key manipulation is whether the colors of the letters are congruent or incongruent to the individual’s synesthetic palette. For the example in Figure 5.49, when the letter A is presented in red, the physical color and synesthetic color are congruent. However, if the A is presented in green, the physical and concurrent colors are incongruent. Synesthetes are faster to name the colors of the letters when the physical color matches the concurrent colors for the particular letter (Mattingley et al., 2001). People without synesthesia, of course, do not show this effect. To them, any color–letter pairing is equally acceptable.

Brain-imaging studies indicate that the multisensory experience of synesthesia arises and is manifest at various stages along the visual pathway. Jeffrey Gray at King’s College in London performed an fMRI study with a group of individuals who had colored-hearing synesthesia (Nunn et al., 2002). When listening to words, these individuals reported seeing specific colors; when listening to tones, they had no visual experience. Compared to control participants, the synesthetes showed increased activation in V4, similar to what we have seen in other studies of illusory color perception, and in the STS, one of the brain regions associated with multimodal perception. Other studies have shown recruitment of the left medial lingual gyrus (a higher-order color processing area previously implicated in color knowledge) in synesthetes during the perception of colored-grapheme synesthesia (Rich et al., 2006).

A different approach is to ask if synesthesia is the result of abnormal anatomical connections. For example, do synesthetes have more connectivity between sensory regions than non-synesthetes? Using diffusion tensor imaging (DTI), Steven Scholte (2007) and his colleagues at the University of Amsterdam showed that grapheme– color synesthetes had greater anisotropic diffusion, a marker of larger white matter tracts, in the right inferior temporal cortex, the left parietal cortex, and bilaterally in the frontal cortex (green lines in Figure 5.50). Moreover, the researchers found that individual differences in the amount of connectivity in the inferior temporal cortex differentiated between subtypes of synesthetes. Participants who saw color in the outside world (known as “projectors”) had greater connectivity in the inferior temporal cortex compared with those who saw color in their “mind’s eye” only (known as “associators”).

As we have just seen, people with synesthesia provide a dramatic example of how the brain is able to link information between distinct sensory systems. The extent of the connectivity between sensory systems is also revealed by studies on people who are deprived of input from one of their senses. When a person is blind, what happens to those regions of the brain that are usually used for visual perception? Might this unused neural tissue be able to reorganize to process other information, as it does on the somatosensory cortex (see Figure 5.16)? Is the situation for individuals who have been blind since birth different from that of individuals who became blind after having had vision?

The results of one PET study suggest that a remarkable degree of functional reorganization goes on (Sadato et al., 1996). The participants in this study included people with normal vision and people who were congenitally blind—that is, blind from birth. The participants were scanned under two experimental conditions. In one condition, they were simply required to sweep their fingers back and forth over a rough surface covered with dots. In the second condition, they were given tactile discrimination tasks such as deciding whether two grooves in the surface were the same or different. Blood flow in the visual cortex during each of these tasks was compared to that during a rest condition in which the participants were scanned while keeping their hands still.

Amazingly, changes in activation in the visual cortex were in opposite directions for the two groups. For the sighted participants, a significant drop in activation was found in the primary visual cortex during the tactile discrimination tasks. Analogous decreases in the auditory or somatosensory cortex occurred during visual tasks. Therefore, as attention was directed to one modality, activation (as measured by blood flow) decreased in other sensory systems. In blind participants, however, the activation in the primary visual cortex increased during discrimination tasks, but only when they were actively using the tactile information. Interestingly, a second group of participants, who had become blind early in childhood (before their fifth year), also showed the same recruitment of visual cortex when performing the tactile discrimination task.

A second experiment explored the same issue but used a task that is of great practical value to the blind: reading Braille (Sadato et al., 1998). Here, the participants explored strings of eight Braille letters and had to decide whether the strings formed a word. In accord with the results of the first study, activation of the primary and secondary visual cortex increased during Braille reading in comparison with the resting state, but only in the blind participants.

FIGURE 5.51 Perceptual and neural changes resulting from extended visual deprivation in sighted individuals.
(a) fMRI activation during tactile exploration. By Day 5, the blindfolded group showed greater activation than the controls in the occipital cortex. This effect disappeared after the blindfold was removed. (b) Performance on tactile acuity after one or five days of practice. Lower values correspond to greater sensitivity. (Green: blindfolded participants; Red: Controls.) (c) Difference in occipital activation between blindfolded and control participants across days.

Of course the term visual cortex is a misnomer when applied to blind individuals. The results of the studies just described indicate that tissue, which during normal development will become sensitive to visual inputs, can be exploited in a radically different manner when the environmental context is changed—for example, when all visual input is lost. Currently, it is unclear how tactile information ends up activating neurons in the visual cortex of blind people. One possibility is that somatosensory projections to thalamic relays spread into the nearby lateral geniculate nucleus, exploiting the geniculostriate pathway. This hypothesis is unlikely, since the activation changes in the blind participants’ visual cortices were bilateral. Somatosensory inputs to the thalamus are strictly lateralized. Because they performed the tactile tasks with the right hand, the blood-flow changes should have been restricted to the left hemisphere. A more viable hypothesis is that a massive reorganization of corticocortical connections follows peripheral blindness. The sensory-deprived visual cortex is taken over, perhaps through back-projections originating in polymodal association cortical areas.

Alvaro Pascual-Leone and his colleagues at Harvard Medical School (Merabet et al., 2008) have studied cortical plasticity effects that occur when sighted volunteers are deprived of visual information for an extended period. These participants were blindfolded for 5 days and received intensive Braille training (Figure 5.51). A matched control group was given the same training, but they were not blindfolded. At the end of training, the blindfolded participants could discriminate Braille letters better than the nonblindfolded participants did; those who wore blindfolds were also better at other tactile discrimination tasks. Furthermore, fMRI tests of these participants revealed activation in the visual cortex during tactile stimulation of the right or left fingertips, even with stimuli that would not be expected to generate visual images. Interestingly, just 20 hours after the blindfold was removed (on day 6), the activation in visual cortex during tactile stimulation disappeared (Figure 5.51a, c). These data argue that, when deprived of normal input, the adult visual system rapidly reorganizes to become more proficient in processing information from the other senses.

Although these studies are a dramatic demonstration of cortical plasticity, the results also suggest a neurobiological mechanism for the greater nonvisual perceptual acuity exhibited by blind people. Indeed, Louis Braille’s motivation to develop his tactile reading system was spurred by his belief that vision loss was offset by heightened sensitivity in the fingertips. One account of this compensation focuses on nonperceptual mechanisms. Though the sensory representation of somatosensory information is similar for blind and sighted participants, the former group is not distracted by vision (or visual imagery). If the focus of attention is narrowed, somatosensory information can be used more efficiently. The imaging results reviewed here, though, suggest a more perceptual account: Sensitivity increases because more cortical tissue is devoted to representing nonvisual information.

Summary

The five basic sensory systems of audition, olfaction, gustation, somatosensation, and vision allow us to interpret the environment. Each sense involves unique pathways and processes to translate external stimuli into neural signals that are interpreted by the brain. Within each sense, specialized sensory mechanisms have evolved to solve computational problems to facilitate and enhance our perceptual capabilities. As shown in neuroimaging and neuropsychological studies, specialization is found across the sensory cortices of the brain; thus, people may retain the ability to see, even in the absence of cortical mechanisms for color or motion perception. In extreme situations of sensory deprivation, the cortical systems for perception may become radically reorganized. Even in people with intact sensory systems, the five senses do not work in isolation, but rather work in concert to construct a rich interpretation of the world. It is this integration that underlies much of human cognition and allows us to survive, and indeed thrive, in a multisensory world.

Key Terms

achromatopsia (p. 201)

akinetopsia (p. 204)

area MT (p. 190)

area V4 (p. 194)

chemical senses (p. 176)

corpuscle (p. 179)

cortical visual area (p. 189)

extrastriate visual area (p. 189)

fovea (p. 186)

ganglion cells (p. 186)

glomerulus (p.173) hemianopia (p. 206)

inferior colliculus (p. 169)

interaural time (p. 171)

lateral geniculate nucleus (LGN) (p. 187)

medial geniculate nucleus (MGN) (p. 167)

multisensory integration (p. 167)

nociceptor (p. 179)

odorant (p. 173)

photoreceptor (p. 185)

primary auditory cortex (A1) (p. 169)

primary gustatory cortex (p. 176)

primary olfactory cortex (p. 173)

primary somatosensory cortex (S1) (p. 180)

primary visual cortex (V1) (p. 187)

proprioception (p. 180)

receptive field (p. 187)

retina (p. 185)

retinotopic map (p. 187)

scotoma (p. 206)

secondary somatosensory cortex (S2) (p. 180)

superior colliculus (p. 187)

synesthesia (p. 212)

tastant (p. 176)

Thought Questions

Compare and contrast the functional organization of the visual and auditory systems. What computational problems must each system solve, and how are these solutions achieved in the nervous system?
A person arrives at the hospital in a confused state and appears to have some impairment in visual perception. As the attending neurologist, you suspect that the person has had a stroke. How would you go about examining the patient to determine at which level in the visual pathways the damage has occurred? Emphasize the behavioral tests you would administer, but feel free to make predictions about what you expect to see on MRI scans.
Define the physiological concepts of receptive field and visual area. How is the receptive field of a cell established? How are the boundaries between visual areas identified by researchers using either single-cell recording methods or fMRI?
This chapter has focused mainly on salient visual properties such as color, shape, and motion. In looking around the environment, do you think these properties seem to reflect the most important cues for a highly skilled visual creature? What other sources of information might an adaptive visual system exploit?
How might abnormalities in multisensory processing (e.g., synesthesia) be important for understanding how and why information becomes integrated across different sensory channels? Similarly, given the plasticity of the brain, does it even make sense to talk about a “visual system” or an “auditory system”?


FIGURE 5.3 Overview of the auditory pathway. The hair cells of the cochlea are the primary receptors. The output from the auditory nerve projects to the cochlear nuclei in the brainstem. Ascending fibers reach the auditory cortex following synapses in the inferior colliculus and medial geniculate nucleus.