Featured Research

from universities, journals, and other organizations

Look at what I'm saying: Engineers show brain depends on vision to hear

Date:
September 4, 2013
Source:
University of Utah
Summary:
Bioengineers have discovered our understanding of language may depend more heavily on vision than previously thought: under the right conditions, what you see can override what you hear.

Bioengineers measured electrical signals on the brain surface of four adults with epilepsy to understand how the brain processes speech. Their findings show our understanding of language may depend more heavily on vision than previously thought.
Credit: Department of Neurosurgery, University of Utah

University of Utah bioengineers discovered our understanding of language may depend more heavily on vision than previously thought: under the right conditions, what you see can override what you hear. These findings suggest artificial hearing devices and speech-recognition software could benefit from a camera, not just a microphone.

Related Articles


"For the first time, we were able to link the auditory signal in the brain to what a person said they heard when what they actually heard was something different. We found vision is influencing the hearing part of the brain to change your perception of reality -- and you can't turn off the illusion," says the new study's first author, Elliot Smith, a bioengineering and neuroscience graduate student at the University of Utah. "People think there is this tight coupling between physical phenomena in the world around us and what we experience subjectively, and that is not the case."

The brain considers both sight and sound when processing speech. However, if the two are slightly different, visual cues dominate sound. This phenomenon is named the McGurk effect for Scottish cognitive psychologist Harry McGurk, who pioneered studies on the link between hearing and vision in speech perception in the 1970s. The McGurk effect has been observed for decades. However, its origin has been elusive.

In the new study, which appears today in the journal PLOS ONE, the University of Utah team pinpointed the source of the McGurk effect by recording and analyzing brain signals in the temporal cortex, the region of the brain that typically processes sound.

Working with University of Utah bioengineer Bradley Greger and neurosurgeon Paul House, Smith recorded electrical signals from the brain surfaces of four severely epileptic adults (two male, two female) from Utah and Idaho. House placed three button-sized electrodes on the left, right or both brain hemispheres of each test subject, depending on where each patient's seizures were thought to originate. The experiment was done on volunteers with severe epilepsy who were undergoing surgery to treat their epilepsy.

These four test subjects were then asked to watch and listen to videos focused on a person's mouth as they said the syllables "ba," "va," "ga" and "tha." Depending on which of three different videos were being watched, the patients had one of three possible experiences as they watched the syllables being mouthed:

-- The motion of the mouth matched the sound. For example, the video showed "ba" and the audio sound also was "ba," so the patients saw and heard "ba."

-- The motion of the mouth obviously did not match the corresponding sound, like a badly dubbed movie. For example, the video showed "ga" but the audio was "tha," so the patients perceived this disconnect and correctly heard "tha."

-- The motion of the mouth only was mismatched slightly with the corresponding sound. For example, the video showed "ba" but the audio was "va," and patients heard "ba" even though the sound really was "va." This demonstrates the McGurk effect -- vision overriding hearing.

By measuring the electrical signals in the brain while each video was being watched, Smith and Greger could pinpoint whether auditory or visual brain signals were being used to identify the syllable in each video. When the syllable being mouthed matched the sound or didn't match at all, brain activity increased in correlation to the sound being watched. However, when the McGurk effect video was viewed, the activity pattern changed to resemble what the person saw, not what they heard. Statistical analyses confirmed the effect in all test subjects.

"We've shown neural signals in the brain that should be driven by sound are being overridden by visual cues that say, 'Hear this!'" says Greger. "Your brain is essentially ignoring the physics of sound in the ear and following what's happening through your vision."

Greger was senior author of the study as an assistant professor of bioengineering at the University of Utah. He recently took a faculty position at Arizona State University.

The new findings could help researchers understand what drives language processing in humans, especially in a developing infant brain trying to connect sounds and lip movement to learn language. These findings also may help researchers sort out how language processing goes wrong when visual and auditory inputs are not integrated correctly, such as in dyslexia, Greger says.

Equipment used for these experiments was purchased as part of a Defense Advance Research Project Agency project to control and communicate with prostheses, in particular for recording signals to control a prosthetic limb. Additional funding for the study was provided by the University of Utah Research Foundation.


Story Source:

The above story is based on materials provided by University of Utah. Note: Materials may be edited for content and length.


Journal Reference:

  1. Elliot Smith, Scott Duede, Sara Hanrahan, Tyler Davis, Paul House, Bradley Greger. Seeing Is Believing: Neural Representations of Visual Stimuli in Human Auditory Cortex Correlate with Illusory Auditory Perceptions. PLoS ONE, 2013; 8 (9): e73148 DOI: 10.1371/journal.pone.0073148

Cite This Page:

University of Utah. "Look at what I'm saying: Engineers show brain depends on vision to hear." ScienceDaily. ScienceDaily, 4 September 2013. <www.sciencedaily.com/releases/2013/09/130904203532.htm>.
University of Utah. (2013, September 4). Look at what I'm saying: Engineers show brain depends on vision to hear. ScienceDaily. Retrieved December 20, 2014 from www.sciencedaily.com/releases/2013/09/130904203532.htm
University of Utah. "Look at what I'm saying: Engineers show brain depends on vision to hear." ScienceDaily. www.sciencedaily.com/releases/2013/09/130904203532.htm (accessed December 20, 2014).

Share This


More From ScienceDaily



More Mind & Brain News

Saturday, December 20, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Researchers Test Colombian Village With High Alzheimer's Rates

Researchers Test Colombian Village With High Alzheimer's Rates

AFP (Dec. 19, 2014) In Yarumal, a village in N. Colombia, Alzheimer's has ravaged a disproportionately large number of families. A genetic "curse" that may pave the way for research on how to treat the disease that claims a new victim every four seconds. Duration: 02:42 Video provided by AFP
Powered by NewsLook.com
Double-Amputee Becomes First To Move Two Prosthetic Arms With His Mind

Double-Amputee Becomes First To Move Two Prosthetic Arms With His Mind

Buzz60 (Dec. 19, 2014) A double-amputee makes history by becoming the first person to wear and operate two prosthetic arms using only his mind. Jen Markham has the story. Video provided by Buzz60
Powered by NewsLook.com
Prenatal Exposure To Pollution Might Increase Autism Risk

Prenatal Exposure To Pollution Might Increase Autism Risk

Newsy (Dec. 18, 2014) Harvard researchers found children whose mothers were exposed to high pollution levels in the third trimester were twice as likely to develop autism. Video provided by Newsy
Powered by NewsLook.com
Yoga Could Be As Beneficial For The Heart As Walking, Biking

Yoga Could Be As Beneficial For The Heart As Walking, Biking

Newsy (Dec. 17, 2014) Yoga can help your weight, blood pressure, cholesterol and heart just as much as biking and walking does, a new study suggests. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:

Strange & Offbeat Stories


Health & Medicine

Mind & Brain

Living & Well

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins