Featured Research

from universities, journals, and other organizations

Scientists decode brain waves to eavesdrop on what we hear

Date:
January 31, 2012
Source:
University of California - Berkeley
Summary:
Neuroscientists and surgeons have recorded electrical activity in the temporal lobe -- the seat of the auditory system -- to discover how the brain encodes sound. Their model allows them to predict what a person heard based solely on temporal lobe activity. If, as studies suggest, internal "imagined" conversations activate similar areas of the temporal lobe, it may be possible to hear the internal verbalizations of people who cannot talk because of paralysis or stroke.

An X-ray CT scan of the head of one of the volunteers, showing electrodes distributed over the brain’s temporal lobe, where sounds are processed.
Credit: Adeen Flinker, UC Berkeley

Neuroscientists may one day be able to hear the imagined speech of a patient unable to speak due to stroke or paralysis, according to University of California, Berkeley, researchers.

These scientists have succeeded in decoding electrical activity in the brain's temporal lobe -- the seat of the auditory system -- as a person listens to normal conversation. Based on this correlation between sound and brain activity, they then were able to predict the words the person had heard solely from the temporal lobe activity.

"This research is based on sounds a person actually hears, but to use it for reconstructing imagined conversations, these principles would have to apply to someone's internal verbalizations," cautioned first author Brian N. Pasley, a post-doctoral researcher in the center. "There is some evidence that hearing the sound and imagining the sound activate similar areas of the brain. If you can understand the relationship well enough between the brain recordings and sound, you could either synthesize the actual sound a person is thinking, or just write out the words with a type of interface device."

"This is huge for patients who have damage to their speech mechanisms because of a stroke or Lou Gehrig's disease and can't speak," said co-author Robert Knight, a UC Berkeley professor of psychology and neuroscience. "If you could eventually reconstruct imagined conversations from brain activity, thousands of people could benefit."

In addition to the potential for expanding the communication ability of the severely disabled, he noted, the research also "is telling us a lot about how the brain in normal people represents and processes speech sounds."

Pasley and his colleagues at UC Berkeley, UC San Francisco, University of Maryland and The Johns Hopkins University report their findings Jan. 31 in the open-access journal PLoS Biology.

Help from epilepsy patients

They enlisted the help of people undergoing brain surgery to determine the location of intractable seizures so that the area can be removed in a second surgery. Neurosurgeons typically cut a hole in the skull and safely place electrodes on the brain surface or cortex -- in this case, up to 256 electrodes covering the temporal lobe -- to record activity over a period of a week to pinpoint the seizures. For this study, 15 neurosurgical patients volunteered to participate.

Pasley visited each person in the hospital to record the brain activity detected by the electrodes as they heard 5-10 minutes of conversation. Pasley used this data to reconstruct and play back the sounds the patients heard. He was able to do this because there is evidence that the brain breaks down sound into its component acoustic frequencies -- for example, between a low of about 1 Hertz (cycles per second) to a high of about 8,000 Hertz -that are important for speech sounds.

Pasley tested two different computational models to match spoken sounds to the pattern of activity in the electrodes. The patients then heard a single word, and Pasley used the models to predict the word based on electrode recordings.

"We are looking at which cortical sites are increasing activity at particular acoustic frequencies, and from that, we map back to the sound," Pasley said. He compared the technique to a pianist who knows the sounds of the keys so well that she can look at the keys another pianist is playing in a sound-proof room and "hear" the music, much as Ludwig van Beethoven was able to "hear" his compositions despite being deaf.

The better of the two methods was able to reproduce a sound close enough to the original word for Pasley and his fellow researchers to correctly guess the word.

"We think we would be more accurate with an hour of listening and recording and then repeating the word many times," Pasley said. But because any realistic device would need to accurately identify words heard the first time, he decided to test the models using only a single trial.

"This research is a major step toward understanding what features of speech are represented in the human brain" Knight said. "Brian's analysis can reproduce the sound the patient heard, and you can actually recognize the word, although not at a perfect level."

Knight predicts that this success can be extended to imagined, internal verbalizations, because scientific studies have shown that when people are asked to imagine speaking a word, similar brain regions are activated as when the person actually utters the word.

"With neuroprosthetics, people have shown that it's possible to control movement with brain activity," Knight said. "But that work, while not easy, is relatively simple compared to reconstructing language. This experiment takes that earlier work to a whole new level."

Based on earlier work with ferrets

The current research builds on work by other researchers about how animals encode sounds in the brain's auditory cortex. In fact, some researchers, including the study's coauthors at the University of Maryland, have been able to guess the words ferrets were read by scientists based on recordings from the brain, even though the ferrets were unable to understand the words.

The ultimate goal of the UC Berkeley study was to explore how the human brain encodes speech and determine which aspects of speech are most important for understanding.

"At some point, the brain has to extract away all that auditory information and just map it onto a word, since we can understand speech and words regardless of how they sound," Pasley said. "The big question is, What is the most meaningful unit of speech? A syllable, a phone, a phoneme? We can test these hypotheses using the data we get from these recordings."

Coauthors of the study are electrical engineers Stephen V. David, Nima Mesgarani and Shihab A. Shamma of the University of Maryland; Adeen Flinker of UC Berkeley's Helen Wills Neuroscience Institute; and neurologist Nathan E. Crone of The Johns Hopkins University in Baltimore, Md. The work was done principally in the labs of Robert Knight at UC Berkeley and Edward Chang, a neurosurgeon at UCSF.

Chang and Knight are members of the Center for Neural Engineering and Prostheses, a joint UC Berkeley/UCSF group focused on using brain activity to develop neural prostheses for motor and speech disorders in disabling neurological disorders.

The work is supported by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health and the Humboldt Foundation.


Story Source:

The above story is based on materials provided by University of California - Berkeley. The original article was written by Robert Sanders. Note: Materials may be edited for content and length.


Journal Reference:

  1. Brian N. Pasley, Stephen V. David, Nima Mesgarani, Adeen Flinker, Shihab A. Shamma, Nathan E. Crone, Robert T. Knight, Edward F. Chang. Reconstructing Speech from Human Auditory Cortex. PLoS Biology, 2012; 10 (1): e1001251 DOI: 10.1371/journal.pbio.1001251

Cite This Page:

University of California - Berkeley. "Scientists decode brain waves to eavesdrop on what we hear." ScienceDaily. ScienceDaily, 31 January 2012. <www.sciencedaily.com/releases/2012/01/120131175158.htm>.
University of California - Berkeley. (2012, January 31). Scientists decode brain waves to eavesdrop on what we hear. ScienceDaily. Retrieved August 27, 2014 from www.sciencedaily.com/releases/2012/01/120131175158.htm
University of California - Berkeley. "Scientists decode brain waves to eavesdrop on what we hear." ScienceDaily. www.sciencedaily.com/releases/2012/01/120131175158.htm (accessed August 27, 2014).

Share This




More Mind & Brain News

Wednesday, August 27, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Have You Ever Been 'Sleep Drunk?' 1 in 7 Has

Have You Ever Been 'Sleep Drunk?' 1 in 7 Has

Newsy (Aug. 26, 2014) A study published in the journal "Neurology" interviewed more than 19,000 people and found 15 percent suffer from being "sleep drunk." Video provided by Newsy
Powered by NewsLook.com
Does Medical Marijuana Reduce Painkiller Overdose Deaths?

Does Medical Marijuana Reduce Painkiller Overdose Deaths?

Newsy (Aug. 26, 2014) A new study found fewer deaths from prescription drug overdoses in states that have legalized medical marijuana. But experts disagree on the results. Video provided by Newsy
Powered by NewsLook.com
Heart Group: E-Cigarettes May Help Smokers Quit

Heart Group: E-Cigarettes May Help Smokers Quit

AP (Aug. 25, 2014) The American Heart Association's first policy statement on electronic cigarettes backs them as a last resort to help smokers quit and calls for more regulation to keep them away from youth. (Aug. 25) Video provided by AP
Powered by NewsLook.com
Doctors Push For Later Start Times As School Year Kicks Off

Doctors Push For Later Start Times As School Year Kicks Off

Newsy (Aug. 25, 2014) The American Academy of Pediatrics is the latest group pushing for middle schools and high schools to start later, for the sake of their kids. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

    Environment News

    Technology News



    Save/Print:
    Share:

    Free Subscriptions


    Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

    Get Social & Mobile


    Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

    Have Feedback?


    Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
    Mobile: iPhone Android Web
    Follow: Facebook Twitter Google+
    Subscribe: RSS Feeds Email Newsletters
    Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins