Featured Research

from universities, journals, and other organizations

Music to my eyes: Device converting images into music helps visually impaired find things with ease

Date:
July 5, 2012
Source:
IOS Press BV
Summary:
Sensory substitution devices (SSDs) use sound or touch to help the visually impaired perceive the visual scene surrounding them. The ideal SSD would assist not only in sensing the environment but also in performing daily activities based on this input. For example, accurately reaching for a coffee cup, or shaking a friend's hand. In a new study, scientists trained blindfolded sighted participants to perform fast and accurate movements using a new SSD, called EyeMusic.

Left: An illustration of the EyeMusic SSD, showing a user with a camera mounted on the glasses, and scalp headphones, hearing musical notes that create a mental image of the visual scene in front of him. He is reaching for the red apple in a pile of green ones. Top right: close-up of the glasses-mounted camera and headphones; bottom right: hand-held camera pointed at the object of interest.
Credit: Maxim Dupliy, Amir Amedi and Shelly Levy-Tzedek

Sensory substitution devices (SSDs) use sound or touch to help the visually impaired perceive the visual scene surrounding them. The ideal SSD would assist not only in sensing the environment but also in performing daily activities based on this input. For example, accurately reaching for a coffee cup, or shaking a friend's hand. In a new study, scientists trained blindfolded sighted participants to perform fast and accurate movements using a new SSD, called EyeMusic. Their results are published in the July issue of Restorative Neurology and Neuroscience.

The EyeMusic, developed by a team of researchers at the Hebrew University of Jerusalem, employs pleasant musical tones and scales to help the visually impaired "see" using music. This non-invasive SSD converts images into a combination of musical notes, or "soundscapes."

The device was developed by the senior author Prof. Amir Amedi and his team at the Edmond and Lily Safra Center for Brain Sciences (ELSC) and the Institute for Medical Research Israel-Canada at the Hebrew University. The EyeMusic scans an image and represents pixels at high vertical locations as high-pitched musical notes and low vertical locations as low-pitched notes according to a musical scale that will sound pleasant in many possible combinations. The image is scanned continuously, from left to right, and an auditory cue is used to mark the start of the scan. The horizontal location of a pixel is indicated by the timing of the musical notes relative to the cue (the later it is sounded after the cue, the farther it is to the right), and the brightness is encoded by the loudness of the sound.

The EyeMusic's algorithm uses different musical instruments for each of the five colors: white (vocals), blue (trumpet), red (reggae organ), green (synthesized reed), yellow (violin); Black is represented by silence. Prof. Amedi mentions that "The notes played span five octaves and were carefully chosen by musicians to create a pleasant experience for the users." Sample sound recordings are available at http://brain.huji.ac.il/em/.

"We demonstrated in this study that the EyeMusic, which employs pleasant musical scales to convey visual information, can be used after a short training period (in some cases, less than half an hour) to guide movements, similar to movements guided visually," explain lead investigators Drs. Shelly Levy-Tzedek, an ELSC researcher at the Faculty of Medicine, Hebrew University, Jerusalem, and Prof. Amir Amedi. "The level of accuracy reached in our study indicates that performing daily tasks with an SSD is feasible, and indicates a potential for rehabilitative use."

The study tested the ability of 18 blindfolded sighted individuals to perform movements guided by the EyeMusic, and compared those movements to those performed with visual guidance. At first, the blindfolded participants underwent a short familiarization session, where they learned to identify the location of a single object (a white square) or of two adjacent objects (a white and a blue square).

In the test sessions, participants used a stylus on a digitizing tablet to point to a white square located either in the north, the south, the east or the west. In one block of trials they were blindfolded (SSD block), and in the other block (VIS block) the arm was placed under an opaque cover, so they could see the screen but did not have direct visual feedback from the hand. The endpoint location of their hand was marked by a blue square. In the SSD block, they received feedback via the EyeMusic. In the VIS block, the feedback was visual.

"Participants were able to use auditory information to create a relatively precise spatial representation," notes Dr. Levy-Tzedek.

The study lends support to the hypothesis that representation of space in the brain may not be dependent on the modality with which the spatial information is received, and that very little training is required to create a representation of space without vision, using sounds to guide fast and accurate movements. "SSDs may have great potential to provide detailed spatial information for the visually impaired, allowing them to interact with their external environment and successfully make movements based on this information, but further research is now required to evaluate the use of our device in the blind " concludes Dr. Levy-Tzedek. These results demonstrate the potential application of the EyeMusic in performing everyday tasks -- from accurately reaching for the red (but not the green!) apples in the produce aisle, to, perhaps one day, playing a Kinect / Xbox game.


Story Source:

The above story is based on materials provided by IOS Press BV. Note: Materials may be edited for content and length.


Journal Reference:

  1. S. Levy-Tzedek, S. Hanassy, S. Abboud, S. Maidenbaum, A. Amedi. Fast, Accurate Reaching Movements with a Visual-to-Auditory Sensory Substitution Device. Restorative Neurology and Neuroscience, July 2012 DOI: 10.3233/RNN-2012-110219

Cite This Page:

IOS Press BV. "Music to my eyes: Device converting images into music helps visually impaired find things with ease." ScienceDaily. ScienceDaily, 5 July 2012. <www.sciencedaily.com/releases/2012/07/120705161229.htm>.
IOS Press BV. (2012, July 5). Music to my eyes: Device converting images into music helps visually impaired find things with ease. ScienceDaily. Retrieved April 18, 2014 from www.sciencedaily.com/releases/2012/07/120705161229.htm
IOS Press BV. "Music to my eyes: Device converting images into music helps visually impaired find things with ease." ScienceDaily. www.sciencedaily.com/releases/2012/07/120705161229.htm (accessed April 18, 2014).

Share This



More Matter & Energy News

Friday, April 18, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Small Reactors Could Be Future of Nuclear Energy

Small Reactors Could Be Future of Nuclear Energy

AP (Apr. 17, 2014) After the Fukushima nuclear disaster, the industry fell under intense scrutiny. Now, small underground nuclear power plants are being considered as the possible future of the nuclear energy. (April 17) Video provided by AP
Powered by NewsLook.com
Honda's New ASIMO Robot, More Human-Like Than Ever

Honda's New ASIMO Robot, More Human-Like Than Ever

AFP (Apr. 17, 2014) It walks and runs, even up and down stairs. It can open a bottle and serve a drink, and politely tries to shake hands with a stranger. Meet the latest ASIMO, Honda's humanoid robot. Duration: 00:54 Video provided by AFP
Powered by NewsLook.com
German Researchers Crack Samsung's Fingerprint Scanner

German Researchers Crack Samsung's Fingerprint Scanner

Newsy (Apr. 16, 2014) German researchers have used a fake fingerprint made from glue to bypass the fingerprint security system on Samsung's new Galaxy S5 smartphone. Video provided by Newsy
Powered by NewsLook.com
Porsche CEO Says Supercar Is Not Dead: Cue the Spyder 918

Porsche CEO Says Supercar Is Not Dead: Cue the Spyder 918

TheStreet (Apr. 16, 2014) The Porsche Spyder 918 proves that, in an automotive world obsessed with fuel efficiency, the supercar is not dead. Porsche North America CEO Detlev von Platen attributes the brand's consistent sales growth -- 21% in 2013 -- with an investment in new technology and expanded performance dynamics. The hybrid Spyder 918 has 887 horsepower and 944 lb-ft of torque, but it can run 18 miles on just an electric charge. The $845,000 vehicle is not a consumer-targeted vehicle but a brand statement. Video provided by TheStreet
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins