Featured Research

from universities, journals, and other organizations

Computerized Map Responds To Speech And Gestures

Date:
August 20, 1999
Source:
Penn State
Summary:
Penn State researchers have developed a prototype system to help visitors locate campus parking lots and buildings by talking with a computer-controlled map that responds not only to the spoken word but also to natural hand gestures.

University Park, Pa. --- Penn State researchers have developed a prototype system to help visitors locate campus parking lots and buildings by talking with a computer-controlled map that responds not only to the spoken word but also to natural hand gestures.

Project leader Dr. Rajeev Sharma, assistant professor of computer science and engineering, says, "There still is a lot of work to be done but we have a pretty fair, ground- level, demonstration model of a system in which a person can interact with a computer by using the most natural human mode of communication - talking while gesturing.

"Besides the current application, the system could potentially be adapted to help tourists locate the sights in large cities, shoppers to find stores in malls, visitors to find patients in hospitals or even for roles in crisis management, mission planning and briefing," he adds.

In a recent demonstration, Sanshzar Kettebekov, a doctoral student, stood about 5 feet away from a map of the Penn State campus projected on an 4-foot-by-3- foot screen. "Scroll," he said gently into the cordless microphone attached to his T-shirt and the map moved. "Stop," Kettebekov directed and the map did. He waved his hand in the air and a little red hand appeared on the screen. As Kettebekov continued to gesture with his hand, the on-screen hand followed it, like a cursor obeying a mouse. When the red hand settled on one of the buildings, Kettebekov said, "Show me the nearest parking lot," and a bright blue line immediately appeared and connected the building to the closest lot.

The system is based on off-the-shelf-equipment. The computer is a standard PC workstation equipped with a video camera, the system's "eye" on the gesturing human. A commercially available speech recognition package currently takes care of the conversation. However, the Penn State researchers developed new gesture recognition software and used footage of TV weather broadcasters narrating the weather to "train" it.

The new Penn State gesture recognition software is based on a technique called Hidden Markov Models (HMM), a time-varying pattern recognition method. HMMs had been used previously in gesture recognition systems. However, only predefined gestures, such as sign language, had been used. The new Penn State approach, based on weathercaster movements, enables the computer to recognize and "understand" a rich store of natural gestures that occur in combination with speech.

At this point, although the system recognizes quite a few human gestures and spoken words, it doesn't like small talk. You can't tell it, "Well, I'd like to go to the Creamery for an ice cream cone first and then stop off at Old Main before parking at Beaver Stadium." At least not yet.

Yuhui Zhou, a master's degree candidate, has a background in linguistics. She is working on dialog design and feedback systems that will enable the computer to extract the most salient information from a human conversation stream. Jiongyu Cai, doctoral candidate, is working on extracting the salient gestures from the random hand waving that most people use while talking. Kettebekov is trying to understand the combination of speech and gestures so that he can develop software that enables the computer to interpret gestures in the speech context.

The research team is also paying attention to the fact that people from different cultures gesture differently but, at present, plans call for the map to respond only to English.

Sharma says, "Computer users have been slaves to the mouse and the keyboard too long. The equipment has, so far, limited the potential for human interaction with computers. Incorporating gesture, which computer vision makes possible, allows us to imagine all kinds of potential applications. For example, I can imagine a computer you wear on your head, like a virtual reality helmet, that could help you repair your PC by telling you what to do and then "watching" as you do it. Or, a wearable computerized surgical aide that could help direct a surgeon to the precise location of a tumor.

"For now, our group will be working on trying to enable the computer to more effectively talk back to the user. We'd like to model the human/computer dialog so that the display could interactively influence the user input enabling the computer to play a more active role in the natural speech/gesture interface," he adds.

The research group has detailed the new system in a paper, Toward Interpretation of Natural Speech/Gesture: Spatial Planning on a Virtual Map published in the Proceedings of the Army Research Laboratory Annual Symposium on Advanced Display, held in February 1999 in Adelphi, Md. The work on gesture recognition is detailed in Indrajit Poddar 's master's thesis, completed in May, entitled "Continuous Recognition of Dieictic Gestures for Multimodal Interfaces."

The research was supported, in part, by grants from the National Science Foundation and the Army Research Laboratory.

Note: Dr. Sharma's website is at http://www.cse.psu.edu/-rsharma


Story Source:

The above story is based on materials provided by Penn State. Note: Materials may be edited for content and length.


Cite This Page:

Penn State. "Computerized Map Responds To Speech And Gestures." ScienceDaily. ScienceDaily, 20 August 1999. <www.sciencedaily.com/releases/1999/08/990820022208.htm>.
Penn State. (1999, August 20). Computerized Map Responds To Speech And Gestures. ScienceDaily. Retrieved July 25, 2014 from www.sciencedaily.com/releases/1999/08/990820022208.htm
Penn State. "Computerized Map Responds To Speech And Gestures." ScienceDaily. www.sciencedaily.com/releases/1999/08/990820022208.htm (accessed July 25, 2014).

Share This




More Computers & Math News

Friday, July 25, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Mobile App Gives Tour of Battle of Atlanta Sites

Mobile App Gives Tour of Battle of Atlanta Sites

AP (July 25, 2014) Emory University's Center for Digital Scholarship has launched a self-guided mobile tour app to coincide with the 150th anniversary of the Civil War's Battle of Atlanta. (July 25) Video provided by AP
Powered by NewsLook.com
Bill Gates: Health, Agriculture Key to Africa's Development

Bill Gates: Health, Agriculture Key to Africa's Development

AFP (July 24, 2014) Health and agriculture development are key if African countries are to overcome poverty and grow, US software billionaire Bill Gates said Thursday, as he received an honourary degree in Ethiopia. Duration: 00:36 Video provided by AFP
Powered by NewsLook.com
Creative Makeovers for Ugly Cellphone Towers

Creative Makeovers for Ugly Cellphone Towers

AP (July 24, 2014) Mobile phone companies and communities across the country are going to new lengths to disguise those unsightly cellphone towers. From a church bell tower to a flagpole, even a pencil, some towers are trying to make a point. (July 24) Video provided by AP
Powered by NewsLook.com
Robot Parking Valet Creates Stress-Free Travel

Robot Parking Valet Creates Stress-Free Travel

AP (July 23, 2014) 'Ray' the robotic parking valet at Dusseldorf Airport in Germany lets travelers to avoid the hassle of finding a parking spot before heading to the check-in desk. (July 23) Video provided by AP
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

    Health News

      Environment News

        Technology News



          Save/Print:
          Share:

          Free Subscriptions


          Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

          Get Social & Mobile


          Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

          Have Feedback?


          Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
          Mobile: iPhone Android Web
          Follow: Facebook Twitter Google+
          Subscribe: RSS Feeds Email Newsletters
          Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins