Featured Research

from universities, journals, and other organizations

Computer program can identify rough sketches

Date:
September 13, 2012
Source:
Brown University
Summary:
Computer scientists have developed a new program that can recognize rough sketches in real time, something that up to now had been very difficult for computers to do. To make the program work, the researchers used 20,000 real sketches to teach the program how humans actually sketch objects.

Ears and whiskers are a hint: Computers can recognize photos and careful drawings of rabbits, but quick sketches by non-artists ... not so much. Researchers in Providence and Berlin say they have produced the first computer application that enables “semantic understanding” of abstract sketches.
Credit: Mathias Eitz, James Hays and Marc Alexa

First they took over chess. Then Jeopardy. Soon, computers could make the ideal partner in a game of Draw Something (or its forebear, Pictionary).

Researchers from Brown University and the Technical University of Berlin have developed a computer program that can recognize sketches as they're drawn in real time. It's the first computer application that enables "semantic understanding" of abstract sketches, the researchers say. The advance could clear the way for vastly improved sketch-based interface and search applications.

The research behind the program was presented last month at SIGGRAPH, the world's premier computer graphics conference. The paper is now available online (http://cybertron.cg.tu-berlin.de/eitz/projects/classifysketch/), together with a video, a library of sample sketches, and other materials.

Computers are already pretty good at matching sketches to objects as long as the sketches are accurate representations. For example, applications have been developed that can match police sketches to actual faces in mug shots. But iconic or abstract sketches -- the kind that most people are able to easily produce -- are another matter entirely.

For example, if you were asked to sketch a rabbit, you might draw a cartoony-looking thing with big ears, buckteeth, and a cotton tail. Another person probably wouldn't have much trouble recognizing your funny bunny as a rabbit -- despite the fact that it doesn't look all that much like a real rabbit.

"It might be that we only recognize it as a rabbit because we all grew up that way," said James Hays, assistant professor of computer science at Brown, who developed the new program with Mathias Eitz and Marc Alexa from the Technical University in Berlin. "Whoever got the ball rolling on caricaturing rabbits like that, that's just how we all draw them now."

Getting a computer to understand what we've come to understand through years of cartoons and coloring books is a monumentally difficult task. The key to making this new program work, Hays says, is a large database of sketches that could be used to teach a computer how humans sketch objects. "This is really the first time anybody has examined a large database of actual sketches," Hays said.

To put the database together, the researchers first came up with a list of everyday objects that people might be inclined to sketch. "We looked at an existing computer vision dataset called LabelMe, which has a lot of annotated photographs," Hays said. "We looked at the label frequency and we got the most popular objects in photographs. Then we added other things of interest that we thought might occur in sketches, like rainbows for example."

They ended up with a set of 250 object categories. Then the researchers used Mechanical Turk, a crowdsourcing marketplace run by Amazon, to hire people to sketch objects from each category -- 20,000 sketches in all. Those data were then fed into existing recognition and machine learning algorithms to teach the program which sketches belong to which categories. From there, the team developed an interface where users input new sketches, and the computer tries to identify them in real time, as quickly as the user draws them.

As it is now, the program successfully identifies sketches with around 56-percent accuracy, as long as the object is included in one of the 250 categories. That's not bad, considering that when the researchers asked actual humans to identify sketches in the database, they managed about 73-percent accuracy. "The gap between human and computational performance is not so big, not as big certainly as it is in other computer vision problems," Hays said.

The program isn't ready to rule Pictionary just yet, mainly because of its limited 250-category vocabulary. But expanding it to include more categories is a possibility, Hays says. One way to do that might be to turn the program into a game and collect the data that players input. The team has already made a free iPhone/iPad app that could be gamified.

"The game could ask you to sketch something and if another person is able to successfully recognize it, then we can say that must have been a decent enough sketch," he said. "You could collect all sorts of training data that way."

And that kind of crowdsourced data has been key to the project so far.

"It was the data gathering that had been holding this back, not the digital representation or the machine learning; those have been around for a decade," Hays said. "There's just no way to learn to recognize say, sketches of lions, based on just a clever algorithm. The algorithm really needs to see close to 100 instances of how people draw lions, and then it becomes possible to tell lions from potted plants."

Ultimately a program like this one could end up being much more than just fun and games. It could be used to develop better sketch-based interface and search applications. Despite the ubiquity of touch screens, sketch-based search still isn't widely used, but that's probably because it simply hasn't worked very well, Hays says.

A better sketch-based interface might improve computer accessibility. "Directly searching for some visual shape is probably easier in some domains," Hays said. "It avoids all language issues; that's certainly one thing."


Story Source:

The above story is based on materials provided by Brown University. Note: Materials may be edited for content and length.


Cite This Page:

Brown University. "Computer program can identify rough sketches." ScienceDaily. ScienceDaily, 13 September 2012. <www.sciencedaily.com/releases/2012/09/120913105010.htm>.
Brown University. (2012, September 13). Computer program can identify rough sketches. ScienceDaily. Retrieved April 17, 2014 from www.sciencedaily.com/releases/2012/09/120913105010.htm
Brown University. "Computer program can identify rough sketches." ScienceDaily. www.sciencedaily.com/releases/2012/09/120913105010.htm (accessed April 17, 2014).

Share This



More Computers & Math News

Thursday, April 17, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Heartbleed Hack Leads To Arrest

Heartbleed Hack Leads To Arrest

Newsy (Apr. 17, 2014) A 19-year-old computer science student has been arrested in relation to a data breach of 900 social insurance numbers from Canada's revenue agency. Video provided by Newsy
Powered by NewsLook.com
German Researchers Crack Samsung's Fingerprint Scanner

German Researchers Crack Samsung's Fingerprint Scanner

Newsy (Apr. 16, 2014) German researchers have used a fake fingerprint made from glue to bypass the fingerprint security system on Samsung's new Galaxy S5 smartphone. Video provided by Newsy
Powered by NewsLook.com
Twitter, Apple Social Data Purchases Likely to Spur More Mergers and Acquisitions

Twitter, Apple Social Data Purchases Likely to Spur More Mergers and Acquisitions

TheStreet (Apr. 16, 2014) The social media data space is likely to see more mergers and acquisitions following Twitter Inc.'s acquisition of tweet analyzer Gnip Inc. on Tuesday and Apples Inc.'s purchase of Topsy Labs Inc. back in December. One firm in particular, the U.K.'s DataSift Inc., could be on the list of potential buyers. Among other social media startups that could be ripe for picking is Banjo, whose mobile app provides aggregated content by topic and location. Banjo could also be a good fit for Twitter. Video provided by TheStreet
Powered by NewsLook.com
Bitcoin Exchange Mt. Gox to Liquidate After Rebuilding Rejected

Bitcoin Exchange Mt. Gox to Liquidate After Rebuilding Rejected

TheStreet (Apr. 16, 2014) Bitcoin exchange Mt. Gox has agreed to liquidate after a Japanese court rejected its plans to rebuild, according to a report by the Wall Street Journal. Mt. Gox filed for bankruptcy protection in February after announcing about 850,000 bitcoins, worth around $454 million at today's rates, may have been stolen by hackers. It has since recovered 200,000 of the missing bitcoins. The court put Mt. Gox's assets under a provisional administrator's control until bankruptcy proceedings begin. Video provided by TheStreet
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins