Featured Research

from universities, journals, and other organizations

Tool detects patterns hidden in vast data sets

Date:
December 16, 2011
Source:
Broad Institute of MIT and Harvard
Summary:
Researchers have developed a tool that can tackle large data sets in a way that no other software program can. Part of a suite of statistical tools called MINE, it can tease out multiple patterns hidden in health information, statistics amassed from a season of major league baseball, data on the changing bacterial landscape of the gut, and more.

If researchers printed on paper each potential relationship in a recent data set containing abundance levels of bacteria in the human gut, the stack of paper would reach to a height of 1.4 miles, six times the height of the Empire State Building.
Credit: Image courtesy of Sigrid Knemeyer, Broad Communications

Researchers from the Broad Institute and Harvard University have developed a tool that can tackle large data sets in a way that no other software program can. Part of a suite of statistical tools called MINE, it can tease out multiple patterns hidden in health information from around the globe, statistics amassed from a season of major league baseball, data on the changing bacterial landscape of the gut, and much more.

The researchers report their findings in a paper appearing in the December 16 issue of the journal Science.

From Facebook to physics to the global economy, the world is filled with data sets that could take a person hundreds of years to analyze by eye. Sophisticated computer programs can search these data sets with great speed, but fall short when researchers attempt to even-handedly detect different kinds of patterns in large data collections.

"There are massive data sets that we want to explore, and within them, there may be many relationships that we want to understand," said Broad Institute associate member Pardis Sabeti, senior author of the paper and an assistant professor at the Center for Systems Biology at Harvard University. "The human eye is the best way to find these relationships, but these data sets are so vast that we can't do that. This toolkit gives us a way of mining the data to look for relationships."

The researchers tested their analytical toolkit on several large data sets, including one provided by Harvard colleague Peter Turnbaugh who is interested in the trillions of microorganisms that live in the gut. Working with Turnbaugh, the research team harnessed MINE to make more than 22 million comparisons and narrowed in on a few hundred patterns of interest that had not been observed before.

"The goal of this statistic is to take data with a lot of different dimensions and many possible correlations and pick out the top ones," said Michael Mitzenmacher, a senior author of the paper and professor of computer science at Harvard University. "We view this as an exploration tool -- it can find patterns and rank them in an equitable way."

One of the tool's greatest strengths is that it can detect a wide range of patterns and characterize them according to a number of different parameters a researcher might be interested in. Other statistical tools work well for searching for a specific pattern in a large data set, but cannot score and compare different kinds of possible relationships. MINE, which stands for Maximal Information-based Nonparametric Exploration, is able to analyze a broad spectrum of patterns.

"Standard methods will see one pattern as signal and others as noise," said David Reshef, a co-first author of the paper who is currently a graduate student in the Harvard-MIT Health Sciences and Technology program and also worked on this project as a graduate student in the department of statistics at the University of Oxford. "There can potentially be a variety of different types of relationships in a given data set. What's exciting about our method is that it looks for any type of clear structure within the data, attempting to find all of them."

Not only does MINE attempt to identify any pattern within the data, but it also attempts to do so with an eye toward capturing different types of patterns equally well. "This ability to search for patterns in an equitable way offers tremendous exploratory potential in terms of searching for patterns without having to know ahead of time what to search for," said David Reshef.

MINE is especially powerful in exploring data sets with relationships that may harbor more than one important pattern. As a proof of concept, the researchers applied MINE to social, economic, health, and political data from the World Health Organization (WHO) and its partners. When they compared the relationship between household income and female obesity, they found two contrasting trends in the data. Many countries follow a parabolic rate, with obesity rates rising with income but peaking and tapering off after income reaches a certain level. But in the Pacific Islands, where female obesity is a sign of status, countries follow a steep trend, with the rate of obesity climbing as income increases.

"Many data sets will contain these types of complicated relationships that are guided by multiple drivers," said Sabeti. MINE is able to identify these. "This greatly extends our capability to find interesting relationships in data."

Researchers can use MINE to generate new ideas and connections that no one has thought to look for before.

"Our tool is a hypothesis generator," said Yakir Reshef, a co-first author of the paper and a Fulbright scholar at the Weizmann Institute of Science. "The standard paradigm is hypothesis-driven science, where you come up with a hypothesis based on your personal observations. But by exploring the data, you get ideas for hypotheses that would never have occurred to you otherwise."

In addition to testing the ability of the suite of tools to detect patterns in biological and health data, the researchers examined data collected from the 2008 baseball season.

"One question that we thought would be particularly interesting would be to see what things were most strongly associated with salary," said David Reshef. The researchers generated a list of relationships, finding that the strongest associations with salary were hits, total bases, and an aggregate statistic that reflects how many runs a player generated for a team. "Given the stakes, baseball is so well documented. We're curious to see what can be done in this realm with tools like MINE."

Researchers from many different fields, including systems biology, computer science, statistics, and mathematics, all contributed to this project. "People are getting better at combining data from different sources, and in some ways, this project is in the spirit of that," said Yakir Reshef. "The project brought together authors from many disciplines. It symbolizes the kind of collaborations that we hope people will use this for in the future."

Other authors who contributed to this work include Hilary Finucane, Sharon Grossman, Gilean McVean, and Eric Lander. Funding for this work was provided by the Packard Foundation, Marshall Aid Commemoration Commission, National Science Foundation, European Research Council, and the National Institutes of Health.


Story Source:

The above story is based on materials provided by Broad Institute of MIT and Harvard. Note: Materials may be edited for content and length.


Journal Reference:

  1. Reshef, DN et al. Detecting novel associations in large data sets. Science, Dec 15, 2011 DOI: 10.1126/Science1205438

Cite This Page:

Broad Institute of MIT and Harvard. "Tool detects patterns hidden in vast data sets." ScienceDaily. ScienceDaily, 16 December 2011. <www.sciencedaily.com/releases/2011/12/111215141611.htm>.
Broad Institute of MIT and Harvard. (2011, December 16). Tool detects patterns hidden in vast data sets. ScienceDaily. Retrieved July 29, 2014 from www.sciencedaily.com/releases/2011/12/111215141611.htm
Broad Institute of MIT and Harvard. "Tool detects patterns hidden in vast data sets." ScienceDaily. www.sciencedaily.com/releases/2011/12/111215141611.htm (accessed July 29, 2014).

Share This




More Computers & Math News

Tuesday, July 29, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

It's Not Just Facebook: OKCupid Experiments With Users Too

It's Not Just Facebook: OKCupid Experiments With Users Too

Newsy (July 29, 2014) If you've been looking for love online, there's a chance somebody has been looking at how you're looking. Video provided by Newsy
Powered by NewsLook.com
Why Facebook Wants You To Download Its Messenger App

Why Facebook Wants You To Download Its Messenger App

Newsy (July 29, 2014) Facebook will start requiring users to download a separate Messenger application if they wish to continue using Facebook for mobile messaging. Video provided by Newsy
Powered by NewsLook.com
Teen's Phone Ignites Under Her Pillow; How Real Is The Risk?

Teen's Phone Ignites Under Her Pillow; How Real Is The Risk?

Newsy (July 28, 2014) A Texas teen's Samsung phone apparently ignited while she slept, but what was the real problem here? Video provided by Newsy
Powered by NewsLook.com
Zillow Snaps Up Web Real Estate With Trulia Deal

Zillow Snaps Up Web Real Estate With Trulia Deal

Reuters - US Online Video (July 28, 2014) Zillow's decision to buy rival Trulia is just one step in a continuing string of acquisitions, and Zillow CEO Spencer Rascoff is already thinking about his next big deal. Bobbi Rebell reports. Video provided by Reuters
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins