Featured Research

from universities, journals, and other organizations

New Computer Program Enables Powerful Data Analysis On Small Computers

Date:
January 10, 2009
Source:
University of California - Davis
Summary:
A powerful new tool that can extract features and patterns from enormously large and complex data sets has been developed. The tool -- a set of problem-solving calculations known as an algorithm -- is compact enough to run on computers with as little as two gigabytes of memory.

Meteorite Hits Aerogel. After using the algorithm to determine the filament structure of an aerogel -- a lightweight foam used in shielding electronic equipment in satellites -- the researchers were able to compute changes to its structural integrity by the simulated impact of a micrometeorite traveling at 10,000 miles per hour (red sphere on left).
Credit: Attila Gyulassy/UC Davis Copyright UC Regents

A powerful yet compact algorithm has been developed that can be used on laptop computers to extract features and patterns from huge and complex data sets.

Related Articles


A powerful computing tool that allows scientists to extract features and patterns from enormously large and complex sets of raw data has been developed by scientists at University of California, Davis, and Lawrence Livermore National Laboratory. The tool – a set of problem-solving calculations known as an algorithm – is compact enough to run on computers with as little as two gigabytes of memory.

The team that developed this algorithm has already used it to probe a slew of phenomena represented by billions of data points, including analyzing and creating images of flame surfaces; searching for clusters and voids in a virtual universe experiment; and identifying and tracking pockets of fluid in a simulated mixing of two fluids.

"What we've developed is a workable system of handling any data in any dimension," said Attila Gyulassy, who led the five-year development effort while pursuing a PhD in computer science at UC Davis. "We expect this algorithm will become an integral part of a scientist's toolbox to answer questions about data."

A paper describing the new algorithm was published in the November-December issue of IEEE Transactions on Visualization and Computer Graphics.

Computers are widely used to perform simulations of real-world phenomena and to capture results of physical experiments and observations, storing this information as collections of numbers. But as the size of these data sets has burgeoned, hand-in-hand with computer capacity, analysis has grown increasingly difficult.

A mathematical tool to extract and visualize useful features from data sets has existed for nearly 40 years – in theory. Called the Morse-Smale complex, it partitions sets by similarity of features and encodes them into mathematical terms. But working with the Morse-Smale complex is not easy. "It's a powerful language. But a cost of that, is that using it meaningfully for practical applications is very difficult," Gyulassy said.

Gyulassy's algorithm divides data sets into parcels of cells, then analyzes each parcel separately using the Morse-Smale complex. Results of those computations are then merged together. As new parcels are created from merged parcels, they are analyzed and merged yet again. At each step, data that do not need to be stored in memory are discarded, drastically reducing the computing power required to run the calculations.

One of Gyulassy's tests of the algorithm was to use it to analyze and track the formation and movement of pockets of fluid in the simulated mixing of two fluids: one dense, one light. The complexity of this data set is so vast – it consists of more than one billion data points on a three-dimensional grid – it challenges even supercomputers, Gyulassy said. Yet the new algorithm with its streamlining features was able to perform the analysis on a laptop computer with just two gigabytes of memory. Although Gyulassy had to wait nearly 24 hours for the little machine to complete its calculations, at the end of this process he could pull up images in mere seconds to illustrate phenomena he was interested in, such as the branching of fluid pockets in the mixture.

Two main factors are driving the need for analysis of large data sets, said co-author Bernd Hamann: a surge in the use of powerful computers that can produce huge amounts of data, and an upswing in affordability and availability of sensing devices that researchers deploy in the field and lab to collect a profusion of data.

"Our data files are becoming larger and larger, while the scientist has less and less time to understand them," said Hamann, a professor of computer science and associate vice chancellor for research at UC Davis. "But what are the data good for if we don't have the means of applying mathematically sound and computationally efficient computer analysis tools to look for what is captured in them?"

Gyulassy is currently developing software that will allow others to put the algorithm to use. He expects the learning curve to be steep for this open-source product, "but if you just learn the minimal amount about what a Morse-Smale complex is," he said, "it will be pretty intuitive."

Other authors of the paper are Valerio Pascucci, who was an adjunct professor of computer science at UC Davis and a computer scientist and project leader at Lawrence Livermore National Laboratory when he did the work (he is now at University of Utah); and Peer-Timo Bremer, a computer scientist at Lawrence Livermore National Laboratory.

The research was supported in part by the National Science Foundation. Gyulassy also received support from the Lawrence Scholar Program.


Story Source:

The above story is based on materials provided by University of California - Davis. Note: Materials may be edited for content and length.


Cite This Page:

University of California - Davis. "New Computer Program Enables Powerful Data Analysis On Small Computers." ScienceDaily. ScienceDaily, 10 January 2009. <www.sciencedaily.com/releases/2009/01/090108082531.htm>.
University of California - Davis. (2009, January 10). New Computer Program Enables Powerful Data Analysis On Small Computers. ScienceDaily. Retrieved December 18, 2014 from www.sciencedaily.com/releases/2009/01/090108082531.htm
University of California - Davis. "New Computer Program Enables Powerful Data Analysis On Small Computers." ScienceDaily. www.sciencedaily.com/releases/2009/01/090108082531.htm (accessed December 18, 2014).

Share This


More From ScienceDaily



More Computers & Math News

Thursday, December 18, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Navy Unveils Robot Fish

Navy Unveils Robot Fish

Reuters - Light News Video Online (Dec. 18, 2014) The U.S. Navy unveils an underwater device that mimics the movement of a fish. Tara Cleary reports. Video provided by Reuters
Powered by NewsLook.com
How 2014 Shaped The Future Of The Internet

How 2014 Shaped The Future Of The Internet

Newsy (Dec. 18, 2014) It has been a long, busy year for Net Neutrality. The stage is set for an expected landmark FCC decision sometime in 2015. Video provided by Newsy
Powered by NewsLook.com
Amazon Offering One-Hour Delivery Through Prime Now

Amazon Offering One-Hour Delivery Through Prime Now

Newsy (Dec. 18, 2014) Amazon is now offering one-hour delivery to Amazon Prime members in Manhattan and hopes to expand to other cities soon. Video provided by Newsy
Powered by NewsLook.com
Jaguar Unveils 360 Virtual Windshield Making Car Pillars Appear Transparent

Jaguar Unveils 360 Virtual Windshield Making Car Pillars Appear Transparent

Buzz60 (Dec. 17, 2014) Jaguar unveils a virtual 360 degree windshield that may be the most futuristic automotive development yet. Jen Markham explains. Video provided by Buzz60
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:

Strange & Offbeat Stories


Space & Time

Matter & Energy

Computers & Math

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins