Featured Research

from universities, journals, and other organizations

Statistical Analysis Of Complex Data Sets With Robust Statistical Methods

Date:
April 12, 2007
Source:
European Science Foundation
Summary:
Robust statistical analysis methods capable of dealing with large complex data sets are required more than ever before in almost all branches of science. The European Science Foundation's three-year SACD network developed new methods for extracting key structural features within the data. Such features can include outlying values that may be particularly significant within the increasingly large and complex data sets generated in financial markets, medical diagnostics, environmental surveys, and other sources.

Robust statistical analysis methods capable of dealing with large complex data sets are required more than ever before in almost all branches of science. The European Science Foundation’s three-year SACD network, which was completed in December 2006, developed new methods for extracting key structural features within the data. Such features can include outlying values that may be particularly significant within the increasingly large and complex data sets generated in financial markets, medical diagnostics, environmental surveys, and other sources.

“Outliers often indicate the most interesting data points, like polluted areas for environmental data, or irregularities in online monitoring of patients,” said SACD chair Christophe Croux. On this front the programme has almost completely achieved its objectives, according to Croux. “A lot of work has been done in developing new methods, especially for analyzing large data sets, that can cope with outlying atypical values. This resulted in a number of publications related to the subject of the network”.

Particular progress has been made detecting outliers in multivariate time series, Croux added. This is a significant development for a number of analysis and monitoring applications involving measurements of different but related quantities that vary over time. Among many such applications are: monitoring of telecommunication networks to assess how performance and reliability are affected by events such as upgrades, surges in demand, and local link failures; monitoring noise in the vicinity of an airport; modeling the behaviour of financial markets in response to geopolitical events; and tracking the condition of patients in intensive care via several measurements such as pulse rate, blood pressure, lung water etc.

Without robust analysis methods it is easy to miss significant outlyers in such multivariate data. In some cases the outlyers only show up clearly when considering all the variables together, and yet may indicate something significant that could easily be missed, such as a sudden deterioration in a critical patient’s condition.

SACD has also advanced the field of chemometrics, which is the application of multivariate analysis methods to data of chemical interest, with some of the developments now implemented in software written by members of the network. The same principles have been applied to analysis of risks of stock investments, and measuring volatility of financial markets.

In some cases it is desirable to eliminate outlyers from data sets in order to identify the most likely response of a particular variable to different events. Within SACD, a method was developed to do this for analysis of the relationship between various economic parameters and the yield of stocks. For this it is necessary to concentrate on the bulk of the data rather than the exceptions or outlyers. “In order to do so we have to identify these extreme observations in order to downweight or reject them from the computations,” said Croux. When there are multiple variables this is more difficult, and one of the major achievements of SACD has been to find new ways of condensing and summarizing the data in such a way that the main structure of the data can be retrieved, making it also easier to detect the outliers.

Croux admits there is more work to be done, particularly in dealing with highly complex data sets, and with problems involving many variables and small sample sizes. “Important steps to be taken include robust methods that can deal with categorical data and missing values.”


Story Source:

The above story is based on materials provided by European Science Foundation. Note: Materials may be edited for content and length.


Cite This Page:

European Science Foundation. "Statistical Analysis Of Complex Data Sets With Robust Statistical Methods." ScienceDaily. ScienceDaily, 12 April 2007. <www.sciencedaily.com/releases/2007/04/070411110003.htm>.
European Science Foundation. (2007, April 12). Statistical Analysis Of Complex Data Sets With Robust Statistical Methods. ScienceDaily. Retrieved April 17, 2014 from www.sciencedaily.com/releases/2007/04/070411110003.htm
European Science Foundation. "Statistical Analysis Of Complex Data Sets With Robust Statistical Methods." ScienceDaily. www.sciencedaily.com/releases/2007/04/070411110003.htm (accessed April 17, 2014).

Share This



More Computers & Math News

Thursday, April 17, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

German Researchers Crack Samsung's Fingerprint Scanner

German Researchers Crack Samsung's Fingerprint Scanner

Newsy (Apr. 16, 2014) German researchers have used a fake fingerprint made from glue to bypass the fingerprint security system on Samsung's new Galaxy S5 smartphone. Video provided by Newsy
Powered by NewsLook.com
Twitter, Apple Social Data Purchases Likely to Spur More Mergers and Acquisitions

Twitter, Apple Social Data Purchases Likely to Spur More Mergers and Acquisitions

TheStreet (Apr. 16, 2014) The social media data space is likely to see more mergers and acquisitions following Twitter Inc.'s acquisition of tweet analyzer Gnip Inc. on Tuesday and Apples Inc.'s purchase of Topsy Labs Inc. back in December. One firm in particular, the U.K.'s DataSift Inc., could be on the list of potential buyers. Among other social media startups that could be ripe for picking is Banjo, whose mobile app provides aggregated content by topic and location. Banjo could also be a good fit for Twitter. Video provided by TheStreet
Powered by NewsLook.com
Bitcoin Exchange Mt. Gox to Liquidate After Rebuilding Rejected

Bitcoin Exchange Mt. Gox to Liquidate After Rebuilding Rejected

TheStreet (Apr. 16, 2014) Bitcoin exchange Mt. Gox has agreed to liquidate after a Japanese court rejected its plans to rebuild, according to a report by the Wall Street Journal. Mt. Gox filed for bankruptcy protection in February after announcing about 850,000 bitcoins, worth around $454 million at today's rates, may have been stolen by hackers. It has since recovered 200,000 of the missing bitcoins. The court put Mt. Gox's assets under a provisional administrator's control until bankruptcy proceedings begin. Video provided by TheStreet
Powered by NewsLook.com
BlackBerry: The Crash That Launched 1,000 Startups

BlackBerry: The Crash That Launched 1,000 Startups

Reuters - Business Video Online (Apr. 16, 2014) Tech startups in BlackBerry's hometown of Waterloo, Ontario, are tapping talent from the struggling smartphone company and filling the void left in the region by its meltdown. Reuters correspondent Euan Rocha visits the region that could become Canada's Silicon Valley. Video provided by Reuters
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins