Featured Research

from universities, journals, and other organizations

Statistical Analysis Of Complex Data Sets With Robust Statistical Methods

Date:
April 12, 2007
Source:
European Science Foundation
Summary:
Robust statistical analysis methods capable of dealing with large complex data sets are required more than ever before in almost all branches of science. The European Science Foundation's three-year SACD network developed new methods for extracting key structural features within the data. Such features can include outlying values that may be particularly significant within the increasingly large and complex data sets generated in financial markets, medical diagnostics, environmental surveys, and other sources.

Robust statistical analysis methods capable of dealing with large complex data sets are required more than ever before in almost all branches of science. The European Science Foundation’s three-year SACD network, which was completed in December 2006, developed new methods for extracting key structural features within the data. Such features can include outlying values that may be particularly significant within the increasingly large and complex data sets generated in financial markets, medical diagnostics, environmental surveys, and other sources.

“Outliers often indicate the most interesting data points, like polluted areas for environmental data, or irregularities in online monitoring of patients,” said SACD chair Christophe Croux. On this front the programme has almost completely achieved its objectives, according to Croux. “A lot of work has been done in developing new methods, especially for analyzing large data sets, that can cope with outlying atypical values. This resulted in a number of publications related to the subject of the network”.

Particular progress has been made detecting outliers in multivariate time series, Croux added. This is a significant development for a number of analysis and monitoring applications involving measurements of different but related quantities that vary over time. Among many such applications are: monitoring of telecommunication networks to assess how performance and reliability are affected by events such as upgrades, surges in demand, and local link failures; monitoring noise in the vicinity of an airport; modeling the behaviour of financial markets in response to geopolitical events; and tracking the condition of patients in intensive care via several measurements such as pulse rate, blood pressure, lung water etc.

Without robust analysis methods it is easy to miss significant outlyers in such multivariate data. In some cases the outlyers only show up clearly when considering all the variables together, and yet may indicate something significant that could easily be missed, such as a sudden deterioration in a critical patient’s condition.

SACD has also advanced the field of chemometrics, which is the application of multivariate analysis methods to data of chemical interest, with some of the developments now implemented in software written by members of the network. The same principles have been applied to analysis of risks of stock investments, and measuring volatility of financial markets.

In some cases it is desirable to eliminate outlyers from data sets in order to identify the most likely response of a particular variable to different events. Within SACD, a method was developed to do this for analysis of the relationship between various economic parameters and the yield of stocks. For this it is necessary to concentrate on the bulk of the data rather than the exceptions or outlyers. “In order to do so we have to identify these extreme observations in order to downweight or reject them from the computations,” said Croux. When there are multiple variables this is more difficult, and one of the major achievements of SACD has been to find new ways of condensing and summarizing the data in such a way that the main structure of the data can be retrieved, making it also easier to detect the outliers.

Croux admits there is more work to be done, particularly in dealing with highly complex data sets, and with problems involving many variables and small sample sizes. “Important steps to be taken include robust methods that can deal with categorical data and missing values.”


Story Source:

The above story is based on materials provided by European Science Foundation. Note: Materials may be edited for content and length.


Cite This Page:

European Science Foundation. "Statistical Analysis Of Complex Data Sets With Robust Statistical Methods." ScienceDaily. ScienceDaily, 12 April 2007. <www.sciencedaily.com/releases/2007/04/070411110003.htm>.
European Science Foundation. (2007, April 12). Statistical Analysis Of Complex Data Sets With Robust Statistical Methods. ScienceDaily. Retrieved September 2, 2014 from www.sciencedaily.com/releases/2007/04/070411110003.htm
European Science Foundation. "Statistical Analysis Of Complex Data Sets With Robust Statistical Methods." ScienceDaily. www.sciencedaily.com/releases/2007/04/070411110003.htm (accessed September 2, 2014).

Share This




More Computers & Math News

Tuesday, September 2, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Google Teases India Event, Possible Android One Reveal

Google Teases India Event, Possible Android One Reveal

Newsy (Sep. 1, 2014) Google has announced a Sept. 15 event in India during which they're expected to reveal their Android One phones. Video provided by Newsy
Powered by NewsLook.com
Google's Self-Driving Car Still Has Many Flaws

Google's Self-Driving Car Still Has Many Flaws

Newsy (Sep. 1, 2014) You've seen a lot of Google's self-driving car, but that doesn't mean it's coming soon. A new report says the vehicle is nowhere near road ready. Video provided by Newsy
Powered by NewsLook.com
Apple's Rumored iWatch Could Cost $400

Apple's Rumored iWatch Could Cost $400

Newsy (Aug. 31, 2014) Apple is expected to charge a premium for its still-rumored wearable device. Video provided by Newsy
Powered by NewsLook.com
Amazon Chases Netflix And HBO With Five New Pilots

Amazon Chases Netflix And HBO With Five New Pilots

Newsy (Aug. 31, 2014) Amazon has released another batch of five pilots, allowing viewers to vote on which shows will get full seasons on the company's streaming service. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins