Featured Research

from universities, journals, and other organizations

25 Years Of Conventional Evaluation Of Data Analysis Proves Worthless In Practice

Date:
September 5, 2008
Source:
Uppsala Universitet
Summary:
So-called ‘intelligent’ computer-based methods for classifying patient samples, for example, have been evaluated with the help of two methods that have completely dominated research for 25 years. Now Swedish researchers are revealing that this methodology is worthless when it comes to practical problems.

So-called ‘intelligent’ computer-based methods for classifying patient samples, for example, have been evaluated with the help of two methods that have completely dominated research for 25 years.

Related Articles


Now Swedish researchers at Uppsala University are revealing that this methodology is worthless when it comes to practical problems. The article is published in the journal Pattern Recognition Letters.

Today there is rapidly growing interest in ‘intelligent’ computer-based methods that use various classes of measurement signals, from different patient samples, for instance, to create a model for classifying new observations. This type of method is the basis for many technical applications, such as recognition of human speech, images, and fingerprints, and is now also beginning to attract new fields such as health care.

“Especially in applications in which faulty classification decisions can lead to catastrophic consequences, such as choosing the wrong form of therapy for treating cancer, it is extremely important to be able to make a reliable estimate of the performance of the classification model,” explains Mats Gustafsson, Professor of signal processing and medical bioinformatics at Uppsala University, who co-directed the new study together with Associate Professor Anders Isaksson.

To evaluate the performance of a classification model, one normally tests it on a number of trial examples that have never been involved in the design of the model. Unfortunately there are seldom tens of thousands of test examples available for this type of evaluation. In biomedicine, for instance, it is often expensive and difficult to collect the patient samples needed, especially if one wishes to analyze a rare disease. To solve this problem, many different methods have been proposed. Since the 1980s two methods have completely dominated research, namely, cross validation and resampling/bootstrapping.

“This has entailed that the performance assessment of virtually all new methods and applications reported in the scientific literature in the last 25 years has been carried out using one of these two methods,” says Mats Gustafsson.

In the new study, the Uppsala researchers use both theory and convincing computer simulations to show that this methodology is worthless in practice when the total number of examples is small in relation to the natural variation that exists among different observations. What is considered a small number depends in turn on what problem is being studied-­in other words, it is impossible to determine whether the number of examples is sufficient.

“Our main conclusion is that this methodology cannot be depended on at all, and that it therefore needs to be immediately replaces by Bayesian methods, for example, which can deliver reliable measures of the uncertainty that exists. Only then will multivariate analyses be in any position to be adopted in such critical applications as health care,” says Mats Gustafsson.


Story Source:

The above story is based on materials provided by Uppsala Universitet. Note: Materials may be edited for content and length.


Cite This Page:

Uppsala Universitet. "25 Years Of Conventional Evaluation Of Data Analysis Proves Worthless In Practice." ScienceDaily. ScienceDaily, 5 September 2008. <www.sciencedaily.com/releases/2008/09/080903101416.htm>.
Uppsala Universitet. (2008, September 5). 25 Years Of Conventional Evaluation Of Data Analysis Proves Worthless In Practice. ScienceDaily. Retrieved March 2, 2015 from www.sciencedaily.com/releases/2008/09/080903101416.htm
Uppsala Universitet. "25 Years Of Conventional Evaluation Of Data Analysis Proves Worthless In Practice." ScienceDaily. www.sciencedaily.com/releases/2008/09/080903101416.htm (accessed March 2, 2015).

Share This


More From ScienceDaily



More Computers & Math News

Monday, March 2, 2015

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

HP to Buy Aruba Networks in $3B Deal

HP to Buy Aruba Networks in $3B Deal

Reuters - Business Video Online (Mar. 2, 2015) — Hewlett-Packard is boosting its mobile computing business... buying California-based Aruba Networks- a wi-fi network gear maker for $24.67 per share. Leah Duncan reports. Video provided by Reuters
Powered by NewsLook.com
Can Curved Screen Give Samsung the Edge?

Can Curved Screen Give Samsung the Edge?

Reuters - Business Video Online (Mar. 2, 2015) — South Korea&apos;s Samsung Electronics Co Ltd unveiled its latest Galaxy S smartphones, featuring a slim body made from aircraft-grade metal, in a bid to reclaim the throne of undisputed global smartphone leader from Apple Inc. Hayley Platt reports. Video provided by Reuters
Powered by NewsLook.com
Smartphone Giants Unveil Latest Models at Technology Show

Smartphone Giants Unveil Latest Models at Technology Show

AFP (Mar. 2, 2015) — Mobile providers have been unveiling their upcoming models at the Mobile World Congress in Barcelona, showing off the latest in smartphone technology. Duration: 00:57 Video provided by AFP
Powered by NewsLook.com
Mobile World Looks to 5G

Mobile World Looks to 5G

Reuters - Business Video Online (Mar. 2, 2015) — The wireless industry&apos;s annual conference gets underway in Barcelona with 85,000 executives taking part and numerous new smartphones and watches being launched. As Ivor Bennett reports from the show the race for 5G is one of the key themes. Video provided by Reuters
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
 
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:  

Breaking News:

Strange & Offbeat Stories

 

Space & Time

Matter & Energy

Computers & Math

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:  

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile iPhone Android Web
Follow Facebook Twitter Google+
Subscribe RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins