Featured Research

from universities, journals, and other organizations

Scientists 'bad at judging peers' published work,' says new study

Date:
October 8, 2013
Source:
Public Library of Science
Summary:
Are scientists any good at judging the importance of the scientific work of others? According to a new study scientists are unreliable judges of the importance of fellow researchers' published papers.

Are scientists any good at judging the importance of the scientific work of others? According to a study published 8 October in the open access journal PLOS Biology (with an accompanying editorial), scientists are unreliable judges of the importance of fellow researchers' published papers.

The article's lead author, Professor Adam Eyre-Walker of the University of Sussex, says: "Scientists are probably the best judges of science, but they are pretty bad at it."

Prof. Eyre-Walker and Dr Nina Stoletzki studied three methods of assessing published scientific papers, using two sets of peer-reviewed articles. The three assessment methods the researchers looked at were:

  • Peer review: subjective post-publication peer review where other scientists give their opinion of a published work;
  • Number of citations: the number of times a paper is referenced as a recognised source of information in another publication;
  • Impact factor: a measure of a journal's importance, determined by the average number of times papers in a journal are cited by other scientific papers.

The findings, say the authors, show that scientists are unreliable judges of the importance of a scientific publication: they rarely agree on the importance of a particular paper and are strongly influenced by where the paper is published, over-rating science published in high-profile scientific journals. Furthermore, the authors show that the number of times a paper is subsequently referred to by other scientists bears little relation to the underlying merit of the science.

As Eyre-Walker puts it: "The three measures of scientific merit considered here are poor; in particular subjective assessments are an error-prone, biased and expensive method by which to assess merit. While the impact factor may be the most satisfactory of the methods considered, since it is a form of prepublication review, it is likely to be a poor measure of merit, since it depends on subjective assessment."

The authors argue that the study's findings could have major implications for any future assessment of scientific output, such as currently being carried out for the UK Government's forthcoming Research Excellence Framework (REF). Eyre-Walker adds: "The quality of the assessments generated during the REF is likely to be very poor, and calls into question whether the REF in its current format is a suitable method to assess scientific output."

PLOS Biology is also publishing an accompanying Editorial by Dr Jonathan Eisen of the University of California, Davis, and Drs Catriona MacCallum and Cameron Neylon from the Advocacy department of the open access organization the Public Library of Science (PLOS).

These authors welcome Eyre-Walker and Stoletski's study as being "among the first to provide a quantitative assessment of the reliability of evaluating research," and encourage scientists and other to read it. They also support their call for openness in research assessment processes. However, they caution that assessment of merit is intrinsically a complex and subjective process, with "merit" itself meaning different things to different people, and point out that Eyre-Walker and Stoletski's study "purposely avoids defining what merit is."

Dr Eisen and co-authors also tackle the suggestion that the impact factor is the "least bad" form of assessment, recommending the use of multiple metrics that appraise the article rather than the journal ("a suite of article level metrics"), an approach that PLOS has been pioneering. Such metrics might include "number of views, researcher bookmarking, social media discussions, mentions in the popular press, or the actual outcomes of the work (e.g. for practice and policy)."


Story Source:

The above story is based on materials provided by Public Library of Science. Note: Materials may be edited for content and length.


Journal Reference:

  1. Adam Eyre-Walker, Nina Stoletzki. The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations. PLoS Biology, 2013; 11 (10): e1001675 DOI: 10.1371/journal.pbio.1001675

Cite This Page:

Public Library of Science. "Scientists 'bad at judging peers' published work,' says new study." ScienceDaily. ScienceDaily, 8 October 2013. <www.sciencedaily.com/releases/2013/10/131008182345.htm>.
Public Library of Science. (2013, October 8). Scientists 'bad at judging peers' published work,' says new study. ScienceDaily. Retrieved October 23, 2014 from www.sciencedaily.com/releases/2013/10/131008182345.htm
Public Library of Science. "Scientists 'bad at judging peers' published work,' says new study." ScienceDaily. www.sciencedaily.com/releases/2013/10/131008182345.htm (accessed October 23, 2014).

Share This



More Science & Society News

Thursday, October 23, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Chameleon Camouflage to Give Tanks Cloaking Capabilities

Chameleon Camouflage to Give Tanks Cloaking Capabilities

Reuters - Innovations Video Online (Oct. 22, 2014) Inspired by the way a chameleon changes its colour to disguise itself; scientists in Poland want to replace traditional camouflage paint with thousands of electrochromic plates that will continuously change colour to blend with its surroundings. The first PL-01 concept tank prototype will be tested within a few years, with scientists predicting that a similar technology could even be woven into the fabric of a soldiers' clothing making them virtually invisible to the naked eye. Matthew Stock reports. Video provided by Reuters
Powered by NewsLook.com
Jet Sales Lift Boeing Profit 18 Pct.

Jet Sales Lift Boeing Profit 18 Pct.

Reuters - Business Video Online (Oct. 22, 2014) Strong jet demand has pushed Boeing to raise its profit forecast for the third time, but analysts were disappointed by its small cash flow. Fred Katayama reports. Video provided by Reuters
Powered by NewsLook.com
U.S. Issues Ebola Travel Restrictions, Are Visa Bans Next?

U.S. Issues Ebola Travel Restrictions, Are Visa Bans Next?

Newsy (Oct. 22, 2014) Now that the U.S. is restricting travel from West Africa, some are dropping questions about a travel ban and instead asking about visa bans. Video provided by Newsy
Powered by NewsLook.com
US to Track Everyone Coming from Ebola Nations

US to Track Everyone Coming from Ebola Nations

AP (Oct. 22, 2014) Stepping up their vigilance against Ebola, federal authorities said Wednesday that everyone traveling into the US from Ebola-stricken nations will be monitored for symptoms for 21 days. (Oct. 22) Video provided by AP
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:

Strange & Offbeat Stories


Science & Society

Business & Industry

Education & Learning

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins